标题： 用于脉冲强化学习的非线性光子神经形态芯片 显示英文标题

标题： Nonlinear Photonic Neuromorphic Chips for Spiking Reinforcement Learning

摘要： 光子计算芯片在加速线性计算方面取得了显著进展，但非线性计算通常在数字域中实现，这会引入额外的系统延迟和功耗，并阻碍了全功能光子神经网络芯片的实现。 在这里，我们通过协同设计简化了MZI网格和分布式反馈激光器与饱和吸收器阵列，使用不同材料，提出了并制造了一个16通道可编程的非相干光子神经形态计算芯片，使得光学域中可以实现线性和非线性脉冲计算。 此外，以前的研究主要集中在监督学习和简单的图像分类任务上。 在这里，我们首次提出了一种光子脉冲强化学习（RL）架构，并开发了一个软硬件协同训练-推理框架，以解决脉冲RL模型训练的挑战。 我们实现了整个光子脉冲RL层的大规模、高效能（光子线性计算：1.39 TOPS/W，光子非线性计算：987.65 GOPS/W）和低延迟（320 ps）端到端部署。 两个RL基准测试包括离散的CartPole任务和连续的Pendulum任务，基于脉冲近端策略优化算法进行了实验演示。 硬件-软件协同计算奖励值分别收敛到200（-250），与传统PPO算法相当。 这一实验演示解决了大规模光子非线性脉冲计算和脉冲RL训练难度的问题，并展示了一种高速、低延迟的光子脉冲RL解决方案，在机器人和自动驾驶等实时决策和控制领域具有广阔的应用前景。 显示英文摘要

摘要： Photonic computing chips have made significant progress in accelerating linear computations, but nonlinear computations are usually implemented in the digital domain, which introduces additional system latency and power consumption, and hinders the implementation of fully-functional photonic neural network chips. Here, we propose and fabricate a 16-channel programmable incoherent photonic neuromorphic computing chip by co-designing a simplified MZI mesh and distributed feedback lasers with saturable absorber array using different materials, enabling implementation of both linear and nonlinear spike computations in the optical domain. Furthermore, previous studies mainly focused on supervised learning and simple image classification tasks. Here, we propose a photonic spiking reinforcement learning (RL) architecture for the first time, and develop a software-hardware collaborative training-inference framework to address the challenge of training spiking RL models. We achieve large-scale, energy-efficient (photonic linear computation: 1.39 TOPS/W, photonic nonlinear computation: 987.65 GOPS/W) and low-latency (320 ps) end-to-end deployment of an entire layer of photonic spiking RL. Two RL benchmarks include the discrete CartPole task and the continuous Pendulum tasks are demonstrated experimentally based on spiking proximal policy optimization algorithm. The hardware-software collaborative computing reward value converges to 200 (-250) for the CartPole tasks, respectively, comparable to that of a traditional PPO algorithm. This experimental demonstration addresses the challenge of the absence of large-scale photonic nonlinear spike computation and spiking RL training difficulty, and presents a high-speed and low-latency photonic spiking RL solution with promising application prospects in fields such as real-time decision-making and control for robots and autonomous driving.