标题： 基于分布约束的强化学习在供应链优化中的应用 显示英文标题

标题： Distributional constrained reinforcement learning for supply chain optimization

摘要： 这项工作研究了在受约束的多周期供应链背景下强化学习（RL）的问题，例如生产与库存方面的约束。我们引入了分布约束策略优化（DCPO），这是一种用于强化学习中可靠约束满足的新方法。我们的方法基于约束策略优化（CPO），但由于近似误差，在实践中会导致其收敛到不可行的策略。我们通过将分布强化学习的方面纳入DCPO来解决这一问题。具体来说，我们使用输出离散分布的神经网络来表示回报和成本价值函数，并根据相关置信度重塑成本。通过一个供应链案例研究，我们展示了DCPO能够提高强化学习策略的收敛速度，并在训练结束时确保可靠的约束满足。所提出的方法还提高了可预测性，大大减少了运行之间的回报方差；这一结果在策略梯度方法的背景下具有重要意义，因为这些方法在训练过程中固有地引入了较大的方差。 显示英文摘要

摘要： This work studies reinforcement learning (RL) in the context of multi-period supply chains subject to constraints, e.g., on production and inventory. We introduce Distributional Constrained Policy Optimization (DCPO), a novel approach for reliable constraint satisfaction in RL. Our approach is based on Constrained Policy Optimization (CPO), which is subject to approximation errors that in practice lead it to converge to infeasible policies. We address this issue by incorporating aspects of distributional RL into DCPO. Specifically, we represent the return and cost value functions using neural networks that output discrete distributions, and we reshape costs based on the associated confidence. Using a supply chain case study, we show that DCPO improves the rate at which the RL policy converges and ensures reliable constraint satisfaction by the end of training. The proposed method also improves predictability, greatly reducing the variance of returns between runs, respectively; this result is significant in the context of policy gradient methods, which intrinsically introduce significant variance during training.