标题： 加快供应链强化学习中的策略模拟 显示英文标题

标题： Speeding up Policy Simulation in Supply Chain RL

摘要： 在某种状态依赖策略下对动力系统进行单个轨迹的模拟是策略优化（PO）算法的核心瓶颈。 在单次模拟中必须执行的许多本质上串行的策略评估构成了这一瓶颈的主要部分。 在将PO应用于供应链优化（SCO）问题时，模拟对应于供应链一个月的一个样本路径可能需要几个小时。 我们提出了一种迭代算法来加速策略模拟，称为皮卡德迭代。 该方案仔细地将策略评估任务分配给独立的进程。 在一个迭代中，任何给定的进程仅在其分配的任务上评估策略，同时假设其他任务的“缓存”评估；缓存在迭代结束时更新。 在GPU上实现此方案，允许在单个轨迹上批量评估策略。 我们证明了SCO问题所具有的结构使得迭代次数较少即可收敛，且与时间范围无关。 我们在大规模SCO问题上实现了400倍的实际加速，即使只使用一个GPU，还在其他强化学习环境中展示了实际有效性。 显示英文摘要

摘要： Simulating a single trajectory of a dynamical system under some state-dependent policy is a core bottleneck in policy optimization (PO) algorithms. The many inherently serial policy evaluations that must be performed in a single simulation constitute the bulk of this bottleneck. In applying PO to supply chain optimization (SCO) problems, simulating a single sample path corresponding to one month of a supply chain can take several hours. We present an iterative algorithm to accelerate policy simulation, dubbed Picard Iteration. This scheme carefully assigns policy evaluation tasks to independent processes. Within an iteration, any given process evaluates the policy only on its assigned tasks while assuming a certain "cached" evaluation for other tasks; the cache is updated at the end of the iteration. Implemented on GPUs, this scheme admits batched evaluation of the policy across a single trajectory. We prove that the structure afforded by many SCO problems allows convergence in a small number of iterations independent of the horizon. We demonstrate practical speedups of 400x on large-scale SCO problems even with a single GPU, and also demonstrate practical efficacy in other RL environments.