标题： 具有随机延迟的强化学习中的状态增强方法再探 显示英文标题

标题： Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delays

摘要： 许多现实世界场景（如远程控制和传感）都涉及动作和观察的延迟。 延迟的存在会降低强化学习（RL）算法的性能，常常严重到这些算法无法学到实质性内容的地步。 本文正式描述了带有随机延迟的马尔可夫决策过程（MDP）的概念，并证明带延迟的MDP可以被转化为具有显著简化成本结构的等效标准MDP（无延迟）。 我们利用这种等价性推导出了一种无模型的延迟解析RL框架，并表明即使基于此框架构建的简单RL算法，在带有随机动作和观察延迟的环境中也能获得接近最优的奖励。 延迟解析深度Q网络（DRDQN）算法在包含多步和随机延迟的各种环境中进行了基准测试，在实现接近最优奖励以及最小化计算开销方面，相对于目前已确立的算法表现更好。 显示英文摘要

摘要： Several real-world scenarios, such as remote control and sensing, are comprised of action and observation delays. The presence of delays degrades the performance of reinforcement learning (RL) algorithms, often to such an extent that algorithms fail to learn anything substantial. This paper formally describes the notion of Markov Decision Processes (MDPs) with stochastic delays and shows that delayed MDPs can be transformed into equivalent standard MDPs (without delays) with significantly simplified cost structure. We employ this equivalence to derive a model-free Delay-Resolved RL framework and show that even a simple RL algorithm built upon this framework achieves near-optimal rewards in environments with stochastic delays in actions and observations. The delay-resolved deep Q-network (DRDQN) algorithm is bench-marked on a variety of environments comprising of multi-step and stochastic delays and results in better performance, both in terms of achieving near-optimal rewards and minimizing the computational overhead thereof, with respect to the currently established algorithms.