Title: Testing Stationarity and Change Point Detection in Reinforcement Learning Show Chinese title

Title: 测试平稳性和强化学习中的变化点检测

Abstract: We consider offline reinforcement learning (RL) methods in possibly nonstationary environments. Many existing RL algorithms in the literature rely on the stationarity assumption that requires the system transition and the reward function to be constant over time. However, the stationarity assumption is restrictive in practice and is likely to be violated in a number of applications, including traffic signal control, robotics and mobile health. In this paper, we develop a consistent procedure to test the nonstationarity of the optimal Q-function based on pre-collected historical data, without additional online data collection. Based on the proposed test, we further develop a sequential change point detection method that can be naturally coupled with existing state-of-the-art RL methods for policy optimization in nonstationary environments. The usefulness of our method is illustrated by theoretical results, simulation studies, and a real data example from the 2018 Intern Health Study. A Python implementation of the proposed procedure is available at https://github.com/limengbinggz/CUSUM-RL. Show Chinese abstract

Abstract: 我们考虑在可能非平稳环境中的离线强化学习（RL）方法。 文献中许多现有的RL算法依赖于平稳性假设，该假设要求系统转移和奖励函数随时间保持恒定。 然而，在实践中，平稳性假设是有限制的，并且在许多应用中可能会被违反，包括交通信号控制、机器人技术和移动健康。 在本文中，我们开发了一种一致的程序，基于预先收集的历史数据来测试最优Q函数的非平稳性，而无需额外的在线数据收集。 基于所提出的检验，我们进一步开发了一种顺序变化点检测方法，可以自然地与现有最先进的RL方法结合，用于非平稳环境中的策略优化。 我们的方法的有效性通过理论结果、模拟研究和2018年Intern Health Study的真实数据例子得到了说明。 所提出程序的Python实现可在https://github.com/limengbinggz/CUSUM-RL获得。