标题： 连续Q分数匹配：连续时间控制的扩散引导强化学习 显示英文标题

标题： Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control

摘要： 强化学习（RL）在众多领域取得了显著成功，然而，大多数现有方法是在离散时间框架下制定的。 在本工作中，我们引入了一种用于连续时间控制的新颖RL方法，其中随机微分方程支配状态-动作动力学。 与传统的基于价值函数的方法不同，我们的主要贡献是通过鞅条件对连续时间Q函数进行表征，并通过动态规划原理将扩散策略得分与学习到的连续Q函数的动作梯度联系起来。 这一见解促使提出了连续Q得分匹配（CQSM），一种基于得分的策略改进算法。 值得注意的是，我们的方法解决了连续时间RL中的一个长期挑战：在不依赖时间离散化的情况下保持Q函数的动作评估能力。 我们进一步在我们的框架内提供了线性二次（LQ）控制问题的理论闭式解。 模拟环境中的数值结果展示了我们所提出方法的有效性，并将其与流行的基线方法进行了比较。 显示英文摘要

摘要： Reinforcement learning (RL) has achieved significant success across a wide range of domains, however, most existing methods are formulated in discrete time. In this work, we introduce a novel RL method for continuous-time control, where stochastic differential equations govern state-action dynamics. Departing from traditional value function-based approaches, our key contribution is the characterization of continuous-time Q-functions via a martingale condition and the linking of diffusion policy scores to the action gradient of a learned continuous Q-function by the dynamic programming principle. This insight motivates Continuous Q-Score Matching (CQSM), a score-based policy improvement algorithm. Notably, our method addresses a long-standing challenge in continuous-time RL: preserving the action-evaluation capability of Q-functions without relying on time discretization. We further provide theoretical closed-form solutions for linear-quadratic (LQ) control problems within our framework. Numerical results in simulated environments demonstrate the effectiveness of our proposed method and compare it to popular baselines.