标题： TGPO：用于信号时序逻辑任务的时序基础策略优化 显示英文标题

标题： TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks

摘要： 学习控制策略以完成复杂且长期任务是机器人和自主系统中的一个核心挑战。信号时序逻辑（STL）为指定此类任务提供了一种强大且表达能力强的语言，但其非马尔可夫性质和固有的稀疏奖励使得难以通过标准强化学习（RL）算法解决。先前的RL方法仅关注有限的STL片段或使用STL鲁棒性评分作为稀疏的终端奖励。在本文中，我们提出了TGPO，即时间锚定策略优化，以解决一般的STL任务。TGPO将STL分解为定时子目标和不变约束，并提供了一个分层框架来处理这个问题。TGPO的高层组件为这些子目标提出具体的定时分配，而低层的时间条件策略则利用密集的阶段式奖励信号来实现序列化的子目标。在推理过程中，我们采样各种时间分配，并选择最有可能的分配供策略网络进行求解轨迹的展开。为了促进具有多个子目标的复杂STL的有效策略学习，我们利用已学习的评论家通过Metropolis-Hastings采样引导高层的时间搜索，将探索集中在时间可行的解决方案上。我们在五个环境中进行了实验，从低维导航到操作、无人机和四足动物运动。在广泛的STL任务下，TGPO显著优于最先进的基线（尤其是在高维和长时域情况下），与最佳基线相比，任务成功率平均提高了31.6%。代码将在https://github.com/mengyuest/TGPO提供。 显示英文摘要

摘要： Learning control policies for complex, long-horizon tasks is a central challenge in robotics and autonomous systems. Signal Temporal Logic (STL) offers a powerful and expressive language for specifying such tasks, but its non-Markovian nature and inherent sparse reward make it difficult to be solved via standard Reinforcement Learning (RL) algorithms. Prior RL approaches focus only on limited STL fragments or use STL robustness scores as sparse terminal rewards. In this paper, we propose TGPO, Temporal Grounded Policy Optimization, to solve general STL tasks. TGPO decomposes STL into timed subgoals and invariant constraints and provides a hierarchical framework to tackle the problem. The high-level component of TGPO proposes concrete time allocations for these subgoals, and the low-level time-conditioned policy learns to achieve the sequenced subgoals using a dense, stage-wise reward signal. During inference, we sample various time allocations and select the most promising assignment for the policy network to rollout the solution trajectory. To foster efficient policy learning for complex STL with multiple subgoals, we leverage the learned critic to guide the high-level temporal search via Metropolis-Hastings sampling, focusing exploration on temporally feasible solutions. We conduct experiments on five environments, ranging from low-dimensional navigation to manipulation, drone, and quadrupedal locomotion. Under a wide range of STL tasks, TGPO significantly outperforms state-of-the-art baselines (especially for high-dimensional and long-horizon cases), with an average of 31.6% improvement in task success rate compared to the best baseline. The code will be available at https://github.com/mengyuest/TGPO