标题： 结构增强的深度强化学习在最优传输调度中的应用 显示英文标题

标题： Structure-Enhanced DRL for Optimal Transmission Scheduling

摘要： 大规模分布式动态过程的远程状态估计在工业4.0应用中起着重要作用。本文聚焦于远程估计系统的传输调度问题。首先，我们推导了衰落信道下最优传感器调度策略的一些结构特性。然后，基于这些理论指导，我们开发了一种增强结构的深度强化学习（DRL）框架，用于实现系统调度以达到最小化整体估计均方误差（MSE）。特别是，我们提出了一种增强结构的动作选择方法，倾向于选择遵循策略结构的动作。这能更有效地探索动作空间，并提高DRL代理的学习效率。此外，我们引入了一种增强结构的损失函数，对不符合策略结构的动作施加惩罚。新的损失函数引导DRL快速收敛到最优策略结构。我们的数值实验表明，与基准DRL算法相比，所提出的增强结构的DRL算法可以节省50%的训练时间，并将远程估计的MSE降低10%至25%。此外，我们还表明，所推导出的结构特性存在于广泛的动态调度问题中，而不仅仅局限于远程状态估计。 显示英文摘要

摘要： Remote state estimation of large-scale distributed dynamic processes plays an important role in Industry 4.0 applications. In this paper, we focus on the transmission scheduling problem of a remote estimation system. First, we derive some structural properties of the optimal sensor scheduling policy over fading channels. Then, building on these theoretical guidelines, we develop a structure-enhanced deep reinforcement learning (DRL) framework for optimal scheduling of the system to achieve the minimum overall estimation mean-square error (MSE). In particular, we propose a structure-enhanced action selection method, which tends to select actions that obey the policy structure. This explores the action space more effectively and enhances the learning efficiency of DRL agents. Furthermore, we introduce a structure-enhanced loss function to add penalties to actions that do not follow the policy structure. The new loss function guides the DRL to converge to the optimal policy structure quickly. Our numerical experiments illustrate that the proposed structure-enhanced DRL algorithms can save the training time by 50% and reduce the remote estimation MSE by 10% to 25% when compared to benchmark DRL algorithms. In addition, we show that the derived structural properties exist in a wide range of dynamic scheduling problems that go beyond remote state estimation.