Title: Efficient Learning of Voltage Control Strategies via Model-based Deep Reinforcement Learning Show Chinese title

Title: 基于模型的深度强化学习的电压控制策略高效学习

Abstract: This article proposes a model-based deep reinforcement learning (DRL) method to design emergency control strategies for short-term voltage stability problems in power systems. Recent advances show promising results in model-free DRL-based methods for power systems, but model-free methods suffer from poor sample efficiency and training time, both critical for making state-of-the-art DRL algorithms practically applicable. DRL-agent learns an optimal policy via a trial-and-error method while interacting with the real-world environment. And it is desirable to minimize the direct interaction of the DRL agent with the real-world power grid due to its safety-critical nature. Additionally, state-of-the-art DRL-based policies are mostly trained using a physics-based grid simulator where dynamic simulation is computationally intensive, lowering the training efficiency. We propose a novel model-based-DRL framework where a deep neural network (DNN)-based dynamic surrogate model, instead of a real-world power-grid or physics-based simulation, is utilized with the policy learning framework, making the process faster and sample efficient. However, stabilizing model-based DRL is challenging because of the complex system dynamics of large-scale power systems. We solved these issues by incorporating imitation learning to have a warm start in policy learning, reward-shaping, and multi-step surrogate loss. Finally, we achieved 97.5% sample efficiency and 87.7% training efficiency for an application to the IEEE 300-bus test system. Show Chinese abstract

Abstract: 本文提出了一种基于模型的深度强化学习（DRL）方法，用于设计电力系统短期电压稳定问题的紧急控制策略。 近期研究表明，无模型的基于DRL的方法在电力系统中取得了有前景的结果，但无模型方法存在样本效率低和训练时间长的问题，这两点对于使最先进的DRL算法具有实际应用价值至关重要。 DRL代理通过与真实环境交互的试错方法学习最优策略。由于其安全性要求高，希望尽量减少DRL代理直接与真实电网的交互。此外，最先进的基于DRL的策略大多使用基于物理的电网模拟器进行训练，动态仿真计算量大，降低了训练效率。 我们提出了一种新颖的基于模型的DRL框架，在该框架中，利用基于深度神经网络（DNN）的动态代理模型代替真实电网或基于物理的仿真，与策略学习框架结合，使得过程更快且样本高效。然而，由于大规模电力系统的复杂动力学特性，稳定基于模型的DRL具有挑战性。我们通过引入模仿学习来实现策略学习的热启动、奖励塑造以及多步代理损失来解决这些问题。最终，我们在IEEE 300节点测试系统中的应用达到了97.5%的样本效率和87.7%的训练效率。