Title: Decoupled Hierarchical Reinforcement Learning with State Abstraction for Discrete Grids Show Chinese title

Title: 离散网格的基于状态抽象的分层强化学习

Abstract: Effective agent exploration remains a core challenge in reinforcement learning (RL) for complex discrete state-space environments, particularly under partial observability. This paper presents a decoupled hierarchical RL framework integrating state abstraction (DcHRL-SA) to address this issue. The proposed method employs a dual-level architecture, consisting of a high level RL-based actor and a low-level rule-based policy, to promote effective exploration. Additionally, state abstraction method is incorporated to cluster discrete states, effectively lowering state dimensionality. Experiments conducted in two discrete customized grid environments demonstrate that the proposed approach consistently outperforms PPO in terms of exploration efficiency, convergence speed, cumulative reward, and policy stability. These results demonstrate a practical approach for integrating decoupled hierarchical policies and state abstraction in discrete grids with large-scale exploration space. Code will be available at https://github.com/XQY169/DcHRL-SA. Show Chinese abstract

Abstract: 在复杂离散状态空间环境中，尤其是在部分可观察的情况下，强化学习（RL）中的有效智能体探索仍然是一个核心挑战。 本文提出了一种解耦分层强化学习框架（DcHRL-SA），通过引入状态抽象来解决这一问题。 所提出的方法采用双层架构，包括基于高阶的RL策略网络和基于低阶规则的策略网络，以促进有效的探索。 此外，还结合了状态抽象方法来聚类离散状态，从而有效地降低状态维度。 在两个定制的离散网格环境中的实验表明，所提出的算法在探索效率、收敛速度、累积奖励和策略稳定性方面始终优于PPO。 这些结果展示了在具有大规模探索空间的离散网格中整合解耦分层策略和状态抽象的一种实用方法。 代码将在 https://github.com/XQY169/DcHRL-SA 提供。