标题： 具有多维动作空间的策略梯度的有效熵 显示英文标题

标题： Efficient Entropy for Policy Gradient with Multidimensional Action Space

摘要： 近年来，深度强化学习已被证明在解决高维状态空间的顺序决策过程方面非常擅长，例如在Atari游戏中。 许多强化学习问题也涉及高维离散动作空间以及高维状态空间。 本文考虑了熵奖励，该奖励用于在策略梯度中鼓励探索。 在高维动作空间的情况下，计算熵及其梯度需要枚举动作空间中的所有动作，并为每个动作进行前向传播和反向传播，这可能在计算上不可行。 我们开发了几种新颖的无偏估计器用于熵奖励及其梯度。 我们将这些估计器应用于几种参数化策略模型，包括独立采样、CommNet、带修改MDP的自回归模型以及带LSTM的自回归模型。 最后，我们在两个环境中测试了我们的算法：一个多猎人多兔子网格游戏和一个多智能体多臂老虎机问题。 结果表明，我们的熵估计器在计算成本增加很小的情况下显著提高了性能。 显示英文摘要

摘要： In recent years, deep reinforcement learning has been shown to be adept at solving sequential decision processes with high-dimensional state spaces such as in the Atari games. Many reinforcement learning problems, however, involve high-dimensional discrete action spaces as well as high-dimensional state spaces. This paper considers entropy bonus, which is used to encourage exploration in policy gradient. In the case of high-dimensional action spaces, calculating the entropy and its gradient requires enumerating all the actions in the action space and running forward and backpropagation for each action, which may be computationally infeasible. We develop several novel unbiased estimators for the entropy bonus and its gradient. We apply these estimators to several models for the parameterized policies, including Independent Sampling, CommNet, Autoregressive with Modified MDP, and Autoregressive with LSTM. Finally, we test our algorithms on two environments: a multi-hunter multi-rabbit grid game and a multi-agent multi-arm bandit problem. The results show that our entropy estimators substantially improve performance with marginal additional computational cost.