标题： 表格方法和深度学习用于威特指数 显示英文标题

标题： Tabular and Deep Learning for the Whittle Index

摘要： Whittle指数策略是一种启发式方法，在应用于被称为“非静止多臂老虎机问题”（RMABPs）的问题类别时表现出显著优异的性能（具有保证的渐近最优性）。 在本文中，我们提出了QWI和QWINN两种强化学习算法，分别采用表格形式和深度学习方法，以学习总折扣准则下的Whittle指数。 关键特点是使用两个时间尺度，一个较快的时间尺度用于更新状态-动作Q值，一个相对较慢的时间尺度用于更新Whittle指数。 在我们的主要理论结果中，我们证明了QWI（一种表格实现）收敛到真实的Whittle指数。 然后我们提出了QWINN，这是一种利用神经网络在较快时间尺度上计算Q值的QWI算法的改进版本，能够从一个状态推断出另一个状态的信息，并自然地扩展到大型状态空间环境。 对于QWINN，我们证明了Bellman误差的所有局部最小值都是局部稳定的平衡点，这是基于DQN的方案中的首个此类结果。 数值计算表明，QWI和QWINN比标准的Q学习算法、基于神经网络的近似Q学习和其他最先进的算法收敛得更快。 显示英文摘要

摘要： The Whittle index policy is a heuristic that has shown remarkably good performance (with guaranteed asymptotic optimality) when applied to the class of problems known as Restless Multi-Armed Bandit Problems (RMABPs). In this paper we present QWI and QWINN, two reinforcement learning algorithms, respectively tabular and deep, to learn the Whittle index for the total discounted criterion. The key feature is the use of two time-scales, a faster one to update the state-action Q -values, and a relatively slower one to update the Whittle indices. In our main theoretical result we show that QWI, which is a tabular implementation, converges to the real Whittle indices. We then present QWINN, an adaptation of QWI algorithm using neural networks to compute the Q -values on the faster time-scale, which is able to extrapolate information from one state to another and scales naturally to large state-space environments. For QWINN, we show that all local minima of the Bellman error are locally stable equilibria, which is the first result of its kind for DQN-based schemes. Numerical computations show that QWI and QWINN converge faster than the standard Q -learning algorithm, neural-network based approximate Q-learning and other state of the art algorithms.