标题： 随机控制下一般信息结构和非马尔可夫环境的Q学习 显示英文标题

标题： Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environments

摘要： 作为主要贡献，我们在一般情况下（可能是非马尔可夫的）随机环境中，针对随机迭代（尤其是Q学习迭代）提出了一个收敛性定理。 我们的收敛条件涉及遍历性和正性准则。我们精确地刻画了迭代的极限以及环境和初始化的收敛条件。 作为第二个贡献，我们讨论了该定理对多种具有非马尔可夫环境的随机控制问题的影响与应用，包括：（i）连续状态空间完全可观测马尔可夫决策过程（MDP）的量化近似（量化破坏了马尔可夫结构），（ii）信念MDP约简的弱Feller连续部分可观测马尔可夫决策过程（POMDP）的量化近似（需要控制器了解模型），（iii）在一致可控滤波稳定性下的POMDP有限窗口近似（不需要控制器了解模型），以及（iv）多智能体模型中学习动态收敛到新一类均衡（主观Q学习均衡）的研究。 除了收敛性定理之外，上述定理的一些推论对文献而言是新的，而另一些则被解释为收敛性定理的应用。一些开放性问题也被提出。 显示英文摘要

摘要： As a primary contribution, we present a convergence theorem for stochastic iterations, and in particular, Q-learning iterates, under a general, possibly non-Markovian, stochastic environment. Our conditions for convergence involve an ergodicity and a positivity criterion. We provide a precise characterization on the limit of the iterates and conditions on the environment and initializations for convergence. As our second contribution, we discuss the implications and applications of this theorem to a variety of stochastic control problems with non-Markovian environments involving (i) quantized approximations of fully observed Markov Decision Processes (MDPs) with continuous spaces (where quantization break down the Markovian structure), (ii) quantized approximations of belief-MDP reduced partially observable MDPS (POMDPs) with weak Feller continuity and a mild version of filter stability (which requires the knowledge of the model by the controller), (iii) finite window approximations of POMDPs under a uniform controlled filter stability (which does not require the knowledge of the model), and (iv) for multi-agent models where convergence of learning dynamics to a new class of equilibria, subjective Q-learning equilibria, will be studied. In addition to the convergence theorem, some implications of the theorem above are new to the literature and others are interpreted as applications of the convergence theorem. Some open problems are noted.