标题： 非凸熵平均场优化的Best Response流方法 显示英文标题

标题： Non-convex entropic mean-field optimization via Best Response flow

摘要： 我们研究了在概率测度空间上最小化非凸泛函的问题，并且这些泛函通过相对于固定参考测度的相对熵（KL散度）正则化，同时研究了相应的熵正则化的非凸非凹极小极大问题。 我们利用最佳响应流（在文献中也称为虚构博弈流），并探讨了其收敛性如何受到所考虑泛函的非凸程度、正则化参数以及参考测度尾部行为之间关系的影响。 具体来说，我们展示了如何根据非凸泛函选择正则化器，使得最佳响应算子成为关于$L^1$-Wasserstein 距离的压缩映射，这确保了其唯一不动点的存在，该不动点随后被证明是我们的优化问题的唯一全局极小值。 这扩展了最近的结果，在这些结果中，最佳响应流被应用于解决由相对于任意参考测度的相对熵正则化的凸优化问题，并且正则化参数具有任意值。 我们的结果精确地解释了如何在选择特定正则化器的情况下放松凸性假设。 此外，我们展示了这些结果如何在强化学习的背景下应用于马尔可夫决策过程和具有平均场 softmax 参数化策略的马尔可夫博弈的策略优化。 显示英文摘要

摘要： We study the problem of minimizing non-convex functionals on the space of probability measures, regularized by the relative entropy (KL divergence) with respect to a fixed reference measure, as well as the corresponding problem of solving entropy-regularized non-convex-non-concave min-max problems. We utilize the Best Response flow (also known in the literature as the fictitious play flow) and study how its convergence is influenced by the relation between the degree of non-convexity of the functional under consideration, the regularization parameter and the tail behaviour of the reference measure. In particular, we demonstrate how to choose the regularizer, given the non-convex functional, so that the Best Response operator becomes a contraction with respect to the $L^1$-Wasserstein distance, which then ensures the existence of its unique fixed point, which is then shown to be the unique global minimizer for our optimization problem. This extends recent results where the Best Response flow was applied to solve convex optimization problems regularized by the relative entropy with respect to arbitrary reference measures, and with arbitrary values of the regularization parameter. Our results explain precisely how the assumption of convexity can be relaxed, at the expense of making a specific choice of the regularizer. Additionally, we demonstrate how these results can be applied in reinforcement learning in the context of policy optimization for Markov Decision Processes and Markov games with softmax parametrized policies in the mean-field regime.