标题： 无限时 horizon 离线策略估计中的双重稳健偏差减少 显示英文标题

标题： Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation

摘要： 无限时域的离策策略评估是一项极具挑战性的任务，因为典型的重要性采样（IS）估计器具有过大的方差。 最近，刘等人（2018a）提出了一种方法，通过估计平稳密度比显著降低了无限时域离策评估的方差，但代价是由于密度比估计中的误差可能会引入潜在的高偏差。 在本文中，我们开发了他们方法的一种减偏增强版，该方法可以利用学到的价值函数以获得更高的准确性。 我们的方法是双重鲁棒的，即当密度比估计或价值函数估计中任意一个达到完美时，偏差就会消失。 一般来说，当其中任意一个估计准确时，偏差也可以被减少。 理论和实验结果均表明，我们的方法相较于以往方法具有显著优势。 显示英文摘要

摘要： Infinite horizon off-policy policy evaluation is a highly challenging task due to the excessively large variance of typical importance sampling (IS) estimators. Recently, Liu et al. (2018a) proposed an approach that significantly reduces the variance of infinite-horizon off-policy evaluation by estimating the stationary density ratio, but at the cost of introducing potentially high biases due to the error in density ratio estimation. In this paper, we develop a bias-reduced augmentation of their method, which can take advantage of a learned value function to obtain higher accuracy. Our method is doubly robust in that the bias vanishes when either the density ratio or the value function estimation is perfect. In general, when either of them is accurate, the bias can also be reduced. Both theoretical and empirical results show that our method yields significant advantages over previous methods.