标题： 一种评估流行病预测的框架 显示英文标题

标题： A Framework for Evaluating Epidemic Forecasts

摘要： 背景：在过去几十年中，流行病预测领域提出了许多预测方法。这些方法可以分为不同的类别，如确定性与概率性、比较方法与生成方法等。在一些更流行的比较方法中，研究人员将疫情爆发早期的观察到的流行病学数据与所提出模型的输出进行比较，以预测大流行的未来趋势和流行程度。这一领域的一个重要问题是缺乏标准的、定义明确的评估指标，以在不同算法之间选择最佳算法，以及为特定算法选择最佳可能的配置。结果：在本文中，我们提出了一种评估框架，该框架允许结合不同的特征、误差度量和排名方案来评估预测结果。我们描述了用于表征预测方法输出的各种流行病特征（Epi-features），并提供了适用于这些Epi-features的适当误差度量，以评估方法的准确性。我们关注长期预测而非短期预测，并通过评估六种预测方法来预测美国的流感，展示了该框架的实用性。我们的结果表明，即使对于单一的Epi-feature，不同的误差度量也会导致不同的排名。此外，我们的实验分析表明，当在误差度量上进行评估时，没有一种方法在预测所有Epi-features方面都优于其他方法。作为替代方案，我们提供了各种共识排名方案，以总结个体排名，从而考虑不同的误差度量。我们认为，如本文所提出的全面评估框架，将为计算流行病学社区带来价值。 显示英文摘要

摘要： Background: Over the past few decades, numerous forecasting methods have been proposed in the field of epidemic forecasting. Such methods can be classified into different categories such as deterministic vs. probabilistic, comparative methods vs. generative methods, and so on. In some of the more popular comparative methods, researchers compare observed epidemiological data from early stages of an outbreak with the output of proposed models to forecast the future trend and prevalence of the pandemic. A significant problem in this area is the lack of standard well-defined evaluation measures to select the best algorithm among different ones, as well as for selecting the best possible configuration for a particular algorithm. Results: In this paper, we present an evaluation framework which allows for combining different features, error measures, and ranking schema to evaluate forecasts. We describe the various epidemic features (Epi-features) included to characterize the output of forecasting methods and provide suitable error measures that could be used to evaluate the accuracy of the methods with respect to these Epi-features. We focus on long-term predictions rather than short-term forecasting and demonstrate the utility of the framework by evaluating six forecasting methods for predicting influenza in the United States. Our results demonstrate that different error measures lead to different rankings even for a single Epi-feature. Further, our experimental analyses show that no single method dominates the rest in predicting all Epi-features, when evaluated across error measures. As an alternative, we provide various consensus ranking schema that summarizes individual rankings, thus accounting for different error measures. We believe that a comprehensive evaluation framework, as presented in this paper, will add value to the computational epidemiology community.