标题： 一种预测性能的诚实交叉验证估计量 显示英文标题

标题： A Honest Cross-Validation Estimator for Prediction Performance

摘要： 交叉验证是获得预测模型性能诚实评估的标准工具。常用的版本反复分割数据，在训练集上训练预测模型，在测试集上评估模型性能，并对不同数据分割的模型性能进行平均。一个广为人知的批评是，这种交叉验证过程并不能直接估计为未来使用而推荐的特定模型的性能。在本文中，我们提出了一种新方法来估计在特定（随机）训练集上训练的模型的性能。可以通过将模型应用于独立的测试集来获得一个简单的估计量。令人惊讶的是，在随机效应模型框架下，可以从其他随机分割中计算出的交叉验证估计量可以用来改进这个简单的估计量。我们开发了两个估计量——分层贝叶斯估计量和经验贝叶斯估计量——其表现与传统的交叉验证估计量和简单的单分割估计量相当或更好。模拟和一个实际数据示例展示了所提出方法的优越性能。 显示英文摘要

摘要： Cross-validation is a standard tool for obtaining a honest assessment of the performance of a prediction model. The commonly used version repeatedly splits data, trains the prediction model on the training set, evaluates the model performance on the test set, and averages the model performance across different data splits. A well-known criticism is that such cross-validation procedure does not directly estimate the performance of the particular model recommended for future use. In this paper, we propose a new method to estimate the performance of a model trained on a specific (random) training set. A naive estimator can be obtained by applying the model to a disjoint testing set. Surprisingly, cross-validation estimators computed from other random splits can be used to improve this naive estimator within a random-effects model framework. We develop two estimators -- a hierarchical Bayesian estimator and an empirical Bayes estimator -- that perform similarly to or better than both the conventional cross-validation estimator and the naive single-split estimator. Simulations and a real-data example demonstrate the superior performance of the proposed method.