标题： 共线性效应在机器学习变量重要性度量中的数学理论 显示英文标题

标题： Mathematical Theory of Collinearity Effects on Machine Learning Variable Importance Measures

摘要： 在许多机器学习问题中，理解变量重要性是一个核心关注点。 两种常见的方法是Permute-and-Predict（PaP），它在验证集中随机打乱一个特征，以及Leave-One-Covariate-Out（LOCO），它在打乱训练特征后重新训练模型。 这两种方法都认为，如果使用原始数据的预测结果明显优于打乱后的预测结果，则该变量是重要的。 在线性回归中，实证研究将PaP与回归系数相关联，将LOCO与$t$统计量相关联，但缺乏正式的理论。 我们推导了这两种度量的闭式表达式，使用平方根变换来表示。 显示PaP与系数和预测变量的变异性成比例：$\text{PaP}_i = \beta_i \sqrt{2\operatorname{Var}(\mathbf{x}^v_i)}$，而LOCO与系数成比例，但受到共线性的抑制（由$\Delta$捕获）：$\text{LOCO}_i = \beta_i (1 -\Delta)\sqrt{1 + c}$。 这些推导解释了为什么PaP基本上不受多重共线性的影响，而LOCO则对其高度敏感。 蒙特卡洛模拟证实了这些发现，在不同共线性水平下均成立。 尽管这些结果是针对线性回归推导的，但我们还表明，这些结果对于像随机森林这样的模型提供了合理的近似。 总体而言，这项工作为两种广泛使用的重要性度量建立了理论基础，帮助分析人员理解它们如何受到真实系数、维度和协方差结构的影响。 这项工作连接了实证证据和理论，增强了变量重要性度量的可解释性和应用性。 显示英文摘要

摘要： In many machine learning problems, understanding variable importance is a central concern. Two common approaches are Permute-and-Predict (PaP), which randomly permutes a feature in a validation set, and Leave-One-Covariate-Out (LOCO), which retrains models after permuting a training feature. Both methods deem a variable important if predictions with the original data substantially outperform those with permutations. In linear regression, empirical studies have linked PaP to regression coefficients and LOCO to $t$-statistics, but a formal theory has been lacking. We derive closed-form expressions for both measures, expressed using square-root transformations. PaP is shown to be proportional to the coefficient and predictor variability: $\text{PaP}_i = \beta_i \sqrt{2\operatorname{Var}(\mathbf{x}^v_i)}$, while LOCO is proportional to the coefficient but dampened by collinearity (captured by $\Delta$): $\text{LOCO}_i = \beta_i (1 -\Delta)\sqrt{1 + c}$. These derivations explain why PaP is largely unaffected by multicollinearity, whereas LOCO is highly sensitive to it. Monte Carlo simulations confirm these findings across varying levels of collinearity. Although derived for linear regression, we also show that these results provide reasonable approximations for models like Random Forests. Overall, this work establishes a theoretical basis for two widely used importance measures, helping analysts understand how they are affected by the true coefficients, dimension, and covariance structure. This work bridges empirical evidence and theory, enhancing the interpretability and application of variable importance measures.