标题： 具有过参数化线性神经网络的线性回归：隐式$\ell^1$-正则化的紧上界和下界 显示英文标题

标题： Linear regression with overparameterized linear neural networks: Tight upper and lower bounds for implicit $\ell^1$-regularization

摘要： 现代机器学习模型通常是在参数数量超过训练样本数量的设定下进行训练的。为了理解梯度下降在这种过参数化模型中的隐式偏差，之前的工作研究了回归设置下的对角线线性神经网络。这些研究表明，当以较小的权重初始化时，梯度下降倾向于偏好具有最小 $\ell^1$-范数的解——这种效应被称为隐式正则化。本文研究了深度为 $D\ge 2$ 的对角线线性神经网络在过参数化线性回归问题中的隐式正则化现象。我们着重分析梯度流轨迹极限点与 $\ell^1$-最小化问题解之间的逼近误差。通过推导出逼近误差的紧致上下界，我们精确描述了逼近误差如何依赖于初始化尺度 $\alpha$。 我们的结果显示了深度之间的定性差异：对于$D \ge 3$，误差随$\alpha$线性减小，而对于$D=2$，误差以速率$\alpha^{1-\varrho}$减小，其中参数$\varrho \in [0,1)$可以被显式刻画。有趣的是，这个参数与稀疏恢复文献中研究的所谓零空间性质常数密切相关。我们通过明确的例子展示了界的有效性。数值实验验证了我们的理论发现，并表明更深的网络，即$D \ge 3$，可能会导致更好的泛化能力，特别是在现实的初始化尺度下。 显示英文摘要

摘要： Modern machine learning models are often trained in a setting where the number of parameters exceeds the number of training samples. To understand the implicit bias of gradient descent in such overparameterized models, prior work has studied diagonal linear neural networks in the regression setting. These studies have shown that, when initialized with small weights, gradient descent tends to favor solutions with minimal $\ell^1$-norm - an effect known as implicit regularization. In this paper, we investigate implicit regularization in diagonal linear neural networks of depth $D\ge 2$ for overparameterized linear regression problems. We focus on analyzing the approximation error between the limit point of gradient flow trajectories and the solution to the $\ell^1$-minimization problem. By deriving tight upper and lower bounds on the approximation error, we precisely characterize how the approximation error depends on the scale of initialization $\alpha$. Our results reveal a qualitative difference between depths: for $D \ge 3$, the error decreases linearly with $\alpha$, whereas for $D=2$, it decreases at rate $\alpha^{1-\varrho}$, where the parameter $\varrho \in [0,1)$ can be explicitly characterized. Interestingly, this parameter is closely linked to so-called null space property constants studied in the sparse recovery literature. We demonstrate the asymptotic tightness of our bounds through explicit examples. Numerical experiments corroborate our theoretical findings and suggest that deeper networks, i.e., $D \ge 3$, may lead to better generalization, particularly for realistic initialization scales.