标题： 深度学习动态的两阶段观点 显示英文标题

标题： A Two-Phase Perspective on Deep Learning Dynamics

摘要： 我们提出，深度神经网络中的学习分为两个阶段：快速曲线拟合阶段，随后是较慢的压缩或粗粒化阶段。 这一观点得到了三种现象共享的时间结构的支持：理解、双下降和信息瓶颈，所有这些现象都在训练误差达到零之后才表现出泛化的延迟开始。 我们通过实证表明，在两种相当不同的设置中，相关的时间尺度是一致的。 隐藏层与输入数据之间的互信息作为一种自然的进展度量出现，补充了基于电路的指标，如局部复杂性和线性映射数。 我们认为第二阶段不是由标准训练算法主动优化的，可能会被不必要的延长。 借鉴重整化群的类比，我们建议这个压缩阶段反映了有原则的遗忘形式，这对于泛化至关重要。 显示英文摘要

摘要： We propose that learning in deep neural networks proceeds in two phases: a rapid curve fitting phase followed by a slower compression or coarse graining phase. This view is supported by the shared temporal structure of three phenomena: grokking, double descent and the information bottleneck, all of which exhibit a delayed onset of generalization well after training error reaches zero. We empirically show that the associated timescales align in two rather different settings. Mutual information between hidden layers and input data emerges as a natural progress measure, complementing circuit-based metrics such as local complexity and the linear mapping number. We argue that the second phase is not actively optimized by standard training algorithms and may be unnecessarily prolonged. Drawing on an analogy with the renormalization group, we suggest that this compression phase reflects a principled form of forgetting, critical for generalization.