标题： 刚性神经微分方程的梯度消失问题 显示英文标题

标题： The Vanishing Gradient Problem for Stiff Neural Differential Equations

摘要： 基于梯度的神经微分方程和其他参数化动力系统优化本质上依赖于对数值解相对于模型参数的微分能力。 在刚性系统中，观察到在训练过程中，控制快速衰减模式的参数敏感性变得非常小，导致优化困难。 在本文中，我们表明这种梯度消失现象不是任何特定方法的产物，而是所有A稳定和L稳定刚性数值积分方案的普遍特征。 我们分析了通用刚性积分方案的有理稳定性函数，并证明由稳定性函数导数所支配的相关参数敏感性在大刚性情况下会衰减至零。 提供了常见刚性积分方案的显式公式，详细说明了这一机制。 最后，我们严格证明了稳定性函数导数的最慢衰减率为$O(|z|^{-1})$，揭示了一个基本限制：所有A稳定的时间步进方法在刚性条件下不可避免地抑制参数梯度，这对刚性神经ODE的训练和参数识别构成了重大障碍。 显示英文摘要

摘要： Gradient-based optimization of neural differential equations and other parameterized dynamical systems fundamentally relies on the ability to differentiate numerical solutions with respect to model parameters. In stiff systems, it has been observed that sensitivities to parameters controlling fast-decaying modes become vanishingly small during training, leading to optimization difficulties. In this paper, we show that this vanishing gradient phenomenon is not an artifact of any particular method, but a universal feature of all A-stable and L-stable stiff numerical integration schemes. We analyze the rational stability function for general stiff integration schemes and demonstrate that the relevant parameter sensitivities, governed by the derivative of the stability function, decay to zero for large stiffness. Explicit formulas for common stiff integration schemes are provided, which illustrate the mechanism in detail. Finally, we rigorously prove that the slowest possible rate of decay for the derivative of the stability function is $O(|z|^{-1})$, revealing a fundamental limitation: all A-stable time-stepping methods inevitably suppress parameter gradients in stiff regimes, posing a significant barrier for training and parameter identification in stiff neural ODEs.