Title: Nonlocal Neural Tangent Kernels via Parameter-Space Interactions Show Chinese title

Title: 通过参数空间交互的非局部神经切线核

Abstract: The Neural Tangent Kernel (NTK) framework has provided deep insights into the training dynamics of neural networks under gradient flow. However, it relies on the assumption that the network is differentiable with respect to its parameters, an assumption that breaks down when considering non-smooth target functions or parameterized models exhibiting non-differentiable behavior. In this work, we propose a Nonlocal Neural Tangent Kernel (NNTK) that replaces the local gradient with a nonlocal interaction-based approximation in parameter space. Nonlocal gradients are known to exist for a wider class of functions than the standard gradient. This allows NTK theory to be extended to nonsmooth functions, stochastic estimators, and broader families of models. We explore both fixed-kernel and attention-based formulations of this nonlocal operator. We illustrate the new formulation with numerical studies. Show Chinese abstract

Abstract: 神经切线核（NTK）框架为在梯度流下神经网络的训练动力学提供了深入的见解。 然而，它依赖于网络相对于其参数是可微的假设，当考虑非光滑目标函数或表现出不可微行为的参数化模型时，这一假设会失效。 在本工作中，我们提出了一种非局部神经切线核（NNTK），该核在参数空间中用基于非局部相互作用的近似代替了局部梯度。 非局部梯度已知存在于比标准梯度更广泛的函数类别中。 这使得NTK理论可以扩展到非光滑函数、随机估计器和更广泛的模型家族。 我们探讨了这种非局部算子的固定核和基于注意力的公式。 我们通过数值研究说明了新的公式。