标题： 从动力系统的角度分析神经网络 显示英文标题

标题： A Dynamical Systems Perspective on the Analysis of Neural Networks

摘要： 在本章中，我们利用动力系统来分析机器学习算法的几个方面。作为说明性的贡献，我们展示了如何将来自深度神经网络、(随机)梯度下降及相关主题的广泛挑战重新表述为动力学陈述。我们还解决了三个具体的挑战。首先，我们考虑信息通过神经网络的传播过程，即我们研究不同架构的输入输出映射。我们解释了表示给定正则性任意函数的增强神经微分方程的普遍嵌入性质，以及基于适当函数类的多层感知器和神经微分方程的分类，以及神经延迟方程中的记忆依赖性。其次，我们从动态角度考虑神经网络的训练方面。我们描述了对梯度下降的动力系统视角，并研究了过定问题的稳定性。然后我们将这种分析扩展到过度参数化设置，并描述了稳定性边缘现象，也包括可能对隐式偏差的解释。对于随机梯度下降，我们通过插值解的李雅普诺夫指数给出了过度参数化设置下的稳定性结果。第三，我们解释了关于神经网络平均场极限的几个结果。我们描述了一个结果，该结果通过有向图测度将现有技术扩展到涉及图极限的异质神经网络。这表明大量神经网络自然落在图上的Kuramoto型模型及其大图极限框架内。最后，我们指出，使用动力学来研究可解释和可靠的人工智能的类似策略也可以应用于生成模型等场景，或者梯度训练方法中的基本问题，如反向传播或梯度消失/爆炸问题。 显示英文摘要

摘要： In this chapter, we utilize dynamical systems to analyze several aspects of machine learning algorithms. As an expository contribution we demonstrate how to re-formulate a wide variety of challenges from deep neural networks, (stochastic) gradient descent, and related topics into dynamical statements. We also tackle three concrete challenges. First, we consider the process of information propagation through a neural network, i.e., we study the input-output map for different architectures. We explain the universal embedding property for augmented neural ODEs representing arbitrary functions of given regularity, the classification of multilayer perceptrons and neural ODEs in terms of suitable function classes, and the memory-dependence in neural delay equations. Second, we consider the training aspect of neural networks dynamically. We describe a dynamical systems perspective on gradient descent and study stability for overdetermined problems. We then extend this analysis to the overparameterized setting and describe the edge of stability phenomenon, also in the context of possible explanations for implicit bias. For stochastic gradient descent, we present stability results for the overparameterized setting via Lyapunov exponents of interpolation solutions. Third, we explain several results regarding mean-field limits of neural networks. We describe a result that extends existing techniques to heterogeneous neural networks involving graph limits via digraph measures. This shows how large classes of neural networks naturally fall within the framework of Kuramoto-type models on graphs and their large-graph limits. Finally, we point out that similar strategies to use dynamics to study explainable and reliable AI can also be applied to settings such as generative models or fundamental issues in gradient training methods, such as backpropagation or vanishing/exploding gradients.