标题： 关于神经网络梯度下降持续学习理论 显示英文标题

标题： On the Theory of Continual Learning with Gradient Descent for Neural Networks

摘要： 持续学习，即模型在不遗忘早期任务的情况下适应不断进行的任务序列的能力，是人工智能的核心目标。 为了揭示其潜在机制，我们在一个易于处理但具有代表性的设置中分析了持续学习的局限性。 具体来说，我们研究了在带有高斯噪声的XOR聚类数据集上由梯度下降训练的一层隐藏层二次神经网络，其中不同的任务对应于正交均值的不同聚类。 我们的结果根据迭代次数、样本量、任务数量和隐藏层大小，对训练和测试时间的遗忘率给出了界限。 我们的结果揭示了不同问题参数在遗忘率中的作用方面的有趣现象。 在多种设置中的数值实验验证了我们的结果，证明了它们在分析设置之外的有效性。 显示英文摘要

摘要： Continual learning, the ability of a model to adapt to an ongoing sequence of tasks without forgetting the earlier ones, is a central goal of artificial intelligence. To shed light on its underlying mechanisms, we analyze the limitations of continual learning in a tractable yet representative setting. In particular, we study one-hidden-layer quadratic neural networks trained by gradient descent on an XOR cluster dataset with Gaussian noise, where different tasks correspond to different clusters with orthogonal means. Our results obtain bounds on the rate of forgetting during train and test-time in terms of the number of iterations, the sample size, the number of tasks, and the hidden-layer size. Our results reveal interesting phenomena on the role of different problem parameters in the rate of forgetting. Numerical experiments across diverse setups confirm our results, demonstrating their validity beyond the analyzed settings.