Title: Order parameters and phase transitions of continual learning in deep neural networks Show Chinese title

Title: 连续学习在深度神经网络中的序参量和相变

Abstract: Continual learning (CL) enables animals to learn new tasks without erasing prior knowledge. CL in artificial neural networks (NNs) is challenging due to catastrophic forgetting, where new learning degrades performance on older tasks. While various techniques exist to mitigate forgetting, theoretical insights into when and why CL fails in NNs are lacking. Here, we present a statistical-mechanics theory of CL in deep, wide NNs, which characterizes the network's input-output mapping as it learns a sequence of tasks. It gives rise to order parameters (OPs) that capture how task relations and network architecture influence forgetting and anterograde interference, as verified by numerical evaluations. For networks with a shared readout for all tasks (single-head CL), the relevant-feature and rule similarity between tasks, respectively measured by two OPs, are sufficient to predict a wide range of CL behaviors. In addition, the theory predicts that increasing the network depth can effectively reduce interference between tasks, thereby lowering forgetting. For networks with task-specific readouts (multi-head CL), the theory identifies a phase transition where CL performance shifts dramatically as tasks become less similar, as measured by another task-similarity OP. While forgetting is relatively mild compared to single-head CL across all tasks, sufficiently low similarity leads to catastrophic anterograde interference, where the network retains old tasks perfectly but completely fails to generalize new learning. Our results delineate important factors affecting CL performance and suggest strategies for mitigating forgetting. Show Chinese abstract

Abstract: 持续学习（CL）使动物能够在不遗忘先前知识的情况下学习新任务。 在人工神经网络（NNs）中，由于灾难性遗忘，即新学习会降低旧任务的性能，因此CL具有挑战性。 虽然存在各种减轻遗忘的技术，但关于NNs中CL何时以及为何失败的理论见解仍然缺乏。 在此，我们提出了一种深度、宽广NNs中CL的统计力学理论，该理论描述了网络在学习一系列任务时的输入输出映射。 它产生了序参数（OPs），这些参数捕捉了任务关系和网络架构如何影响遗忘和顺行性干扰，这通过数值评估得到了验证。 对于所有任务共享一个读出层的网络（单头CL），任务的相关特征和规则相似性分别由两个OP测量，足以预测CL的广泛行为。 此外，该理论预测增加网络深度可以有效减少任务之间的干扰，从而降低遗忘。 对于具有任务特定读出层的网络（多头CL），该理论识别出一个相变，当任务相似性由另一个任务相似性OP测量时，CL性能会随着任务变得不那么相似而发生显著变化。 尽管与单头CL相比，所有任务中的遗忘相对轻微，但足够的低相似性会导致灾难性顺行性干扰，其中网络完美保留旧任务，但完全无法推广新学习。 我们的结果明确了影响CL性能的重要因素，并提出了减轻遗忘的策略。