标题： 神经网络中的算法开发：流奇偶性任务的见解 显示英文标题

标题： Algorithm Development in Neural Networks: Insights from the Streaming Parity Task

摘要： 即使在过度参数化的情况下，深度神经网络也表现出惊人的泛化能力。 关于这一现象的研究主要集中在分布内的泛化，通过平滑插值进行。 然而在某些情况下，神经网络还能学习到超出原始训练集范围的数据外推，有时甚至允许无限泛化，这意味着已经学习到了能够解决该任务的算法。 在此，我们对循环神经网络（RNNs）在流式奇偶性任务上的学习动态进行了案例研究，以发展出一种有效的算法开发理论。 流式奇偶性任务是一个定义在任意长度序列上的简单但非线性的任务。 我们表明，经过足够的有限训练经验后，RNNs 会经历一个相变，从而实现完美的无限泛化。 通过一种表示动力学的有效理论，我们发现了一种隐式的表示融合效应，可以被解释为构建了一个能够重现任务的有限自动机。 总体而言，我们的结果揭示了神经网络如何从有限的训练经验中实现无限泛化的机制之一。 显示英文摘要

摘要： Even when massively overparameterized, deep neural networks show a remarkable ability to generalize. Research on this phenomenon has focused on generalization within distribution, via smooth interpolation. Yet in some settings neural networks also learn to extrapolate to data far beyond the bounds of the original training set, sometimes even allowing for infinite generalization, implying that an algorithm capable of solving the task has been learned. Here we undertake a case study of the learning dynamics of recurrent neural networks (RNNs) trained on the streaming parity task in order to develop an effective theory of algorithm development. The streaming parity task is a simple but nonlinear task defined on sequences up to arbitrary length. We show that, with sufficient finite training experience, RNNs exhibit a phase transition to perfect infinite generalization. Using an effective theory for the representational dynamics, we find an implicit representational merger effect which can be interpreted as the construction of a finite automaton that reproduces the task. Overall, our results disclose one mechanism by which neural networks can generalize infinitely from finite training experience.