Title: Grokking Modular Polynomials Show Chinese title

Title: 理解模多项式

Abstract: Neural networks readily learn a subset of the modular arithmetic tasks, while failing to generalize on the rest. This limitation remains unmoved by the choice of architecture and training strategies. On the other hand, an analytical solution for the weights of Multi-layer Perceptron (MLP) networks that generalize on the modular addition task is known in the literature. In this work, we (i) extend the class of analytical solutions to include modular multiplication as well as modular addition with many terms. Additionally, we show that real networks trained on these datasets learn similar solutions upon generalization (grokking). (ii) We combine these "expert" solutions to construct networks that generalize on arbitrary modular polynomials. (iii) We hypothesize a classification of modular polynomials into learnable and non-learnable via neural networks training; and provide experimental evidence supporting our claims. Show Chinese abstract

Abstract: 神经网络很容易学会模算术任务的一个子集，但在其余任务上无法推广。 这种局限性不受架构选择和训练策略的影响。 另一方面，在文献中已知有多层感知器（MLP）网络权重的分析解，这些网络在模加法任务上具有泛化能力。 在这项工作中，我们（i）将分析解的类扩展到包括模乘法以及具有多个项的模加法。 此外，我们证明了在这些数据集上训练的实际网络在泛化时学习到了类似的解（grokking）。 （ii）我们将这些“专家”解结合起来，构建出能够对任意模多项式进行泛化的网络。 （iii）我们假设可以通过神经网络训练将模多项式分类为可学习和不可学习，并提供了支持我们观点的实验证据。