标题： 通过在小批量数据上使用后果主义权重更新改进反向传播算法 显示英文标题

标题： Improving the Backpropagation Algorithm with Consequentialism Weight Updates over Mini-Batches

摘要： 许多尝试旨在改进自适应滤波器，这些滤波器也可以用于改进反向传播（BP）。 归一化最小均值平方误差（NLMS）是从最小均值平方误差（LMS）派生出的最成功算法之一。 然而，其扩展到多层神经网络之前尚未发生。 在这里，我们首先表明可以将多层神经网络视为自适应滤波器的堆叠。 此外，我们引入了比仿射投影算法（APA）中复杂的几何解释更易理解的NLMS解释，这种解释可以轻松推广到卷积神经网络，并且在小批量训练中表现更好。 通过这种新观点，我们引入了一种更好的算法，该算法可以预测并纠正反向传播中发生的不利后果，甚至在它们发生之前就进行修正。 最后，所提出的方法与随机梯度下降（SGD）兼容，并适用于基于动量的导数，如 RMSProp、Adam 和 NAG。 我们的实验表明了该算法在深度神经网络训练中的有效性。 显示英文摘要

摘要： Many attempts took place to improve the adaptive filters that can also be useful to improve backpropagation (BP). Normalized least mean squares (NLMS) is one of the most successful algorithms derived from Least mean squares (LMS). However, its extension to multi-layer neural networks has not happened before. Here, we first show that it is possible to consider a multi-layer neural network as a stack of adaptive filters. Additionally, we introduce more comprehensible interpretations of NLMS than the complicated geometric interpretation in affine projection algorithm (APA) for a single fully-connected (FC) layer that can easily be generalized to, for instance, convolutional neural networks and also works better with mini-batch training. With this new viewpoint, we introduce a better algorithm by predicting then emending the adverse consequences of the actions that take place in BP even before they happen. Finally, the proposed method is compatible with stochastic gradient descent (SGD) and applicable to momentum-based derivatives such as RMSProp, Adam, and NAG. Our experiments show the usefulness of our algorithm in the training of deep neural networks.