Title: Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization Show Chinese title

Title: 均值漂移拒绝：无需小批量统计或归一化的深度神经网络训练

Abstract: Deep convolutional neural networks are known to be unstable during training at high learning rate unless normalization techniques are employed. Normalizing weights or activations allows the use of higher learning rates, resulting in faster convergence and higher test accuracy. Batch normalization requires minibatch statistics that approximate the dataset statistics but this incurs additional compute and memory costs and causes a communication bottleneck for distributed training. Weight normalization and initialization-only schemes do not achieve comparable test accuracy. We introduce a new understanding of the cause of training instability and provide a technique that is independent of normalization and minibatch statistics. Our approach treats training instability as a spatial common mode signal which is suppressed by placing the model on a channel-wise zero-mean isocline that is maintained throughout training. Firstly, we apply channel-wise zero-mean initialization of filter kernels with overall unity kernel magnitude. At each training step we modify the gradients of spatial kernels so that their weighted channel-wise mean is subtracted in order to maintain the common mode rejection condition. This prevents the onset of mean shift. This new technique allows direct training of the test graph so that training and test models are identical. We also demonstrate that injecting random noise throughout the network during training improves generalization. This is based on the idea that, as a side effect, batch normalization performs deep data augmentation by injecting minibatch noise due to the weakness of the dataset approximation. Our technique achieves higher accuracy compared to batch normalization and for the first time shows that minibatches and normalization are unnecessary for state-of-the-art training. Show Chinese abstract

Abstract: 深度卷积神经网络在高学习率下训练时已知是不稳定的，除非使用了归一化技术。 对权重或激活进行归一化允许使用更高的学习率，从而加快收敛并提高测试准确性。 批量归一化需要小批量统计量来近似数据集统计量，但这会增加额外的计算和内存成本，并在分布式训练中造成通信瓶颈。 仅使用权重归一化和初始化的方案无法达到可比较的测试准确性。 我们提出了一种对训练不稳定原因的新理解，并提供了一种独立于归一化和小批量统计的技术。 我们的方法将训练不稳定视为一种空间共模信号，通过将模型置于通道级零均值等高线上来抑制该信号，并在整个训练过程中保持该等高线。 首先，我们对滤波器核进行通道级零均值初始化，整体核幅度为单位大小。 在每个训练步骤中，我们修改空间核的梯度，以使它们的加权通道级均值被减去，从而维持共模抑制条件。 这可以防止均值偏移的发生。 这种新技术允许直接训练测试图，使得训练模型和测试模型相同。 我们还证明了在训练过程中在整个网络中注入随机噪声可以提高泛化能力。 这是基于这样的想法：作为副作用，批量归一化由于数据集近似的薄弱性，通过注入小批量噪声来进行深度数据增强。 与批量归一化相比，我们的技术实现了更高的准确性，并首次表明小批量和归一化对于最先进的训练是不必要的。