Title: Contrastive Forward-Forward: A Training Algorithm of Vision Transformer Show Chinese title

Title: 对比前向-前向：一种视觉变压器的训练算法

Abstract: Although backpropagation is widely accepted as a training algorithm for artificial neural networks, researchers are always looking for inspiration from the brain to find ways with potentially better performance. Forward-Forward is a new training algorithm that is more similar to what occurs in the brain, although there is a significant performance gap compared to backpropagation. In the Forward-Forward algorithm, the loss functions are placed after each layer, and the updating of a layer is done using two local forward passes and one local backward pass. Forward-Forward is in its early stages and has been designed and evaluated on simple multi-layer perceptron networks to solve image classification tasks. In this work, we have extended the use of this algorithm to a more complex and modern network, namely the Vision Transformer. Inspired by insights from contrastive learning, we have attempted to revise this algorithm, leading to the introduction of Contrastive Forward-Forward. Experimental results show that our proposed algorithm performs significantly better than the baseline Forward-Forward leading to an increase of up to 10% in accuracy and boosting the convergence speed by 5 to 20 times on Vision Transformer. Furthermore, if we take Cross Entropy as the baseline loss function in backpropagation, it will be demonstrated that the proposed modifications to the baseline Forward-Forward reduce its performance gap compared to backpropagation on Vision Transformer, and even outperforms it in certain conditions, such as inaccurate supervision. Show Chinese abstract

Abstract: 尽管反向传播被广泛接受为人工神经网络的训练算法，研究人员一直在寻找来自大脑的灵感，以找到可能表现更好的方法。 正向-正向是一种新的训练算法，与大脑中发生的情况更为相似，尽管其性能与反向传播相比存在显著差距。 在正向-正向算法中，损失函数位于每一层之后，每一层的更新是通过两次局部前向传递和一次局部反向传递完成的。 正向-正向尚处于早期阶段，已在简单的多层感知器网络上进行了设计和评估，以解决图像分类任务。 在本工作中，我们将该算法的应用扩展到更复杂和现代的网络，即视觉Transformer。 受对比学习见解的启发，我们尝试修改该算法，从而引入了对比正向-正向。 实验结果表明，我们提出的算法显著优于基线正向-正向，在视觉Transformer上的准确率提高了高达10%，并且收敛速度提高了5到20倍。 此外，如果我们以交叉熵作为反向传播中的基线损失函数，将证明对基线正向-正向的改进减少了其在视觉Transformer上与反向传播的性能差距，甚至在某些情况下（如监督不准确）超过了反向传播。