标题： 基于能量的Transformer是可扩展的学习者和思考者 显示英文标题

标题： Energy-Based Transformers are Scalable Learners and Thinkers

摘要： 推理时计算技术，类似于人类的系统2思维，最近因其能提高模型性能而变得流行。 然而，大多数现有方法存在一些限制：它们是模态特定的（例如，仅在文本中工作），问题特定的（例如，可验证的领域如数学和编程），或者需要在无监督预训练的基础上进行额外的监督/训练（例如，验证器或可验证奖励）。 在本文中，我们提出问题：“是否可以推广这些系统2思维方法，并开发仅通过无监督学习来学习思考的模型？” 有趣的是，我们发现答案是肯定的，通过学习显式验证输入和候选预测之间的兼容性，然后将预测问题重新表述为相对于此验证器的优化问题。 具体来说，我们训练基于能量的变换器（EBTs）——一种新的基于能量的模型（EBM）——为每个输入和候选预测对分配一个能量值，通过基于梯度下降的能量最小化直到收敛来实现预测。 在离散（文本）和连续（视觉）模态中，我们发现EBTs在训练期间比主导的Transformer++方法扩展得更快，相对于数据、批量大小、参数、FLOPs和深度，其扩展率高达35%。 在推理过程中，EBTs在语言任务上的性能比Transformer++提高了29%，并且在图像去噪方面优于扩散变换器，同时使用更少的前向传递。 此外，我们发现，在相同或更差的预训练性能下，EBTs在大多数下游任务上优于现有模型，这表明EBTs比现有方法具有更好的泛化能力。 因此，EBTs是提升模型学习和思考能力的一种有前景的新范式。 显示英文摘要

摘要： Inference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances. However, most existing approaches suffer from several limitations: they are modality-specific (e.g., working only in text), problem-specific (e.g., verifiable domains like math and coding), or require additional supervision/training on top of unsupervised pretraining (e.g., verifiers or verifiable rewards). In this paper, we ask the question "Is it possible to generalize these System 2 Thinking approaches, and develop models that learn to think solely from unsupervised learning?" Interestingly, we find the answer is yes, by learning to explicitly verify the compatibility between inputs and candidate-predictions, and then re-framing prediction problems as optimization with respect to this verifier. Specifically, we train Energy-Based Transformers (EBTs) -- a new class of Energy-Based Models (EBMs) -- to assign an energy value to every input and candidate-prediction pair, enabling predictions through gradient descent-based energy minimization until convergence. Across both discrete (text) and continuous (visual) modalities, we find EBTs scale faster than the dominant Transformer++ approach during training, achieving an up to 35% higher scaling rate with respect to data, batch size, parameters, FLOPs, and depth. During inference, EBTs improve performance with System 2 Thinking by 29% more than the Transformer++ on language tasks, and EBTs outperform Diffusion Transformers on image denoising while using fewer forward passes. Further, we find that EBTs achieve better results than existing models on most downstream tasks given the same or worse pretraining performance, suggesting that EBTs generalize better than existing approaches. Consequently, EBTs are a promising new paradigm for scaling both the learning and thinking capabilities of models.