标题： HaploOmni：用于多模态视频理解和生成的统一单Transformer 显示英文标题

标题： HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation

摘要： 随着语言模型的发展，统一的多模态理解和生成取得了显著进展，模型架构从分离的组件演变为统一的单模型框架。 本文探讨了一种高效的训练范式，用于构建用于统一多模态理解和生成的单一Transformer。 具体而言，我们提出了一种利用先验知识的多模态预热策略以扩展能力。 为了解决跨模态兼容性挑战，我们引入了特征预缩放和多模态AdaLN技术。 整合所提出的这些技术，我们提出了HaploOmni，这是一种新的单一多模态Transformer。 在有限的训练成本下，HaploOmni在多个图像和视频理解与生成基准测试中实现了与先进统一模型相当的性能。 所有代码将在 https://github.com/Tencent/HaploVLM 公开。 显示英文摘要

摘要： With the advancement of language models, unified multimodal understanding and generation have made significant strides, with model architectures evolving from separated components to unified single-model frameworks. This paper explores an efficient training paradigm to build a single transformer for unified multimodal understanding and generation. Specifically, we propose a multimodal warmup strategy utilizing prior knowledge to extend capabilities. To address cross-modal compatibility challenges, we introduce feature pre-scaling and multimodal AdaLN techniques. Integrating the proposed technologies, we present the HaploOmni, a new single multimodal transformer. With limited training costs, HaploOmni achieves competitive performance across multiple image and video understanding and generation benchmarks over advanced unified models. All codes will be made public at https://github.com/Tencent/HaploVLM.