Title: JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model Show Chinese title

Title: JanusDNA：一种强大的双向混合DNA基础模型

Abstract: Large language models (LLMs) have revolutionized natural language processing and are increasingly applied to other sequential data types, including genetic sequences. However, adapting LLMs to genomics presents significant challenges. Capturing complex genomic interactions requires modeling long-range dependencies within DNA sequences, where interactions often span over 10,000 base pairs, even within a single gene, posing substantial computational burdens under conventional model architectures and training paradigms. Moreover, standard LLM training approaches are suboptimal for DNA: autoregressive training, while efficient, supports only unidirectional understanding. However, DNA is inherently bidirectional, e.g., bidirectional promoters regulate transcription in both directions and account for nearly 11% of human gene expression. Masked language models (MLMs) allow bidirectional understanding but are inefficient, as only masked tokens contribute to the loss per step. To address these limitations, we introduce JanusDNA, the first bidirectional DNA foundation model built upon a novel pretraining paradigm that combines the optimization efficiency of autoregressive modeling with the bidirectional comprehension of masked modeling. JanusDNA adopts a hybrid Mamba, Attention and Mixture of Experts (MoE) architecture, combining long-range modeling of Attention with efficient sequential learning of Mamba. MoE layers further scale model capacity via sparse activation while keeping computational cost low. Notably, JanusDNA processes up to 1 million base pairs at single nucleotide resolution on a single 80GB GPU. Extensive experiments and ablations show JanusDNA achieves new SOTA results on three genomic representation benchmarks, outperforming models with 250x more activated parameters. Code: https://github.com/Qihao-Duan/JanusDNA Show Chinese abstract

Abstract: 大型语言模型（LLMs）已经革新了自然语言处理，并被越来越多地应用于其他序列数据类型，包括基因序列。 然而，将LLMs适应基因组学面临重大挑战。 捕捉复杂的基因组相互作用需要对DNA序列中的长距离依赖关系进行建模，其中相互作用通常跨越超过10,000个碱基对，甚至在一个基因内也存在，这在传统模型架构和训练范式下带来了巨大的计算负担。 此外，标准的LLM训练方法对于DNA来说并不理想：自回归训练虽然高效，但仅支持单向理解。 然而， DNA本质上是双向的，例如，双向启动子在两个方向上调控转录，并占人类基因表达的近11%。 掩码语言模型（MLMs）允许双向理解，但效率低下，因为每一步只有被掩码的标记对损失有贡献。 为了解决这些限制，我们引入了JanusDNA，这是第一个基于新颖预训练范式的双向DNA基础模型，该范式结合了自回归建模的优化效率和掩码建模的双向理解能力。 JanusDNA采用了一种混合Mamba、注意力和 专家混合（MoE）架构，将注意力的长距离建模与Mamba的高效序列学习相结合。 MoE层通过稀疏激活进一步扩展模型容量，同时保持计算成本低。 值得注意的是，JanusDNA可以在单个80GB GPU上以单核苷酸分辨率处理多达100万个碱基对。 广泛的实验和消融研究表明 JanusDNA在三个基因组表示基准上取得了新的SOTA结果，优于参数激活量多250倍的模型。 代码：https://github.com/Qihao-Duan/JanusDNA