标题： 基于注意力的核苷酸分析的上下文感知正则化与马尔可夫集成 显示英文标题

标题： Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis

摘要： Transformer已经彻底改变了核苷酸序列分析，但捕捉长距离依赖关系仍然具有挑战性。 最近的研究表明，自回归的Transformer通常通过依赖固定长度的上下文窗口进行下一个标记预测而表现出马尔可夫行为。 然而，由于其二次复杂度，标准的自注意力机制在处理长序列时计算效率低下，并且不显式强制全局转移一致性。 我们引入了CARMANIA（基于注意力的核苷酸分析中的上下文感知正则化与马尔可夫集成），这是一种自我监督的预训练框架，通过引入转移矩阵（TM）损失来增强下一个标记（NT）预测。 TM损失将预测的标记转移与每个输入序列的经验n-gram统计对齐，鼓励模型捕捉超出局部上下文的高阶依赖关系。 这种集成使CARMANIA能够学习反映进化约束和功能组织的生物特异性序列结构。 我们在多种基因组任务上评估了CARMANIA，包括调控元件预测、功能基因分类、系统发育推断、抗菌素耐药性检测和生物合成基因簇分类。 CARMANIA比之前的最佳长上下文模型至少提高了7个百分点，在较短序列上达到了最先进的水平（在40项任务中有20项超过了之前的结果，同时运行速度大约快2.5倍），并在增强子和管家基因分类任务中表现出特别显著的改进，包括增强子预测的马修斯相关系数（MCC）绝对提高高达34个百分点。 TM损失在40项任务中的33项中提高了准确性，尤其是在局部基序或调控模式驱动预测的情况下。 显示英文摘要

摘要： Transformers have revolutionized nucleotide sequence analysis, yet capturing long-range dependencies remains challenging. Recent studies show that autoregressive transformers often exhibit Markovian behavior by relying on fixed-length context windows for next-token prediction. However, standard self-attention mechanisms are computationally inefficient for long sequences due to their quadratic complexity and do not explicitly enforce global transition consistency. We introduce CARMANIA (Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis), a self-supervised pretraining framework that augments next-token (NT) prediction with a transition-matrix (TM) loss. The TM loss aligns predicted token transitions with empirically derived n-gram statistics from each input sequence, encouraging the model to capture higher-order dependencies beyond local context. This integration enables CARMANIA to learn organism-specific sequence structures that reflect both evolutionary constraints and functional organization. We evaluate CARMANIA across diverse genomic tasks, including regulatory element prediction, functional gene classification, taxonomic inference, antimicrobial resistance detection, and biosynthetic gene cluster classification. CARMANIA outperforms the previous best long-context model by at least 7 percent, matches state-of-the-art on shorter sequences (exceeding prior results on 20 out of 40 tasks while running approximately 2.5 times faster), and shows particularly strong improvements on enhancer and housekeeping gene classification tasks, including up to a 34 percent absolute gain in Matthews correlation coefficient (MCC) for enhancer prediction. The TM loss boosts accuracy in 33 of 40 tasks, especially where local motifs or regulatory patterns drive prediction.