Title: Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data Show Chinese title

Title: 深度正交融合：整合影像学、病理学、基因组学和临床数据的多模态预后生物标志物发现

Abstract: Clinical decision-making in oncology involves multimodal data such as radiology scans, molecular profiling, histopathology slides, and clinical factors. Despite the importance of these modalities individually, no deep learning framework to date has combined them all to predict patient prognosis. Here, we predict the overall survival (OS) of glioma patients from diverse multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to combine information from multiparametric MRI exams, biopsy-based modalities (such as H&E slide images and/or DNA sequencing), and clinical variables into a comprehensive multimodal risk score. Prognostic embeddings from each modality are learned and combined via attention-gated tensor fusion. To maximize the information gleaned from each modality, we introduce a multimodal orthogonalization (MMO) loss term that increases model performance by incentivizing constituent embeddings to be more complementary. DOF predicts OS in glioma patients with a median C-index of 0.788 +/- 0.067, significantly outperforming (p=0.023) the best performing unimodal model with a median C-index of 0.718 +/- 0.064. The prognostic model significantly stratifies glioma patients by OS within clinical subsets, adding further granularity to prognostic clinical grading and molecular subtyping. Show Chinese abstract

Abstract: 肿瘤学中的临床决策涉及多模态数据，如影像学扫描、分子谱分析、组织病理学切片和临床因素。 尽管这些模态各自都很重要，但到目前为止，还没有一种深度学习框架能够将它们全部结合起来预测患者预后。 在此，我们使用深度正交融合（DOF）模型从多种多模态数据中预测胶质瘤患者的总体生存期（OS）。 该模型学习将多参数MRI检查、基于活检的模态（如H&E切片图像和/或DNA测序）和临床变量的信息整合成一个全面的多模态风险评分。 每个模态的预后嵌入通过注意力门控张量融合进行学习和组合。 为了最大化从每个模态中获取的信息，我们引入了一个多模态正交化（MMO）损失项，通过鼓励组成部分嵌入更加互补来提高模型性能。 DOF在胶质瘤患者中预测OS的中位C指数为0.788 +/- 0.067，显著优于最佳单模态模型的中位C指数0.718 +/- 0.064（p=0.023）。 该预后模型能够显著地根据OS对胶质瘤患者进行分层，在临床子集中增加了预后临床分级和分子亚型的精细度。