标题： 我们离用基础模型生成缺失模态还有多远？ 显示英文标题

标题： How Far Are We from Generating Missing Modalities with Foundation Models?

摘要： 多模态基础模型在各种任务中展示了令人印象深刻的能力。 然而，它们作为缺失模态重建的即插即用解决方案的潜力仍未被充分探索。 为了弥补这一差距，我们确定并形式化了三种可能的缺失模态重建范式，并对这些范式进行了全面评估，从重建准确性和对下游任务的适应性方面涵盖了42种模型变体。 我们的分析表明，当前的基础模型在两个关键方面往往不足：(i) 从可用模态中提取细粒度语义，以及 (ii) 对生成模态的鲁棒验证。 这些限制导致了次优甚至有时不一致的生成。 为了解决这些挑战，我们提出了一种针对缺失模态重建的代理框架。 该框架根据输入上下文动态制定模态感知的挖掘策略，促进提取更丰富和更具区分性的语义特征。 此外，我们引入了一种自我精炼机制，通过内部反馈迭代地验证和提升生成模态的质量。 实验结果表明，与基线方法相比，我们的方法在缺失图像重建中的FID降低了至少14%，在缺失文本重建中的MER降低了至少10%。 代码已发布在：https://github.com/Guanzhou-Ke/AFM2. 显示英文摘要

摘要： Multimodal foundation models have demonstrated impressive capabilities across diverse tasks. However, their potential as plug-and-play solutions for missing modality reconstruction remains underexplored. To bridge this gap, we identify and formalize three potential paradigms for missing modality reconstruction, and perform a comprehensive evaluation across these paradigms, covering 42 model variants in terms of reconstruction accuracy and adaptability to downstream tasks. Our analysis reveals that current foundation models often fall short in two critical aspects: (i) fine-grained semantic extraction from the available modalities, and (ii) robust validation of generated modalities. These limitations lead to suboptimal and, at times, misaligned generations. To address these challenges, we propose an agentic framework tailored for missing modality reconstruction. This framework dynamically formulates modality-aware mining strategies based on the input context, facilitating the extraction of richer and more discriminative semantic features. In addition, we introduce a self-refinement mechanism, which iteratively verifies and enhances the quality of generated modalities through internal feedback. Experimental results show that our method reduces FID for missing image reconstruction by at least 14\% and MER for missing text reconstruction by at least 10\% compared to baselines. Code are released at: https://github.com/Guanzhou-Ke/AFM2.