Title: CatchPhrase: EXPrompt-Guided Encoder Adaptation for Audio-to-Image Generation Show Chinese title

Title: 语音到图像生成的EXPrompt引导编码器适应

Abstract: We propose CatchPhrase, a novel audio-to-image generation framework designed to mitigate semantic misalignment between audio inputs and generated images. While recent advances in multi-modal encoders have enabled progress in cross-modal generation, ambiguity stemming from homographs and auditory illusions continues to hinder accurate alignment. To address this issue, CatchPhrase generates enriched cross-modal semantic prompts (EXPrompt Mining) from weak class labels by leveraging large language models (LLMs) and audio captioning models (ACMs). To address both class-level and instance-level misalignment, we apply multi-modal filtering and retrieval to select the most semantically aligned prompt for each audio sample (EXPrompt Selector). A lightweight mapping network is then trained to adapt pre-trained text-to-image generation models to audio input. Extensive experiments on multiple audio classification datasets demonstrate that CatchPhrase improves audio-to-image alignment and consistently enhances generation quality by mitigating semantic misalignment. Show Chinese abstract

Abstract: 我们提出CatchPhrase，一种新颖的音频到图像生成框架，旨在缓解音频输入和生成图像之间的语义不对齐问题。 虽然多模态编码器的最新进展促进了跨模态生成的进步，但同音词和听觉错觉带来的歧义仍然阻碍了准确对齐。 为了解决这个问题， CatchPhrase通过利用大型语言模型（LLMs）和音频描述模型（ACMs），从弱类别标签中生成丰富的跨模态语义提示（EXPrompt Mining）。 为了解决类别级别和实例级别的不对齐问题，我们应用多模态过滤和检索来为每个音频样本选择最语义对齐的提示（EXPrompt Selector）。 然后训练一个轻量级映射网络，以适应预训练的文本到图像生成模型以处理音频输入。 在多个音频分类数据集上的广泛实验表明，CatchPhrase通过缓解语义不对齐提高了音频到图像的对齐度，并且一致地增强了生成质量。