标题： 提示代理：一种用于自动提示语发音识别的协作多代理系统 显示英文标题

标题： Cued-Agent: A Collaborative Multi-Agent System for Automatic Cued Speech Recognition

摘要： 提示语音（CS）是一种视觉交流系统，结合唇读和手部编码，以帮助听力障碍者进行交流。 自动提示语音识别（ACSR）旨在通过人工智能驱动的方法将提示语音的手势和嘴唇动作转换为文本。 传统上，手部和嘴唇动作之间的时序异步性需要设计复杂的模块以促进有效的多模态融合。 然而，由于数据可用性有限，当前方法在充分训练这些融合机制方面表现出不足，导致性能不佳。 最近，多智能体系统在处理数据有限的复杂任务方面显示出有前景的能力。 为此，我们提出了第一个用于ACSR的协作多智能体系统，命名为Cued-Agent。 它集成了四个专业子智能体：基于多模态大语言模型的手部识别智能体，采用关键帧筛选和CS专家提示策略来解码手部动作；基于预训练Transformer的唇部识别智能体，从输入视频中提取唇部特征；手部提示解码智能体，在推理过程中以无训练的方式动态地将手部提示与唇部特征相结合；自我纠正的音素到单词智能体，首次通过语义精炼实现从音素序列到自然语言句子的后处理和端到端转换。 为了支持这项研究，我们通过从八位听力障碍者那里收集数据来扩展现有的普通话CS数据集，建立了包含十四名受试者的混合数据集。 大量实验表明，与最先进的方法相比，我们的Cued-Agent在正常和听力障碍场景下表现优异。 实现代码可在 https://github.com/DennisHgj/Cued-Agent 获取。 显示英文摘要

摘要： Cued Speech (CS) is a visual communication system that combines lip-reading with hand coding to facilitate communication for individuals with hearing impairments. Automatic CS Recognition (ACSR) aims to convert CS hand gestures and lip movements into text via AI-driven methods. Traditionally, the temporal asynchrony between hand and lip movements requires the design of complex modules to facilitate effective multimodal fusion. However, constrained by limited data availability, current methods demonstrate insufficient capacity for adequately training these fusion mechanisms, resulting in suboptimal performance. Recently, multi-agent systems have shown promising capabilities in handling complex tasks with limited data availability. To this end, we propose the first collaborative multi-agent system for ACSR, named Cued-Agent. It integrates four specialized sub-agents: a Multimodal Large Language Model-based Hand Recognition agent that employs keyframe screening and CS expert prompt strategies to decode hand movements, a pretrained Transformer-based Lip Recognition agent that extracts lip features from the input video, a Hand Prompt Decoding agent that dynamically integrates hand prompts with lip features during inference in a training-free manner, and a Self-Correction Phoneme-to-Word agent that enables post-process and end-to-end conversion from phoneme sequences to natural language sentences for the first time through semantic refinement. To support this study, we expand the existing Mandarin CS dataset by collecting data from eight hearing-impaired cuers, establishing a mixed dataset of fourteen subjects. Extensive experiments demonstrate that our Cued-Agent performs superbly in both normal and hearing-impaired scenarios compared with state-of-the-art methods. The implementation is available at https://github.com/DennisHgj/Cued-Agent.