Title: Audio-Based Crowd-Sourced Evaluation of Machine Translation Quality Show Chinese title

Title: 基于音频的众包机器翻译质量评估

Abstract: Machine Translation (MT) has achieved remarkable performance, with growing interest in speech translation and multimodal approaches. However, despite these advancements, MT quality assessment remains largely text centric, typically relying on human experts who read and compare texts. Since many real-world MT applications (e.g Google Translate Voice Mode, iFLYTEK Translator) involve translation being spoken rather printed or read, a more natural way to assess translation quality would be through speech as opposed text-only evaluations. This study compares text-only and audio-based evaluations of 10 MT systems from the WMT General MT Shared Task, using crowd-sourced judgments collected via Amazon Mechanical Turk. We additionally, performed statistical significance testing and self-replication experiments to test reliability and consistency of audio-based approach. Crowd-sourced assessments based on audio yield rankings largely consistent with text only evaluations but, in some cases, identify significant differences between translation systems. We attribute this to speech richer, more natural modality and propose incorporating speech-based assessments into future MT evaluation frameworks. Show Chinese abstract

Abstract: 机器翻译（MT）已经取得了显著的性能，语音翻译和多模态方法的兴趣正在增长。 然而，尽管有这些进展，MT质量评估仍然主要以文本为中心，通常依赖于阅读和比较文本的人类专家。 由于许多现实世界的MT应用（例如Google翻译语音模式，科大讯飞翻译器）涉及的是翻译被说出而不是打印或阅读，通过语音而不是仅文本评估来评估翻译质量将更加自然。 本研究比较了WMT通用MT共享任务中10个MT系统的仅文本和基于音频的评估，使用通过亚马逊机械土耳其人收集的众包判断。 我们还进行了统计显著性测试和自我复制实验，以测试基于音频的方法的可靠性和一致性。 基于音频的众包评估产生的排名与仅文本评估大致一致，但在某些情况下，能够识别出翻译系统之间的显著差异。 我们认为这是由于语音更为丰富、更自然的模态，并建议在未来MT评估框架中纳入基于语音的评估。