标题： OmniEval：一个用于评估具有视觉、听觉和文本输入的全模态模型的基准 显示英文标题

标题： OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputs

摘要： 在本文中，我们介绍了OmniEval，这是一个用于评估像MiniCPM-O 2.6这样的全模态模型的基准测试，它涵盖了视觉、听觉和文本输入。 与现有的基准测试相比，我们的OmniEval有几个独特的特点：（i）全模态协作：我们设计了突出音频和视频之间强耦合的评估任务，要求模型有效利用所有模态的协作感知；（ii）视频的多样性：OmniEval包含810个音视频同步视频，其中285个中文视频和525个英文视频；（iii）任务的多样性和粒度：OmniEval包含2617对问答，包括1412个开放性问题和1205个多选题。 这些问题分为3大类任务类型和12个小任务类型，以实现全面评估。 其中，我们引入了一个更细粒度的视频定位任务，名为Grounding。 然后我们使用几种全模态模型在OmniEval上进行了实验。 我们希望我们的OmniEval能为评估从所有模态上下文中构建和理解连贯性的能力提供一个平台。 代码和数据可在https://omnieval-benchmark.github.io/找到。 显示英文摘要

摘要： In this paper, we introduce OmniEval, a benchmark for evaluating omni-modality models like MiniCPM-O 2.6, which encompasses visual, auditory, and textual inputs. Compared with existing benchmarks, our OmniEval has several distinctive features: (i) Full-modal collaboration: We design evaluation tasks that highlight the strong coupling between audio and video, requiring models to effectively leverage the collaborative perception of all modalities; (ii) Diversity of videos: OmniEval includes 810 audio-visual synchronized videos, 285 Chinese videos and 525 English videos; (iii) Diversity and granularity of tasks: OmniEval contains 2617 question-answer pairs, comprising 1412 open-ended questions and 1205 multiple-choice questions. These questions are divided into 3 major task types and 12 sub-task types to achieve comprehensive evaluation. Among them, we introduce a more granular video localization task named Grounding. Then we conduct experiments on OmniEval with several omni-modality models. We hope that our OmniEval can provide a platform for evaluating the ability to construct and understand coherence from the context of all modalities. Codes and data could be found at https://omnieval-benchmark.github.io/.