Title: Culture In a Frame: C$^3$B as a Comic-Based Benchmark for Multimodal Culturally Awareness Show Chinese title

Title: 框架中的文化：C$^3$B 作为基于漫画的多模态文化意识基准

Abstract: Cultural awareness capabilities has emerged as a critical capability for Multimodal Large Language Models (MLLMs). However, current benchmarks lack progressed difficulty in their task design and are deficient in cross-lingual tasks. Moreover, current benchmarks often use real-world images. Each real-world image typically contains one culture, making these benchmarks relatively easy for MLLMs. Based on this, we propose C$^3$B ($\textbf{C}$omics $\textbf{C}$ross-$\textbf{C}$ultural $\textbf{B}$enchmark), a novel multicultural, multitask and multilingual cultural awareness capabilities benchmark. C$^3$B comprises over 2000 images and over 18000 QA pairs, constructed on three tasks with progressed difficulties, from basic visual recognition to higher-level cultural conflict understanding, and finally to cultural content generation. We conducted evaluations on 11 open-source MLLMs, revealing a significant performance gap between MLLMs and human performance. The gap demonstrates that C$^3$B poses substantial challenges for current MLLMs, encouraging future research to advance the cultural awareness capabilities of MLLMs. Show Chinese abstract

Abstract: 文化意识能力已成为多模态大语言模型（MLLMs）的关键能力。 然而，当前的基准测试在任务设计上缺乏逐步增加的难度，并且在跨语言任务方面存在不足。 此外，当前的基准测试通常使用现实世界的图像。 每张现实世界的图像通常包含一种文化，使得这些基准对MLLMs来说相对容易。 基于此，我们提出了C$^3$B（$\textbf{C}$omics$\textbf{C}$ross-$\textbf{C}$ultural$\textbf{B}$enchmark），一个新颖的多文化、多任务和多语言文化意识能力基准。 C$^3$B包含超过2000张图像和超过18000对问答对，构建于三个具有逐步增加难度的任务上，从基本的视觉识别到更高级的文化冲突理解，最后到文化内容生成。 我们在11个开源MLLMs上进行了评估，揭示了MLLMs与人类表现之间的显著性能差距。 这一差距表明，C$^3$B对当前的MLLMs构成了重大挑战，鼓励未来的研究进一步提升MLLMs的文化意识能力。