标题： 实体图像与多模态图像检索数据集 显示英文标题

标题： Entity Image and Mixed-Modal Image Retrieval Datasets

摘要： 尽管多模态学习取得了进展，但缺乏结合视觉和文本信息的混合模态图像检索的具有挑战性的基准测试。 本文介绍了一个新的基准测试，以严格评估需要深入跨模态上下文理解的图像检索。 我们提出了两个新数据集：实体图像数据集（EI），为维基百科实体提供规范图像；以及混合模态图像检索数据集（MMIR），源自WIT数据集。 MMIR基准测试包含两种具有挑战性的查询类型，要求模型将文本描述与提供的视觉实体上下文关联起来：单一实体-图像查询（一个带有描述性文本的实体图像）和多实体-图像查询（多个带有关系文本的实体图像）。 我们通过经验验证了该基准测试作为混合模态检索训练语料库和评估集的实用性。 通过众包人类标注进一步证实了这两个数据集的质量。 这些数据集可以通过GitHub页面访问：https://github.com/google-research-datasets/wit-retrieval。 显示英文摘要

摘要： Despite advances in multimodal learning, challenging benchmarks for mixed-modal image retrieval that combines visual and textual information are lacking. This paper introduces a novel benchmark to rigorously evaluate image retrieval that demands deep cross-modal contextual understanding. We present two new datasets: the Entity Image Dataset (EI), providing canonical images for Wikipedia entities, and the Mixed-Modal Image Retrieval Dataset (MMIR), derived from the WIT dataset. The MMIR benchmark features two challenging query types requiring models to ground textual descriptions in the context of provided visual entities: single entity-image queries (one entity image with descriptive text) and multi-entity-image queries (multiple entity images with relational text). We empirically validate the benchmark's utility as both a training corpus and an evaluation set for mixed-modal retrieval. The quality of both datasets is further affirmed through crowd-sourced human annotations. The datasets are accessible through the GitHub page: https://github.com/google-research-datasets/wit-retrieval.