标题： 视觉上下文学习在组合医学任务中是否可行？ 显示英文标题

标题： Is Visual in-Context Learning for Compositional Medical Tasks within Reach?

摘要： 在本文中，我们探讨了视觉上下文学习的潜力，以使单个模型能够处理多个任务，并在测试时适应新任务而无需重新训练。 与之前的方法不同，我们的重点是训练上下文学习者以适应任务序列，而不是单个任务。 我们的目标是使用单个模型解决涉及多个中间步骤的复杂任务，从而使用户能够在测试时灵活定义完整的视觉流程。 为此，我们首先研究了视觉上下文学习架构的特性和限制，特别关注代码本的作用。 然后，我们引入了一种使用合成组合任务生成引擎来训练上下文学习者的新方法。 该引擎从任意分割数据集中引导任务序列，使得可以对组合任务进行视觉上下文学习者的训练。 此外，我们研究了不同的基于掩码的训练目标，以深入了解如何更好地训练模型以解决复杂、组合性任务。 我们的探索不仅为多模态医学任务序列提供了重要的见解，还突出了需要解决的挑战。 显示英文摘要

摘要： In this paper, we explore the potential of visual in-context learning to enable a single model to handle multiple tasks and adapt to new tasks during test time without re-training. Unlike previous approaches, our focus is on training in-context learners to adapt to sequences of tasks, rather than individual tasks. Our goal is to solve complex tasks that involve multiple intermediate steps using a single model, allowing users to define entire vision pipelines flexibly at test time. To achieve this, we first examine the properties and limitations of visual in-context learning architectures, with a particular focus on the role of codebooks. We then introduce a novel method for training in-context learners using a synthetic compositional task generation engine. This engine bootstraps task sequences from arbitrary segmentation datasets, enabling the training of visual in-context learners for compositional tasks. Additionally, we investigate different masking-based training objectives to gather insights into how to train models better for solving complex, compositional tasks. Our exploration not only provides important insights especially for multi-modal medical task sequences but also highlights challenges that need to be addressed.