Title: Large Language Models are Demonstration Pre-Selectors for Themselves Show Chinese title

Title: 大型语言模型是自己的演示预选器

Abstract: In-context learning (ICL) with large language models (LLMs) delivers strong few-shot performance by choosing few-shot demonstrations from the entire training data. However, existing ICL methods, which rely on similarity or diversity scores to choose demonstrations, incur high computational costs due to repeatedly retrieval from large-scale datasets for each query. To this end, we propose FEEDER (FEw yet Essential Demonstration prE-selectoR), a novel pre-selection framework that identifies a representative subset of demonstrations containing the most representative examples in the training data, tailored to specific LLMs. To construct this subset, we introduce the "sufficiency" and "necessity" metrics in the pre-selection stage and design a tree-based algorithm to identify representative examples efficiently. Once pre-selected, this representative subset can effectively replace the full training data, improving efficiency while maintaining comparable performance in ICL. Additionally, our pre-selected subset also benefits fine-tuning LLMs, where we introduce a bi-level optimization method that enhances training efficiency without sacrificing performance. Experiments with LLMs ranging from 300M to 8B parameters show that FEEDER can reduce training data size by over 20% while maintaining performance and seamlessly integrating with various downstream demonstration selection strategies in ICL. Show Chinese abstract

Abstract: 上下文学习（ICL）借助大型语言模型（LLMs）通过从整个训练数据中选择少量演示样本能够实现强大的少样本性能。然而，现有的依赖相似性或多样性分数来选择演示样本的 ICL 方法，由于需要对每个查询反复从大规模数据集中检索，计算成本较高。为此，我们提出了 FEEDER（FEw yet Essential Demonstration prE-selectoR），这是一种新颖的预选框架，旨在为特定的 LLMs 识别包含训练数据中最具有代表性的示例的代表性演示子集。为了构建这个子集，我们在预选阶段引入了“充分性”和“必要性”度量，并设计了一种基于树的算法以高效地识别代表性示例。一旦预选完成，这个代表性子集可以有效地替代完整的训练数据，在提高效率的同时保持与 ICL 相当的性能。此外，我们的预选子集还能够促进 LLMs 的微调，我们引入了一种双层优化方法，能够在不牺牲性能的情况下提升训练效率。实验结果显示，使用参数范围从 300M 到 8B 的 LLMs 进行测试时，FEEDER 可以减少超过 20% 的训练数据量，同时保持性能，并且能够无缝集成到 ICL 中的各种下游演示选择策略中。