标题： 基于LLM的重排序器的效率-效果重新排序FLOPs 显示英文标题

标题： Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers

摘要： 大型语言模型（LLMs）最近被应用于信息检索中的重新排序任务，取得了强大的性能。 然而，它们的高计算需求通常会阻碍实际部署。 现有的研究使用代理指标（如延迟、前向传递次数、输入标记数和输出标记数）来评估基于LLM的重新排序器的效率。 然而，这些指标依赖于硬件和运行时选择（\eg 并行与否，批处理大小等），并且常常无法考虑模型大小，使得难以解释，并模糊了对效率与效果权衡的评估。 为了解决这个问题，我们提出了 E\textsuperscript{2}R-FLOPs，用于基于LLM的重新排序器：每PetaFLOP的排名指标（RPP）用于按计算的相关性以及每PetaFLOP的查询数（QPP）用于与硬件无关的吞吐量。 伴随着新指标，构建了一个可解释的 FLOPs估算器，即使不运行任何实验也能估计基于LLM的重新排序器的FLOPs。 基于所提出的指标，我们进行了全面的实验，评估了具有不同架构的广泛范围的基于LLM的重新排序器，研究了效率与效果的权衡，并将这一问题引起研究界的关注。 显示英文摘要

摘要： Large Language Models (LLMs) have recently been applied to reranking tasks in information retrieval, achieving strong performance. However, their high computational demands often hinder practical deployment. Existing studies evaluate the efficiency of LLM-based rerankers using proxy metrics such as latency, the number of forward passes, input tokens, and output tokens. However, these metrics depend on hardware and running-time choices (\eg parallel or not, batch size, etc), and often fail to account for model size, making it difficult to interpret and obscuring the evaluation of the efficiency-effectiveness tradeoff. To address this issue, we propose E\textsuperscript{2}R-FLOPs, for LLM-based rerankers: ranking metrics per PetaFLOP (RPP) for relevance per compute and queries per PetaFLOP (QPP) for hardware-agnostic throughput. Companied with the new metrics, an interpretable FLOPs estimator is built to estimate the FLOPs of an LLM-based reranker even without running any experiments. Based on the proposed metrics, we conduct comprehensive experiments to evaluate a wide range of LLM-based rerankers with different architecture, studying the efficiency-effectiveness trade-off and bringing this issue to the attention of the research community.