Title: Reliable Evaluation Protocol for Low-Precision Retrieval Show Chinese title

Title: 低精度检索的可靠评估协议

Abstract: Lowering the numerical precision of model parameters and computations is widely adopted to improve the efficiency of retrieval systems. However, when computing relevance scores between the query and documents in low-precision, we observe spurious ties due to the reduced granularity. This introduces high variability in the results based on tie resolution, making the evaluation less reliable. To address this, we propose a more robust retrieval evaluation protocol designed to reduce score variation. It consists of: (1) High-Precision Scoring (HPS), which upcasts the final scoring step to higher precision to resolve tied candidates with minimal computational cost; and (2) Tie-aware Retrieval Metrics (TRM), which report expected scores, range, and bias to quantify order uncertainty of tied candidates. Our experiments test multiple models with three scoring functions on two retrieval datasets to demonstrate that HPS dramatically reduces tie-induced instability, and TRM accurately recovers expected metric values. This combination enables a more consistent and reliable evaluation system for lower-precision retrievals. Show Chinese abstract

Abstract: 降低模型参数和计算的数值精度被广泛采用以提高检索系统的效率。 然而，在低精度下计算查询与文档之间的相关性得分时，我们观察到由于粒度减少而产生的虚假平局。 这会导致基于平局解决的结果具有较高的变异性，使得评估不够可靠。 为了解决这个问题，我们提出了一种更稳健的检索评估协议，旨在减少得分变化。 它包括：(1) 高精度评分（HPS），将最终评分步骤提升到更高精度，以最低的计算成本解决平局候选；以及 (2) 有意识处理平局的检索指标（TRM），报告期望得分、范围和偏差，以量化平局候选的顺序不确定性。 我们的实验在两个检索数据集上测试了多种模型和三种评分函数，以证明HPS显著减少了由平局引起的不稳定性，而TRM准确恢复了期望的指标值。 这种组合使更低精度的检索评估系统更加一致和可靠。