Title: BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology Show Chinese title

Title: BixBench：计算生物学中基于大语言模型的智能体综合基准

Abstract: Large Language Models (LLMs) and LLM-based agents show great promise in accelerating scientific research. Existing benchmarks for measuring this potential and guiding future development continue to evolve from pure recall and rote knowledge tasks, towards more practical work such as literature review and experimental planning. Bioinformatics is a domain where fully autonomous AI-driven discovery may be near, but no extensive benchmarks for measuring progress have been introduced to date. We therefore present the Bioinformatics Benchmark (BixBench), a dataset comprising over 50 real-world scenarios of practical biological data analysis with nearly 300 associated open-answer questions designed to measure the ability of LLM-based agents to explore biological datasets, perform long, multi-step analytical trajectories, and interpret the nuanced results of those analyses. We evaluate the performance of two frontier LLMs (GPT-4o and Claude 3.5 Sonnet) using a custom agent framework we open source. We find that even the latest frontier models only achieve 17% accuracy in the open-answer regime, and no better than random in a multiple-choice setting. By exposing the current limitations of frontier models, we hope BixBench can spur the development of agents capable of conducting rigorous bioinformatic analysis and accelerate scientific discovery. Show Chinese abstract

Abstract: 大型语言模型（LLMs）和基于LLM的智能体在加速科学研究方面展现出巨大潜力。 目前用于衡量这种潜力并指导未来发展的基准测试，正从纯记忆和机械知识任务，逐渐转向更实际的工作，如文献综述和实验设计。 生物信息学是一个完全自主的AI驱动发现可能接近的领域，但迄今为止尚未引入广泛的基准测试来衡量进展。 因此，我们提出了生物信息学基准（BixBench），这是一个包含超过50个现实世界场景的数据库，涉及近300个相关的开放性问题，旨在衡量基于LLM的智能体探索生物数据集、执行长期多步骤分析轨迹以及解释这些分析的细微结果的能力。 我们使用一个我们开源的自定义智能体框架来评估两种前沿LLM（GPT-4o和Claude 3.5 Sonnet）的性能。 我们发现，即使是最新的前沿模型，在开放性问题设置中也只能达到17%的准确率，在多项选择设置中甚至不如随机猜测。 通过揭示前沿模型的当前局限性，我们希望BixBench能够推动能够进行严格生物信息学分析的智能体的发展，并加速科学发现。