标题： FIXME：面向端到端的LLM辅助设计验证基准测试 显示英文标题

标题： FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification

摘要： 尽管大型语言模型（LLMs）在硬件设计中的变革潜力巨大，但对其在设计验证方面能力的全面评估仍缺乏深入研究。 当前的努力主要集中在RTL生成和基本调试上，忽视了功能验证这一关键领域，由于硬件复杂性的迅速提升，功能验证是现代设计方法的主要瓶颈。 我们提出了FIXME，第一个端到端、多模型和开源的评估框架，用于评估LLM在硬件功能验证（FV）中的性能，以解决这一关键差距。 FIXME引入了一个结构化的三级难度层次，涵盖六个验证子领域和180个多样化的任务，使设计生命周期内的深入分析成为可能。 通过协作的人工智能与人类方法，我们使用100%经过硅验证的设计构建了一个高质量的数据集，确保全面覆盖现实世界中的挑战。 此外，通过专家指导的优化，我们将功能覆盖率提高了45.57%。 通过对最先进的LLMs如GPT-4、Claude3和LlaMA3进行严格评估，我们确定了需要改进的关键领域，并指出了有希望的研究方向，以释放LLM驱动的自动化在硬件设计验证中的全部潜力。 该基准测试可在https://github.com/ChatDesignVerification/FIXME获取。 显示英文摘要

摘要： Despite the transformative potential of Large Language Models (LLMs) in hardware design, a comprehensive evaluation of their capabilities in design verification remains underexplored. Current efforts predominantly focus on RTL generation and basic debugging, overlooking the critical domain of functional verification, which is the primary bottleneck in modern design methodologies due to the rapid escalation of hardware complexity. We present FIXME, the first end-to-end, multi-model, and open-source evaluation framework for assessing LLM performance in hardware functional verification (FV) to address this crucial gap. FIXME introduces a structured three-level difficulty hierarchy spanning six verification sub-domains and 180 diverse tasks, enabling in-depth analysis across the design lifecycle. Leveraging a collaborative AI-human approach, we construct a high-quality dataset using 100% silicon-proven designs, ensuring comprehensive coverage of real-world challenges. Furthermore, we enhance the functional coverage by 45.57% through expert-guided optimization. By rigorously evaluating state-of-the-art LLMs such as GPT-4, Claude3, and LlaMA3, we identify key areas for improvement and outline promising research directions to unlock the full potential of LLM-driven automation in hardware design verification. The benchmark is available at https://github.com/ChatDesignVerification/FIXME.