标题： 我们设计了可泛化的结构知识提示吗？ 系统评估与重新思考 显示英文标题

标题： Have We Designed Generalizable Structural Knowledge Promptings? Systematic Evaluation and Rethinking

摘要： 大型语言模型（LLMs）在当前的自然语言处理研究中展示了卓越的文本生成性能。 然而，事实准确性不足仍然是悬挂在LLM摩天大楼上的阴云。 结构化知识提示（SKP）是一种通过整合结构化表示来将外部知识引入LLMs的显著范式，在许多知识密集型任务中取得了最先进的结果。 然而，现有方法通常专注于特定问题，缺乏对SKP泛化能力和能力边界的全面探索。 本文旨在从四个角度——粒度、可迁移性、可扩展性和普遍性——评估和重新思考SKP范式的泛化能力。 为了提供全面的评估，我们引入了一个新颖的多粒度、多层级基准测试SUBARU，包含9个不同粒度和难度的任务。 显示英文摘要

摘要： Large language models (LLMs) have demonstrated exceptional performance in text generation within current NLP research. However, the lack of factual accuracy is still a dark cloud hanging over the LLM skyscraper. Structural knowledge prompting (SKP) is a prominent paradigm to integrate external knowledge into LLMs by incorporating structural representations, achieving state-of-the-art results in many knowledge-intensive tasks. However, existing methods often focus on specific problems, lacking a comprehensive exploration of the generalization and capability boundaries of SKP. This paper aims to evaluate and rethink the generalization capability of the SKP paradigm from four perspectives including Granularity, Transferability, Scalability, and Universality. To provide a thorough evaluation, we introduce a novel multi-granular, multi-level benchmark called SUBARU, consisting of 9 different tasks with varying levels of granularity and difficulty.