Title: Directed Information $γ$-covering: An Information-Theoretic Framework for Context Engineering Show Chinese title

Title: 有向信息$γ$-覆盖：一种上下文工程的信息理论框架

Abstract: We introduce \textbf{Directed Information $\gamma$-covering}, a simple but general framework for redundancy-aware context engineering. Directed information (DI), a causal analogue of mutual information, measures asymmetric predictiveness between chunks. If $\operatorname{DI}_{i \to j} \ge H(C_j) - \gamma$, then $C_i$ suffices to represent $C_j$ up to $\gamma$ bits. Building on this criterion, we formulate context selection as a $\gamma$-cover problem and propose a greedy algorithm with provable guarantees: it preserves query information within bounded slack, inherits $(1+\ln n)$ and $(1-1/e)$ approximations from submodular set cover, and enforces a diversity margin. Importantly, building the $\gamma$-cover is \emph{query-agnostic}: it incurs no online cost and can be computed once offline and amortized across all queries. Experiments on HotpotQA show that $\gamma$-covering consistently improves over BM25, a competitive baseline, and provides clear advantages in hard-decision regimes such as context compression and single-slot prompt selection. These results establish DI $\gamma$-covering as a principled, self-organizing backbone for modern LLM pipelines. Show Chinese abstract

Abstract: 我们引入了\textbf{有向信息$\gamma$-覆盖}，一个简单但通用的冗余感知上下文工程框架。 有向信息（DI），互信息的因果类比，衡量块之间的非对称可预测性。 如果$\operatorname{DI}_{i \to j} \ge H(C_j) - \gamma$，则$C_i$足以在最多$\gamma$位的情况下表示$C_j$。 基于这一标准，我们将上下文选择建模为一个$\gamma$-覆盖问题，并提出一种具有可证明保证的贪心算法：它在有限的松弛范围内保留查询信息，继承来自子模集合覆盖的$(1+\ln n)$和$(1-1/e)$近似，并强制执行多样性边界。 重要的是，构建$\gamma$-覆盖是\emph{与查询无关}：它不会产生在线成本，可以一次性离线计算并在所有查询中分摊。 在 HotpotQA 上的实验表明，$\gamma$-覆盖始终优于 BM25，一个有竞争力的基线，并在诸如上下文压缩和单槽提示选择等困难决策情况下提供了明显的优势。 这些结果确立了 DI$\gamma$-覆盖作为现代 LLM 流水线的一种原理性、自组织的核心结构。