Title: Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator Show Chinese title

Title: 高效的高精度PDE求解器与线性注意力神经算子

Abstract: Neural operators offer a powerful data-driven framework for learning mappings between function spaces, in which the transformer-based neural operator architecture faces a fundamental scalability-accuracy trade-off: softmax attention provides excellent fidelity but incurs quadratic complexity $\mathcal{O}(N^2 d)$ in the number of mesh points $N$ and hidden dimension $d$, while linear attention variants reduce cost to $\mathcal{O}(N d^2)$ but often suffer significant accuracy degradation. To address the aforementioned challenge, in this paper, we present a novel type of neural operators, Linear Attention Neural Operator (LANO), which achieves both scalability and high accuracy by reformulating attention through an agent-based mechanism. LANO resolves this dilemma by introducing a compact set of $M$ agent tokens $(M \ll N)$ that mediate global interactions among $N$ tokens. This agent attention mechanism yields an operator layer with linear complexity $\mathcal{O}(MN d)$ while preserving the expressive power of softmax attention. Theoretically, we demonstrate the universal approximation property, thereby demonstrating improved conditioning and stability properties. Empirically, LANO surpasses current state-of-the-art neural PDE solvers, including Transolver with slice-based softmax attention, achieving average $19.5\%$ accuracy improvement across standard benchmarks. By bridging the gap between linear complexity and softmax-level performance, LANO establishes a scalable, high-accuracy foundation for scientific machine learning applications. Show Chinese abstract

Abstract: 神经算子提供了一个强大的数据驱动框架，用于学习函数空间之间的映射，在这种框架中，基于变压器的神经算子架构面临一个基本的可扩展性-准确性权衡：softmax注意力提供了出色的保真度，但会带来与网格点数量$N$和隐藏维度$d$的二次复杂度$\mathcal{O}(N^2 d)$，而线性注意力变体将成本降低到$\mathcal{O}(N d^2)$，但通常会经历显著的准确性下降。 为了解决上述挑战，本文中，我们提出了一种新型的神经算子，即线性注意力神经算子（LANO），它通过一种基于代理的机制重新定义注意力，从而实现可扩展性和高准确性。 LANO通过引入一组紧凑的$M$代理标记$(M \ll N)$来解决这一困境，这些代理标记在$N$标记之间协调全局交互。 这种代理注意力机制产生了一个线性复杂度的运算层$\mathcal{O}(MN d)$，同时保留了softmax注意力的表达能力。 理论上，我们证明了通用逼近性质，从而证明了改进的条件性和稳定性特性。 在实验上，LANO超越了当前最先进的神经微分方程求解器，包括使用基于切片的softmax注意力的Transolver，在标准基准测试中平均实现了$19.5\%$的准确率提升。 通过弥合线性复杂度与softmax级别性能之间的差距，LANO为科学机器学习应用建立了可扩展、高精度的基础。