Title: Forward- or Reverse-Mode Automatic Differentiation: What's the Difference? Show Chinese title

Title: 前向或反向模式自动微分：有什么区别？

Abstract: Automatic differentiation (AD) has been a topic of interest for researchers in many disciplines, with increased popularity since its application to machine learning and neural networks. Although many researchers appreciate and know how to apply AD, it remains a challenge to truly understand the underlying processes. From an algebraic point of view, however, AD appears surprisingly natural: it originates from the differentiation laws. In this work we use Algebra of Programming techniques to reason about different AD variants, leveraging Haskell to illustrate our observations. Our findings stem from three fundamental algebraic abstractions: (1) the notion of module over a semiring, (2) Nagata's construction of the 'idealization of a module', and (3) Kronecker's delta function, that together allow us to write a single-line abstract definition of AD. From this single-line definition, and by instantiating our algebraic structures in various ways, we derive different AD variants, that have the same extensional behaviour, but different intensional properties, mainly in terms of (asymptotic) computational complexity. We show the different variants equivalent by means of Kronecker isomorphisms, a further elaboration of our Haskell infrastructure which guarantees correctness by construction. With this framework in place, this paper seeks to make AD variants more comprehensible, taking an algebraic perspective on the matter. Show Chinese abstract

Abstract: 自动微分（AD）一直是许多学科研究人员关注的主题，自其应用于机器学习和神经网络以来，其受欢迎程度不断提高。 尽管许多研究人员欣赏并知道如何应用AD，但真正理解其底层过程仍是一个挑战。 从代数的角度来看，AD却显得出乎意料地自然：它源于微分法则。 在本工作中，我们使用编程代数技术来推理不同的AD变体，并利用Haskell来说明我们的观察结果。 我们的发现源于三个基本的代数抽象：（1）半环上的模的概念，（2）Nagata对“模的理想化”的构造，以及（3）Kronecker的delta函数，这些共同使我们能够写出一个单行的AD抽象定义。 从这个单行定义出发，并通过以不同方式实例化我们的代数结构，我们推导出不同的AD变体，它们具有相同的外延行为，但在内延属性上有所不同，主要是在（渐近）计算复杂性方面。 我们通过Kronecker同构来证明不同变体的等价性，这是对我们Haskell基础设施的进一步完善，确保了构造的正确性。 有了这个框架，本文旨在使AD变体更易理解，从代数的角度来看待这一问题。