标题： 在定点算术下量化神经网络的表达能力 显示英文标题

标题： On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmetic

摘要： 关于神经网络表达能力的研究通常考虑实参数和没有舍入误差的操作。在这项工作中，我们研究了在可能由于舍入而产生错误的离散定点参数和定点操作下的量化网络的通用逼近性质。我们首先给出了量化网络通用逼近的定点算术和激活函数的一个必要条件和一个充分条件。然后，我们证明了许多流行的激活函数满足我们的充分条件，例如Sigmoid、ReLU、ELU、SoftPlus、SiLU、Mish和GELU。换句话说，使用这些激活函数的网络能够实现通用逼近。我们进一步表明，在激活函数的一个温和条件下，我们的必要条件和充分条件是一致的：例如，对于激活函数$\sigma$，存在一个定点数$x$满足$\sigma(x)=0$。也就是说，我们找到了一类广泛激活函数的必要且充分条件。最后，我们展示了即使在$\{-1,1\}$中使用二进制权重的量化网络也可以对实际激活函数实现通用逼近。 显示英文摘要

摘要： Research into the expressive power of neural networks typically considers real parameters and operations without rounding error. In this work, we study universal approximation property of quantized networks under discrete fixed-point parameters and fixed-point operations that may incur errors due to rounding. We first provide a necessary condition and a sufficient condition on fixed-point arithmetic and activation functions for universal approximation of quantized networks. Then, we show that various popular activation functions satisfy our sufficient condition, e.g., Sigmoid, ReLU, ELU, SoftPlus, SiLU, Mish, and GELU. In other words, networks using those activation functions are capable of universal approximation. We further show that our necessary condition and sufficient condition coincide under a mild condition on activation functions: e.g., for an activation function $\sigma$, there exists a fixed-point number $x$ such that $\sigma(x)=0$. Namely, we find a necessary and sufficient condition for a large class of activation functions. We lastly show that even quantized networks using binary weights in $\{-1,1\}$ can also universally approximate for practical activation functions.