Title: Neural networks with redundant representation: detecting the undetectable Show Chinese title

Title: 具有冗余表示的神经网络：检测不可检测的

Abstract: We consider a three-layer Sejnowski machine and show that features learnt via contrastive divergence have a dual representation as patterns in a dense associative memory of order P=4. The latter is known to be able to Hebbian-store an amount of patterns scaling as N^{P-1}, where N denotes the number of constituting binary neurons interacting P-wisely. We also prove that, by keeping the dense associative network far from the saturation regime (namely, allowing for a number of patterns scaling only linearly with N, while P>2) such a system is able to perform pattern recognition far below the standard signal-to-noise threshold. In particular, a network with P=4 is able to retrieve information whose intensity is O(1) even in the presence of a noise O(\sqrt{N}) in the large N limit. This striking skill stems from a redundancy representation of patterns -- which is afforded given the (relatively) low-load information storage -- and it contributes to explain the impressive abilities in pattern recognition exhibited by new-generation neural networks. The whole theory is developed rigorously, at the replica symmetric level of approximation, and corroborated by signal-to-noise analysis and Monte Carlo simulations. Show Chinese abstract

Abstract: 我们研究了一个三层的Sejnowski机器，并证明了通过对比散度学习到的特征可以以P=4阶密集关联记忆中的模式形式进行双重表示。 后者已知能够Hebbian存储数量按N^{P-1}规模的模式，其中N表示相互P-wise交互的构成二进制神经元的数量。 我们还证明，通过保持密集关联网络远离饱和状态（即，允许模式数量仅线性增长于N，同时P>2），该系统能够在标准信噪比阈值以下执行模式识别。 特别是，一个P=4的网络即使在存在强度为O(\sqrt{N})的噪声的情况下，在大N极限下也能够检索信息，其强度为O(1)。 这种惊人的能力源于模式的冗余表示——这是在（相对）低负载信息存储条件下提供的——并有助于解释新一代神经网络在模式识别方面令人印象深刻的能力。 整个理论严格地在复制对称近似水平上发展，并通过信噪分析和蒙特卡洛模拟得到证实。