标题： 学习有序表示的嵌套丢弃法 显示英文标题

标题： Learning Ordered Representations with Nested Dropout

摘要： 本文研究了数据的有序表示形式，其中不同维度具有不同程度的重要性。为了学习这些表示形式，我们引入了一种称为嵌套丢弃（nested dropout）的方法，这是一种从神经网络中随机移除一致且嵌套的隐藏单元集合的过程。首先，我们在半线性自编码器的简单情况下呈现了一系列理论结果。我们严格证明了应用嵌套丢弃可以强制隐藏单元的可辨识性，这与主成分分析（PCA）形成了精确的等价关系。然后，我们将该算法扩展到深度模型，并展示了有序表示在多种应用中的相关性。具体而言，我们利用学到的代码的有序特性构建基于哈希的数据结构，允许非常快速的检索，检索时间与数据库大小成对数关系且不受表示维度的影响。这使得数百倍于目前可行长度的代码成为可能，同时避免了短码导致的质量下降问题，检索速度仍能与现有方法竞争。此外，我们还表明，有序表示是一种有前景的方式，可用于高效在线数据重建的自适应压缩学习。 显示英文摘要

摘要： In this paper, we study ordered representations of data in which different dimensions have different degrees of importance. To learn these representations we introduce nested dropout, a procedure for stochastically removing coherent nested sets of hidden units in a neural network. We first present a sequence of theoretical results in the simple case of a semi-linear autoencoder. We rigorously show that the application of nested dropout enforces identifiability of the units, which leads to an exact equivalence with PCA. We then extend the algorithm to deep models and demonstrate the relevance of ordered representations to a number of applications. Specifically, we use the ordered property of the learned codes to construct hash-based data structures that permit very fast retrieval, achieving retrieval in time logarithmic in the database size and independent of the dimensionality of the representation. This allows codes that are hundreds of times longer than currently feasible for retrieval. We therefore avoid the diminished quality associated with short codes, while still performing retrieval that is competitive in speed with existing methods. We also show that ordered representations are a promising way to learn adaptive compression for efficient online data reconstruction.