标题： 深度可微森林与稀疏注意力用于表格数据 显示英文标题

标题： Deep differentiable forest with sparse attention for the tabular data

摘要： 我们提出了一种深度可微森林的一般架构及其稀疏注意力机制。 可微森林结合了树结构和神经网络的优点。 其结构是一个简单的二叉树，易于使用和理解。 它具有完全的可微性，所有变量都是可学习的参数。 我们将通过基于梯度的优化方法对其进行训练，这种方法在深度CNN的训练中表现出强大的能力。 我们发现并分析了可微森林中的注意力机制。 也就是说，每个决策仅依赖于少数重要的特征，其他特征是无关的。 注意力始终是稀疏的。 基于这一观察，我们通过数据感知初始化来提高其稀疏性。 我们使用属性重要性来初始化注意力权重。 然后，学习到的权重比随机初始化的权重要稀疏得多。 我们在一些大型表格数据集上的实验表明，可微森林的准确性高于GBDT，这是表格数据集的最先进算法。 源代码可在 https://github.com/closest-git/QuantumForest 获取。 显示英文摘要

摘要： We present a general architecture of deep differentiable forest and its sparse attention mechanism. The differentiable forest has the advantages of both trees and neural networks. Its structure is a simple binary tree, easy to use and understand. It has full differentiability and all variables are learnable parameters. We would train it by the gradient-based optimization method, which shows great power in the training of deep CNN. We find and analyze the attention mechanism in the differentiable forest. That is, each decision depends on only a few important features, and others are irrelevant. The attention is always sparse. Based on this observation, we improve its sparsity by data-aware initialization. We use the attribute importance to initialize the attention weight. Then the learned weight is much sparse than that from random initialization. Our experiment on some large tabular dataset shows differentiable forest has higher accuracy than GBDT, which is the state of art algorithm for tabular datasets. The source codes are available at https://github.com/closest-git/QuantumForest