Title: Decision Forests vs. Deep Networks: Conceptual Similarities and Empirical Differences at Small Sample Sizes Show Chinese title

Title: 决策森林与深度网络：小样本情况下的概念相似性和经验差异

Abstract: Deep networks and decision forests (such as random forests and gradient boosted trees) are the leading machine learning methods for structured and tabular data, respectively. Many papers have empirically compared large numbers of classifiers on one or two different domains (e.g., on 100 different tabular data settings). However, a careful conceptual and empirical comparison of these two strategies using the most contemporary best practices has yet to be performed. Conceptually, we illustrate that both can be profitably viewed as "partition and vote" schemes. Specifically, the representation space that they both learn is a partitioning of feature space into a union of convex polytopes. For inference, each decides on the basis of votes from the activated nodes. This formulation allows for a unified basic understanding of the relationship between these methods. Empirically, we compare these two strategies on hundreds of tabular data settings, as well as several vision and auditory settings. Our focus is on datasets with at most 10,000 samples, which represent a large fraction of scientific and biomedical datasets. In general, we found forests to excel at tabular and structured data (vision and audition) with small sample sizes, whereas deep nets performed better on structured data with larger sample sizes. This suggests that further gains in both scenarios may be realized via further combining aspects of forests and networks. We will continue revising this technical report in the coming months with updated results. Show Chinese abstract

Abstract: 深度网络和决策森林（如随机森林和梯度提升树）分别是结构化和表格数据的领先机器学习方法。 许多论文在一种或两种不同的领域（例如，在100种不同的表格数据设置上）对大量分类器进行了经验比较。 然而，使用最新的最佳实践对这两种策略进行仔细的概念和实证比较尚未进行。 概念上，我们说明了这两种方法都可以有益地被视为“划分和投票”方案。 具体来说，它们所学习的表示空间是特征空间的一个划分，由凸多面体的并集组成。 在推理时，每个方法都基于激活节点的投票做出决定。 这种表述允许对这些方法之间的关系有一个统一的基本理解。 在实证方面，我们在数百个表格数据设置以及几个视觉和听觉设置上比较了这两种策略。 我们的重点是最多包含10,000个样本的数据集，这些数据集代表了科学和生物医学数据集的很大一部分。 一般来说，我们发现森林在小样本量的表格和结构化数据（视觉和听觉）中表现优异，而深度网络在大样本量的结构化数据中表现更好。 这表明，通过进一步结合森林和网络的各个方面，可以在两种情况下获得进一步的提升。 我们将继续在接下来的几个月内修订这份技术报告，并更新结果。