Title: When are Deep Networks really better than Decision Forests at small sample sizes, and how? Show Chinese title

Title: 深度网络在小样本情况下何时真正优于决策森林，以及如何实现？

Abstract: Deep networks and decision forests (such as random forests and gradient boosted trees) are the leading machine learning methods for structured and tabular data, respectively. Many papers have empirically compared large numbers of classifiers on one or two different domains (e.g., on 100 different tabular data settings). However, a careful conceptual and empirical comparison of these two strategies using the most contemporary best practices has yet to be performed. Conceptually, we illustrate that both can be profitably viewed as "partition and vote" schemes. Specifically, the representation space that they both learn is a partitioning of feature space into a union of convex polytopes. For inference, each decides on the basis of votes from the activated nodes. This formulation allows for a unified basic understanding of the relationship between these methods. Empirically, we compare these two strategies on hundreds of tabular data settings, as well as several vision and auditory settings. Our focus is on datasets with at most 10,000 samples, which represent a large fraction of scientific and biomedical datasets. In general, we found forests to excel at tabular and structured data (vision and audition) with small sample sizes, whereas deep nets performed better on structured data with larger sample sizes. This suggests that further gains in both scenarios may be realized via further combining aspects of forests and networks. We will continue revising this technical report in the coming months with updated results. Show Chinese abstract

Abstract: 深度网络和决策森林（如随机森林和梯度提升树）分别是结构化和表格数据的领先机器学习方法。 许多论文在一种或两种不同的领域（例如，在100种不同的表格数据设置上）对大量分类器进行了经验比较。 然而，使用最新的最佳实践对这两种策略进行仔细的概念和实证比较尚未进行。 概念上，我们说明了这两种方法都可以有益地视为“划分和投票”方案。 具体来说，它们学习的表示空间是特征空间的一个划分，由凸多面体的并集组成。 在推理时，每个方法都基于激活节点的投票来决定。 这种表述使得对这些方法之间关系的基本统一理解成为可能。 实证上，我们在数百个表格数据设置以及几个视觉和听觉设置上比较了这两种策略。 我们的重点是样本数最多为10,000的数据集，这些数据集代表了科学和生物医学数据集的很大一部分。 一般来说，我们发现森林在小样本量的表格和结构化数据（视觉和听觉）上表现优异，而深度网络在大样本量的结构化数据上表现更好。 这表明，通过进一步结合森林和网络的各个方面，两种情况下都可能获得进一步的提升。 我们将继续在接下来的几个月内对本技术报告进行修订，并更新结果。