Margin distribution and structural diversity guided ensemble pruning.

Ensemble methods that train and combine multiple learners have always been among the state-of-the-art learning methods, and ensemble pruning aims at generating a smaller-sized ensemble with even better generalization performance. Abundant ensemble pruning methods that use evaluation criteria such as diversity or margin together with validation error have been proposed. However, as these evaluation criteria are used together with the validation error, their effect on generalization performance is less clear. In this paper, we propose a margin distribution and structural diversity guided ensemble pruning framework, called Decoupled Ensemble Pruning (DEP). It decouples the optimization of margin distribution and structural diversity and the optimization of validation error into two stages. Our information-theoretic analysis reveals that the expected generalization gap is related to the combination distribution, i.e., validation error distribution of all the combinations of base learners. And show that optimizing margin mean and structural diversity benefits combination distribution. Concretely, we provide an instantiation of DEP framework in the classic tree-based ensemble pruning setting. Experimental results not only verify the effectiveness in optimizing the distribution, but also show that DEP enjoys better test accuracy than existing ensemble pruning methods. [ABSTRACT FROM AUTHOR]