Additive Ensemble Neural Networks

Deep neural networks (DNNs) have been making progress in many ways. DNNs are typically used to model complex nonlinearity of high-dimensional data in regression or classification problems. As DNNs contain additional hidden layers, they generally improve performance but increase the number of parameters to train, thereby extending the learning time. Many studies, such as those employing Dropout and regularization methods, have undertaken to solve these problems. The method proposed in this paper is an additive ensemble neural networks (AENNs), by which a boosting mechanism of an ensemble methodology is applied to the neural networks instead of regularization techniques. That is, the model by AENNs is obtained by sequentially combining several simple shallow network models. Experiments showed that AENNs yield better results than conventional DNNs and machine learning methods for regression and classification problems, thereby alleviating the troublesome model complexity issue.