Boosting

Boosting is a way of combining a set of weak classifiers, known as ensembles, to make a much stronger classifier. This is accomplished through an iterative process where new weak learners are combined to further reduce model residual. It in effect creates a “committee” of models, using weighted results from each model.

The basic precept of boosting is the following:

If we have a weak classifier that performs even slightly better than random, we can combine a series of weak classifiers into a strong classifier.

While there are many options of boosting for both regression and classification, Figure 2-4 illustrates the operation of one of the most common boosting algorithms, known as AdaBoost (from “adaptive boosting” and pronounced accordingly [2]). At each stage m we estimate our classifier using a set of weights for the n data points. From these we develop a set of weights for the prediction of each classifier. The final prediction is based on a weighted sum of outputs from each classifier using the as weights.

Figure 2-4 Illustration of AdaBoost

The AdaBoost algorithm works as follows. We start out with a set of data of independent variables and a set of target values (i.e., classifications)

AdaBoost algorithm

1. Initialize the weighting coefficients by setting for

2. For :

   • Fit a classifier to the training data by minimizing the weighted error function

     

     where is an indicator function that equals 1 when and 0 otherwise.

   • Evaluate the quantities

   • and then set the weights

     

   • Update the data weighting coefficients:

     

3. Make predictions using the final model given by

   

Boosting has the remarkable effect that it seems relatively impervious to overfitting.

A full description of boosting, and its applications in a number of areas, can be found in the book by the developers of the method [3].