Boosting

Boosting is an ensemble meta-algorithm in machine learning to primarily minimize bias, as well as variance in supervised learning, and a family of machine learning algorithms that transform weak learners to strong ones.

Boosting, initially named hypothesis boosting consists of the idea of filtering or weighting the data that is used to train team of weak learners, so that each new learner gives more weight or is only trained with observations that have been poorly classified by the previous learners.

Boosting constructs a sequence of trees in a stepwise manner in regression problems and then chooses the ideal tree using an arbitrary differentiable loss function.

Traditionally, constructing an algorithm for machine learning consists of taking a single learner, such as a logistic regressor, a decision tree, a support vector machine, or an artificial neural network, feeding it data, and teaching it through this data to perform a certain task. The ensemble method involves using learners to enhance the performance of any single one of them individually.

Understanding Boosting concept through Example

In this example we are trying to automatically classify the type of call requested by phone customers like collect, Calling Card, Person To Person, etc.

These are sample phone calls log

• yes I’d like to place a collect call long distance please (Collect)
• operator I need to make a call but I need to bill it to my office (Third Number)
• yes I’d like to place a call on my master card please (Calling Card)
• I just called a number in a city and I musta rang the wrong number because I got the wrong party and I would like to have that taken off of my bill (Billing Credit)

Here we can observe easy to find rules of thumb that are often correct e.g. If ‘card’ occurs in utterance then predict ‘Calling Card’ and there are other hard to find single highly accurate prediction rules.

We can follow these steps to analyze further

• select small subset of examples
• derive rough rule of thumb(Decision stump)
• examine 2nd set of examples
• derive 2nd rule of thumb
• repeat several times

Boosting in general is a method of converting rough boosting rules of thumb into highly accurate prediction rules. It can be represented as follows

Advantages of Boosting

• Easy to interpret and handle.
• Prediction capability is efficient.

Disadvantages

• It is sensitive to outliers as every classifier is obliged to fix the errors in the predecessors.
• Hard to scale up.

Types of Boosting

The following are the types of Boosting

• Adaptive Boosting
• Gradient Boosting
• XG Boosting

Conclusion

Boosting is an iterative technique which adjusts the weight of an observation based on the last classification. Hence Boosting is an ensemble model which can be used for classification and regression to enhance the power of prediction.

Originally published at https://www.numpyninja.com on October 12, 2020.