Ensemble Models Demystified

Deep Learning is all the rage, but ensemble models are still in the game. With libraries such as the recent and performant LightGBM, the Kaggle superstar XGboost or the classic Random Forest from scikit-learn, ensembles models are a must-have in a data scientist’s toolbox. They’ve been proven to provide good performance on a wide range of problems, and are usually simpler to tune and interpret.

This talk focuses on two of the most popular tree-based ensemble models. You will learn about Random Forest and Gradient Boosting, relying respectively on bagging and boosting. This talk will attempt to build a bridge between the theory of ensemble models and their implementation in Python.

Transcript

1. Ensemble Models Demystified @KevinLemagnen
2. Ensembles: Why do we care? • Good performance • General purpose • Usually easier to train than other fancy techniques • Really popular in industry and ML competitions
3. Why this talk? Theory
4. Why this talk? Theory Implementation
5. Why this talk? Theory Implementation
6. Agenda 1. Intuition 2. Weak learner (Decision Tree) 3. Bagging (Random Forest) 4. Boosting (Gradient Boosting) 5. Other boosting libraries
7. 1. Intuition
8. What are ensemble models? • Combining multiple simple models (weak learners) into a larger one (ensemble) • Two popular techniques: ◦ Bagging ◦ Boosting ◦ Usually with decision trees as weak learner
9. Intuition Accuracy = 60% Accuracy = 75% Both say you have X… what's the likelihood that you really have X?
10. Two big assumptions • Weak Learner: “Experts” need to be more right than wrong on average
11. Two big assumptions • Weak Learner: “Experts” need to be more right than wrong on average • Diversity: “Experts” need to make different errors
12. 2. Weak Learners Decision Trees
13. Why are they good? • Can capture complex relationships in the data ◦ We’ll often be able to get our > 50% accuracy! • Overfits easily ◦ We can use that to create diverse models!
14. How do we control them? No constraints = one leaf per sample = massive overfitting Some good constraints: • Pick a maximum depth • Pick a minimum number of samples needed in a new node/leaf
15. 3. Bagging Random Forest
16. How do we build diverse trees? • Each one is trained on a subsample of observations [Bootstrapping]
17. How do we build diverse trees? • Each one is trained on a subsample of observations [Bootstrapping] • Each one is trained on a subsample of features
18. How do we build diverse trees? • Each one is trained on a subsample of observations [Bootstrapping] • Each one is trained on a subsample of features • Loosen your constraints to let your trees overfit
19. How do we build diverse trees? • Each one is trained on a subsample of observations [Bootstrapping] • Each one is trained on a subsample of features • Loosen your constraints to let your trees overfit Don’t overdo it… We still need: • Good performance per tree (no underfitting)
20. How do we build diverse trees? • Each one is trained on a subsample of observations [Bootstrapping] • Each one is trained on a subsample of features • Loosen your constraints to let your trees overfit Don’t overdo it… We still need: • Good performance per tree (no underfitting) • Able to generalise (no overfitting)
21. Bagging - Random Forest
22. Some pros and cons + Easy to run in parallel
23. Some pros and cons + Easy to run in parallel + Decision Trees = we can get feature importance
24. Some pros and cons + Easy to run in parallel + Decision Trees = we can get feature importance − Models remain correlated (similar data)
25. Some pros and cons + Easy to run in parallel + Decision Trees = we can get feature importance − Models remain correlated (similar data) − Hard to interpret
26. Some pros and cons + Easy to run in parallel + Decision Trees = we can get feature importance − Models remain correlated (similar data) − Hard to interpret ? Outliers likely to be ignored by most weak learners
27. 4. Boosting Gradient Boosting
28. Boosting - Intuition We want to build weak learners that actively compensate each others’ errors.
29. Boosting - Intuition We want to build weak learners that actively compensate each others’ errors. Let’s focus on one sample: (X, y) with y = 100
30. Boosting - Intuition We want to build weak learners that actively compensate each others’ errors. Let’s focus on one sample: (X, y) with y = 100 1. Train DT1 on (X, y) DT1(X) = 95
31. Boosting - Intuition We want to build weak learners that actively compensate each others’ errors. Let’s focus on one sample: (X, y) with y = 100 1. Train DT1 on (X, y) DT1(X) = 95 2. Compute residuals r = 100 - 95 = 5
32. Boosting - Intuition We want to build weak learners that actively compensate each others’ errors. Let’s focus on one sample: (X, y) with y = 100 1. Train DT1 on (X, y) DT1(X) = 95 2. Compute residuals r = 100 - 95 = 5 3. Train DT2 on (X, 5) DT2(X) = 4
33. Boosting - Intuition We want to build weak learners that actively compensate each others’ errors. Let’s focus on one sample: (X, y) with y = 100 1. Train DT1 on (X, y) DT1(X) = 95 2. Compute residuals r = 100 - 95 = 5 3. Train DT2 on (X, 5) DT2(X) = 4 4. Aggregate DT1 and DT2 DT1(X) + DT2(X) = 95 + 4 = 99
34. Boosting - Intuition We want to build weak learners that actively compensate each others’ errors. Let’s focus on one sample: (X, y) with y = 100 1. Train DT1 on (X, y) DT1(X) = 95 2. Compute residuals r = 100 - 95 = 5 3. Train DT2 on (X, 5) DT2(X) = 4 4. Aggregate DT1 and DT2 DT1(X) + DT2(X) = 95 + 4 = 99 5. Repeat
35. Boosting - Intuition We want to build weak learners that actively compensate each others’ errors. Let’s focus on one sample: (X, y) with y = 100 1. Train DT1 on (X, y) DT1(X) = 95 2. Compute residuals r = 100 - 95 = 5 3. Train DT2 on (X, 5) DT2(X) = 4 4. Aggregate DT1 and DT2 DT1(X) + DT2(X) = 95 + 4 = 99 5. Repeat Weak learners increasingly focus on "hard points"
36. Boosting - Gradient Boosting too many stages OR too complex trees = overfit to noise
37. Boosting - Gradient Boosting too many stages OR too complex trees = overfit to noise • Getting the number of stages right is extremely important
38. Boosting - Gradient Boosting too many stages OR too complex trees = overfit to noise • Getting the number of stages right is extremely important • We need to build small, constrained trees
39. Boosting - Gradient Boosting
40. Some pros and cons + Great performance (usually)
41. Some pros and cons + Great performance (usually) + Decision Trees = we can get feature importance
42. Some pros and cons + Great performance (usually) + Decision Trees = we can get feature importance − Hard to run in parallel
43. Some pros and cons + Great performance (usually) + Decision Trees = we can get feature importance − Hard to run in parallel − Hard to interpret
44. Some pros and cons + Great performance (usually) + Decision Trees = we can get feature importance − Hard to run in parallel − Hard to interpret − Can easily overfit
45. Any questions?