https://itakigawa.github.io/ A Modern Introduction to Decision Tree Ensembles Ichigaku Takigawa Institute for Liberal Arts and Sciences, Kyoto University Institute for Chemical Reaction Design and Discovery, Hokkaido University RIKEN Center for Advanced Intelligence Project Part 1

About Me (Ichi Takigawa) • From Kyoto University, Japan • Research Interests: Machine Learning & Machine Discovery Algorithms • Machine Learning with Discrete Structures • Machine Discovery for Chemistry & Life Sciences • 34 years ago when I was a kid, I lived in Düsseldorf for 1 year https://itakigawa.github.io/

Outline We will jointly learn decision trees and ensemble learning! 1. Why we should learn decision trees and ensemble learning? 2. History of Decision Trees 3. Basics of Decision Trees 4. Ensemble Learning: Additive ensembles and decision forests 5. Random forest and Regularized gradient boosted trees 6. Practical data analysis with decision tree ensembles Part 1 Part 2

Tree a forest = a set of trees Forest root leaf node pruning depth height Stump Quick note to complete beginners Technical Terms in Computer Science!

Quick note to complete beginners We will get “decision tree ensembles” by integrating multiple “decision trees” in a certain way called “ensemble learning”. “ensemble learning” “decision trees” • CART • C4.5 • Variations (M5, RETIS, OC1, MARS, GOSDT) “decision tree ensembles (decision forests)” • Random Forest • ExtraTrees (Extremely randomized trees) • Gradient boosted trees • XGBoost, LightGBM

Outline We will jointly learn decision trees and ensemble learning! 1. Why we should learn decision trees and ensemble learning? 2. History of Decision Trees 3. Basics of Decision Trees 4. Ensemble Learning: Additive ensembles and decision forests 5. Random forest and Regularized gradient boosted trees 6. Practical data analysis with decision tree ensembles Part 1 Part 2

Trees as representations Understand by “classification” = Simple and intuitive idea for us! https://en.wikipedia.org/ wiki/Tree_(data_structure) 2 7 5 2 10 6 5 11 9 4 https://en.wikipedia.org/ wiki/Tree_of_life_(Kabbalah) https://en.wikipedia.org/ wiki/Tree_of_life_(biology) z 10 z 1 z 2 z 3 z 4 z 5 z 6 z 7 z 8 z 9 w 7 w 8 w 9 w 10 u5 u 6 u7 u8 t 2 t 3 m 1 m 2 m 3 m 4 m 5 m 6 m 7 m 8 m 9 E10 F10 m 10 s 2 i2 i3 k5 k6 k7 k8 l7 l8 a1 a2 a3 a4 a5 a6 a7 a8 a9 f 6 f 7 f 8 f 9 a10 f 10 A B C D E F G H I K L W.West lith. Hatton Garden I II III IV V VI VII VIII IX X XI XII XIII XIV a14 q14 p14 b14 f 14 o14 e 14 m 14 F 14 n14 r 14 w 14 y 14 v 14 z 14 Tree structure (Data structure) Tree of Life, Phylogenetic trees, Family trees • Folders (File systems) • DOM (XML/HTML) • Class inheritance • Syntax tree • Search tree • Library classification • Probabilistic models :

Why we should learn decision tree ensembles? The obsolete method, "decision trees,” which began its research around the 1960s-70s and gained immense popularity in the 1980s-90s, has made a comeback in the modern era through "ensemble learning"! > ≤ ≤ > Classification trees Regression trees

One of three major weapons in data science Decision tree ensemble is one of the three key weapons in modern data science alongside linear models and deep learning. https://www.kaggle.com/kaggle-survey-2021 State of Data Science and Machine Learning 2021 (Kaggle Survey) Q17. Which of the following ML algorithms do you use on a regular basis? Linear or Logistic Regression Decision Trees or Random Forests Gradient Boosting Machines (xgboost, lightgbm, etc) Convolutional Neural Networks Dense Neural Networks (MLPs, etc) Bayesian Approaches Recurrent Neural Networks Transformer Networks (BERT, gpt-3, etc) None or Other Generative Adversarial Networks Evolutionary Approaches 0 3,500 7,000 10,500 14,000 13,852 11,863 7,566 7,410 4,468 4,392 4,228 2,273 1,953 1,353 963 # respondents: 25,973 Linear Models Decision Tree Ensembles Deep Learning

Reason #1: Random Forest as post SVM Random forest showed better performance than SVM, the state of the art at that time, on various benchmarks particularly for high-dimensional data Do we need hundreds of classifiers to solve real world classification problems? (JMLR)
 https://jmlr.org/papers/v15/delgado14a.html An empirical evaluation of supervised learning in high dimensions (ICML 2008)
 https://doi.org/10.1145/1390156.1390169 Report #1: 11 datasets × 10 methods (Caruana+, 2008) Report #2: 121 datasets × 179 methods (Fernández-Delgado+, 2014) # samples: 5.5K - 279.2K # features: 761 - 685K # samples: 16 - 130K # features: 3 - 262 Very high-dimensional data Small-to-middle scale, but extremely diverse types of data

Reason #2: Popularity in ML competitions In ML competitions like Kaggle, it has been widely recognized that decision tree ensembles work pretty well especially for tabular data. Well, we have too many ML algorithms, but what could be the “best method”? https://www.kaggle.com/ • Kaggle is a data science competition platform and online community of data scientists and machine learning practitioners under Google LLC. • It’s also known for its substantial prize money for winners (from $10,000 to $1,000,000) All of the few masters who have ever been ranked No.1 in the world named the same ML method as their “most favorite”!

Reason #2: Popularity in ML competitions Gradient Boosting Machines are the best! Before I knew of GBM, I was a big fan of neural networks.[2] It depends on the problem, but if I have to pick one, then it is GBM (its XGBoost flavor).[1] I like Gradient Boosting and Tree methods in general.[3] Gradient boosted trees by far! [4] [1] https://web.archive.org/web/20190925162927/http://blog.kaggle.com/2015/06/22/profiling-top-kagglers-owen-zhang-currently-1-in-the-world/ [2] https://web.archive.org/web/20190914031256/http://blog.kaggle.com/2015/11/09/profiling-top-kagglers-gilberto-titericz-new-1-in-the-world/ [3] https://web.archive.org/web/20191107132809/http://blog.kaggle.com/2016/02/10/profiling-top-kagglers-kazanova-new-1-in-the-world/ [4] https://web.archive.org/web/20190913005952/http://blog.kaggle.com/2016/02/22/profiling-top-kagglers-leustagos-current-7-highest-1/ That is “Gradient boosted (decision) trees” or GBDT! Aliases: Gradient boosting, GBM, MART, TreeNet, AnyBoost, …

Reason #3: Research outcomes from Microsoft • Real-time Pose Recognition of Kinect (5ms per frame for Xbox360 GPU) • Decision Jungles (NIPS2013), LightGBM (NIPS 2017) • LambdaMART won Yahoo! Learning to Rank Challenge (ICML WS 2010) https://www.microsoft.com/en-us/research/project/ human-pose-estimation-for-kinect/publications/ http://research.microsoft.com/ en-us/projects/decisionforests/ Human Pose Estimation for Kinect Decision Forests https://doi.org/ 10.1007/978-1-4471-4929-3 https://doi.org/10.1109/ CVPR.2011.5995316

Why we should learn decision tree ensembles? • What is a decision tree? (History and basic algorithm) • Why did it become so popular once in the past? (Advantages) Decision tree ensemble is one of the three key weapons in modern data science alongside linear models and deep learning. • Despite this, why did it fall out of favor? (Disadvantages) • What is ensemble learning? Why can it compensate for the shortcomings? • Practice and methods of decision tree ensembles [Part 1] [Part 2]

A visual introduction to machine learning https://www.r2d3.us/visual-intro-to-machine-learning-part-1/

A quick visual intro to decision trees Predict whether each property is in San Francisco or in New York. https://github.com/jadeyee/r2d3-part-1-data/blob/master/part_1_data.csv in_sf beds bath price year_built sqft price_per_sqft elevation Input variables (7) Output variables (1) Supervised Learning in_sf beds bath price year_built sqft price_per_sqft elevation 1 1 1 865,000 1,993 960 901 17 1 4 2.5 995,000 1,915 2,180 456 62 0 0 1 319,000 1,941 500 638 10 0 1 1 1,135,000 2,005 715 1,587 10 0 2 1 545,000 1,939 1,049 520 39 1 2 1.3 1,097,000 1,904 1,493 735 14 0 0.5 1 384,900 1,962 540 713 10 0 5 5 27,500,000 1,930 7,500 3,667 21 1 3 2 1,650,000 1,922 2,025 815 106 0: NY 1: SF A tabular data i.e. Excel-like spreadsheet

A quick visual intro to decision trees sf: 175 ny: 224 total: 399 sf: 93 ny: 0 total: 93 73 > ≤ (224 + 93)/492 = 0.6443 Accuracy 64.43% sf: 268 ny: 224 total: 492 SF would have a higher elevation. Let’s take the elevation variable. meters

A quick visual intro to decision trees sf: 70 ny: 187 total: 257 sf: 198 ny: 37 total: 246 20 > ≤ (187 + 198)/492 = 0.7825 Accuracy 78.25% sf: 268 ny: 224 total: 492 SF would have a higher elevation. Let’s take the elevation variable. meters

A quick visual intro to decision trees sf: 268 ny: 224 total: 492 SF would have a higher elevation. Let’s take the elevation variable. meters

A quick visual intro to decision trees sf: 85 ny: 215 total: 300 sf: 183 ny: 9 total: 192 30.5 > ≤ (215 + 183)/492 = 0.8089 Accuracy 80.89% sf: 268 ny: 224 total: 492 SF would have a higher elevation. Let’s take the elevation variable. meters

A quick visual intro to decision trees ≤ > (215 + 183)/492 = 0.8089 Accuracy 80.89% 183 9 215 85 Split all cases (492) into two groups (300 and 192 cases)! Here we have the simplest decision tree! (with a single split) “Decision Stump”

A quick visual intro to decision trees in_sf beds bath price year_ built sqft price_per _sqft eleva tion 1 1 1 865,000 1,993 960 901 17 1 4 2.5 995,000 1,915 2,180 456 62 0 0 1 319,000 1,941 500 638 10 0 1 1 1,135,000 2,005 715 1,587 10 0 2 1 545,000 1,939 1,049 520 39 1 2 1.3 1,097,000 1,904 1,493 735 14 in_sf beds bath price year_ built sqft price_per _sqft eleva tion 1 1 1 865,000 1,993 960 901 17 0 0 1 319,000 1,941 500 638 10 0 1 1 1,135,000 2,005 715 1,587 10 1 2 1.3 1,097,000 1,904 1,493 735 14 0 0.5 1 384,900 1,962 540 713 10 0 5 5 27,500,000 1,930 7,500 3,667 21 in_sf beds bath price year_ built sqft price_per _sqft eleva tion 1 4 2.5 995,000 1,915 2,180 456 62 0 2 1 545,000 1,939 1,049 520 39 1 3 2 1,650,000 1,922 2,025 815 106 1 1 1 699,000 1,907 932 750 59 1 3 2 1,195,000 1,907 1,396 856 33 1 4 3 1,895,000 2,001 2,041 928 61 elevation > 30.5 > ≤ True False 492 cases 300 cases 192 cases Variable selection Select a single variable Binary partitioning Split the data into two groups Split all cases (492) into two groups (300 and 192 cases)!

A quick visual intro to decision trees in_sf beds bath price year_ built sqft price_per _sqft eleva tion 1 1 1 865,000 1,993 960 901 17 0 0 1 319,000 1,941 500 638 10 0 1 1 1,135,000 2,005 715 1,587 10 1 2 1.3 1,097,000 1,904 1,493 735 14 0 0.5 1 384,900 1,962 540 713 10 0 5 5 27,500,000 1,930 7,500 3,667 21 in_sf beds bath price year_ built sqft price_per _sqft eleva tion 1 4 2.5 995,000 1,915 2,180 456 62 0 2 1 545,000 1,939 1,049 520 39 1 3 2 1,650,000 1,922 2,025 815 106 1 1 1 699,000 1,907 932 750 59 1 3 2 1,195,000 1,907 1,396 856 33 1 4 3 1,895,000 2,001 2,041 928 61 300 cases 192 cases ≤ > Do the same thing again! (recursion) 54 63 161 22 $

A quick visual intro to decision trees ≤ > 183 9 54 63 161 22 (63 + 161 + 183)/492 = 0.8272 Accuracy 82.72% A decision tree with 3 decision nodes!

http://www.r2d3.us/visual-intro- to-machine-learning-part-1/ A visual introduction to machine learning Repeat the recursion until we get the desired accuracy! A quick visual intro to decision trees

A quick visual intro to decision trees ≤ > 183 9 54 63 161 22 if elevation > 30.5 return SF else if price_per_sqft > 1072.5 return NY else return SF A record of any property Prediction SF or NY A decision tree = nested 'if' statements

A quick visual intro to decision trees http://www.r2d3.us/visual-intro- to-machine-learning-part-1/ A visual introduction to machine learning Just check the True/False conditions of the 'if' statements

Outline We will jointly learn decision trees and ensemble learning! 1. Why we should learn decision trees and ensemble learning? 2. History of Decision Trees 3. Basics of Decision Trees 4. Ensemble Learning: Additive ensembles and decision forests 5. Random forest and Regularized gradient boosted trees 6. Practical data analysis with decision tree ensembles Part 1 Part 2

Brief history of decision trees 1) Concept learning 2) Interaction detection 3) Nonlinear regression CLS (Hunt+ 1966) ID3 (Quinlan 1986) C4.5 (Quinlan 1993) C5.0/See5 (Quinlan) AID (Morgan & Sonquist 1963) Breiman & Meisel 1976 Friedman 1979 THAID (Messenger & Mandell, 1972) CHAID (Kass 1980) CART (Breiman+ 1984) M5 (Quinlan 1992) M5’ (Wang & Witten 1997) MARS (Friedman 1991) Cubist (Quinlan) RETIS (Karalic & Cestnik 1991) HTL (Torgo 1997) QUEST (Loh & Shih 1997) CRUISE (Kim & Loh, 2001) OC1 (Murthy+ 1993) Quinlan 1979 Tree-shaped classification Tree-shaped regression Decision tree = A concept so simple that anyone could think of it In fact, this idea was explored in various contexts in the 1970s

Learning of concepts and rules Decision trees are equivalent to If-then rules and have a close relationship with logic and logical circuits Decision Tree https://en.wikipedia.org/wiki/Decision_tree Decision Tables https://en.wikipedia.org/wiki/Decision_table Decision List https://en.wikipedia.org/wiki/Decision_list Decision Diagram https://en.wikipedia.org/wiki/Binary_decision_diagram if elevation > 30.5 return SF else if price_per_sqft > 1072.5 return NY else return SF A record of any property Prediction SF or NY

Learning of concepts and rules In particular, the inductive reasoning problem called concept learning can be regarded as a noiseless version of today's supervised learning Swim Has ns Fly Has lungs Is a sh Herring yes yes no no ✔ Cat no no no yes ✘ Pigeon no no yes yes ✘ Flying Fish yes yes yes no ✔ Otter yes no no yes ✘ Cod yes yes no no ✔ Whale yes yes no yes ✘ It assumes learning of logical expressions and hence variables are only boolean values By applying decision tree learning, one can obtain a consistent 'logical expression (If-then rule)' for the given examples! For example, NOT has lungs Is a sh Concept learning: Creating a description of the concept 'fish' from examples yes/no true/false

Example: C5.0 for iris dataset The “Iris” dataset (3 classes, 4 variables, 150 samples) https://en.wikipedia.org/wiki/Iris_flower_data_set Target Variables Classification problem of 3 species setosa versicolor virginica Variables Petal Petal Sepal Sepal • Petal (Length, Width) • Sepal (Length, Width) The rules are not clear just by seeing many numbers…

Example: C5.0 for iris dataset Let’s see what we get when we apply decision trees. (C5.0 in R) 100 samples for training 50 samples for test load the “iris” dataset inspect it C5.0 [Release 2.07 GPL Edition] Rules: Rule 1: (36, lift 2.7) Petal.Length <= 1.9 -> class setosa [0.974] Rule 2: (34/3, lift 2.9) Petal.Length > 1.9 Petal.Width <= 1.7 -> class versicolor [0.889] Rule 3: (30, lift 2.9) Petal.Width > 1.7 -> class virginica [0.969]

Example: C5.0 for iris dataset Rules that humans can operate: they give a sense of “understanding” If petal is short, it s setosa If petal is wide, it s virginica If petal is long but narrow, it s versicolor ≤ 1.9 cm > 1.9 cm ≤ 1.7 cm > 1.7 cm https://www.kaggle.com/code/necibecan/iris-dataset-eda-n (Plus, we can ignore sepal) 97.4% correct 88.9% correct 96.9% correct Rule 1: (36, lift 2.7) Petal.Length <= 1.9 -> class setosa [0.974] Rule 2: (34/3, lift 2.9) Petal.Length > 1.9 Petal.Width <= 1.7 -> class versicolor [0.889] Rule 3: (30, lift 2.9) Petal.Width > 1.7 -> class virginica [0.969] setosa versicolor virginica

Example: C5.0 for iris dataset When applied to unseen data in training, “predictions” can be obtained Petal.Length 1 ≤ 1.9 > 1.9 Node 2 (n = 36) setosa virginica 0 0.2 0.4 0.6 0.8 1 Petal.Width 3 ≤ 1.7 > 1.7 Node 4 (n = 34) setosa virginica 0 0.2 0.4 0.6 0.8 1 Node 5 (n = 30) setosa virginica 0 0.2 0.4 0.6 0.8 1 1 2 3 4 5 6 0.5 1.0 1.5 2.0 2.5 Petal.Length Petal.Width setosa versicolor virginica Decision tree: Petal.Length <= 1.9: setosa (36) Petal.Length > 1.9: :...Petal.Width > 1.7: versicolor (34/3) Petal.Width <= 1.7: virginica (30)

Example: scikit-learn (Python) Text text !pip install -q dtreeviz !pip install -q svglib from sklearn import datasets from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score from sklearn.tree import DecisionTreeClassifier, export_text from dtreeviz.trees import dtreeviz iris = datasets.load_iris() X_train, X_test, y_train, y_test = train_test_split( iris.data, iris.target, test_size=0.33) model = DecisionTreeClassifier(ccp_alpha=0.2) model.fit(X_train, y_train) y_pred = model.predict(X_test) print(100*accuracy_score(y_test, y_pred), '%') 98 % model.classes_ = iris.target_names print(export_text(model, feature_names=iris.feature_names)) |--- petal length (cm) <= 2.45 | |--- class: setosa |--- petal length (cm) > 2.45 | |--- petal width (cm) <= 1.75 | | |--- class: versicolor | |--- petal width (cm) > 1.75 | | |--- class: virginica model.classes_ = range(3) viz = dtreeviz(model, X_train, y_train, class_names=list(iris.target_names), feature_names=list(iris.feature_names), scale=2.0) display(viz) from svglib.svglib import svg2rlg from reportlab.graphics import renderPDF viz.save("treeout.svg") renderPDF.drawToFile(svg2rlg("treeout.svg"), "treeout.pdf") > ≤

Learning of concepts and rules AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGZiRVNvh8rijIXqiIrkThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOslho= X2  ✓1 yes no AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGRY0WT78eKosyFqswokThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOelho= X1  ✓2 yes no Blue AAAClnichVHLSsNAFD2N7/qquim4KZaKqzItRcWFiCK6rNZqoZWQxGkbTJOYTCta/AF/wIW4UFARP8APcOMPuOgniMsKblx4mwZExXrDZM6cuefOmbmqbeiuYKwRkLq6e3r7+geCg0PDI6OhsfFt16o6Gs9qlmE5OVVxuaGbPCt0YfCc7XClohp8R91fae3v1Ljj6pa5JY5svltRSqZe1DVFECWHxnJyMlIw+EGkIMpcKHJKDkVZnHkR+Q0SPojCj7QVekABe7CgoYoKOEwIwgYUuPTlkQCDTdwu6sQ5hHRvn+MEQdJWKYtThkLsPv1LtMr7rEnrVk3XU2t0ikHDIWUEMfbM7liTPbF79sI+/qxV92q0vBzRrLa13JZHT8OZ939VFZoFyl+qjp4Fipj3vOrk3faY1i20tr52fNbMLGzG6tPsir2S/0vWYI90A7P2pl1v8M3zDn5U8kIvRg1K/GzHb7CdjCdm46mNVHRp2W9VPyYxhRnqxxyWsI40slT/EBe4wa0UlhalVWmtnSoFfM0EvoWU/gQ6DJYd X2  ✓4 yes no Red AAAClnichVHLSsNAFD3GV62vVjeCm2JRXJWJ+MKFFEV0WVurBSshiVMNTZOYTCta/AF/wIW4UFARP8APcOMPuOgniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGGu0SK1t7R2dka5od09vX38sPrDu2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvaOVFf3+jxl3PsK01ceDwrYq6YxklQ1cFUUosXlDkRNHke4mi2OVCVaaUWJKlWBCJn0AOQRJhZOzYHYrYhg0dVVTAYUEQNqHCo28TMhgc4rZQJ84lZAT7HEeIkrZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHaX/eCj39q+qQrPA7qfqT88CJcwGXg3y7gSMfwu9qa8dnrzk5rKj9TF2wZ7J/zlrsHu6gVV71S9Xefb0Dz8aeaEXowbJ39vxE6xPpOTp1OTqZDK9ELYqgmGMYJz6MYM0VpBBnurv4wxXuJaGpHlpSVpupkotoWYQX0LKfAA5/pYd X1  ✓5 yes no Red AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4= X1  ✓6 yes no Red Blue AAAClnichVHLSsNAFD3GV3221Y3gplgUV2WiRcWFFEV0Wa3VgpWQxKmGpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxKWCGxfepAFRUW+YzJkz99w5M1dzTMMTjDVapNa29o7OSFd3T29ffzQWH1j37Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tPKCv79R465n2NaaOHD4VkXdsYySoauCKCUWLyhyomjyvURR7HKhKpNKLMlSLIjETyCHIIkwsnbsDkVsw4aOKirgsCAIm1Dh0bcJGQwOcVuoE+cSMoJ9jiN0k7ZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHSV6PJR7+1dVoVlg91P1p2eBEmYCrwZ5dwLGv4Xe1NcOT15ys6uj9TF2wZ7J/zlrsHu6gVV71S9X+OrpH3408kIvRg2Sv7fjJ1ifSMlTqfRKOpmZD1sVwTBGME79mEYGy8giT/X3cYYrXEtD0py0KC01U6WWUDOILyFlPwA1vpYb X1  ✓3 yes no Red AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykWHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm4/JcPJFJxFJLQav6MIFJzFA/kkhhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bz4+lh8= X1  ✓7 yes no Blue Blue if x2 ≤ θ1 then if x1 ≤ θ2 then return Blue else if x2 ≤ θ4 then return Red else if x1 ≤ θ5 then return Red else if x1 ≤ θ6 then return Blue else return Red else if x1 ≤ θ3 then if x2 ≤ θ7 then return Blue else return Blue else return Red Decision tree Logical rules

For your information Knowledge representation as logical expressions was at the heart of symbolic and knowledge-based machine learning in the 90s • Programming Language Research • Logical Programming • Constraint Programming • Inductive Logic Programming (ILP) • Explanation-based Learning • Predicate Logic Learning • Propositional Logic Learning • Analogy • Inductive Reasoning • Grammar Inference (Inductive Language Learning) • Computational Learning • Gold's Limit Identification ('67) • Plotkin's Least General Generalization ('70) • Angluin's Inductive Inference ('83) • Angluin's Query Learning ('88) • Shapiro's Model Inference ('81) • Valiant's PAC Learning ('84) → Decision tree learning was a research topic that intersected with knowledge representation and logical reasoning. (a typical GOFAI = Good Old-Fashioned AI)

Brief history of decision trees 1) Concept learning 2) Interaction detection 3) Nonlinear regression CLS (Hunt+ 1966) ID3 (Quinlan 1986) C4.5 (Quinlan 1993) C5.0/See5 (Quinlan) AID (Morgan & Sonquist 1963) Breiman & Meisel 1976 Friedman 1979 THAID (Messenger & Mandell, 1972) CHAID (Kass 1980) CART (Breiman+ 1984) M5 (Quinlan 1992) M5’ (Wang & Witten 1997) MARS (Friedman 1991) Cubist (Quinlan) RETIS (Karalic & Cestnik 1991) HTL (Torgo 1997) QUEST (Loh & Shih 1997) CRUISE (Kim & Loh, 2001) OC1 (Murthy+ 1993) Quinlan 1979 Tree-shaped classification Tree-shaped regression Decision tree = A concept so simple that anyone could think of it In fact, this idea was explored in various contexts in the 1970s

Interaction detection in multivariate analysis An early idea toward the decision tree: AID = Automatic Interaction Detection Sonquist & Morgan, 1964 Morgan & Sonquist, 1963 https://doi.org/10.2307/2283276 https://www.amazon.com/dp/B0006BMGNW

Interaction detection in multivariate analysis Sonquist & Morgan, 1964 Morgan & Sonquist, 1963 https://doi.org/10.2307/2283276 https://www.amazon.com/dp/B0006BMGNW THAID (Messenger & Mandell, 1972) CHAID (Kass, 1975, 1980) https://doi.org/10.1080/01621459.1972.10481290 https://doi.org/10.2307/2346565 An early idea toward the decision tree: AID = Automatic Interaction Detection

Interaction detection in multivariate analysis Independence of variables in a linear model ("independent" variables) AAACA3icbVC7SgNBFL0bXzG+opY2g0GITdiVoDZC0MYygnlAsoTZyWwyZGZ3nZkV1iWl32CrtZ3Y+iGW/omTZAuTeODC4Zx7OZfjRZwpbdvfVm5ldW19I79Z2Nre2d0r7h80VRhLQhsk5KFse1hRzgLa0Exz2o4kxcLjtOWNbiZ+65FKxcLgXicRdQUeBMxnBGsjuQm6Ql3FBgKXn057xZJdsadAy8TJSAky1HvFn24/JLGggSYcK9Vx7Ei7KZaaEU7HhW6saITJCA9ox9AAC6rcdPr0GJ0YpY/8UJoJNJqqfy9SLJRKhGc2BdZDtehNxP+8Tqz9SzdlQRRrGpBZkB9zpEM0aQD1maRE88QQTCQzvyIyxBITbXqaS/HE2HTiLDawTJpnFee8Ur2rlmrXWTt5OIJjKIMDF1CDW6hDAwg8wAu8wpv1bL1bH9bnbDVnZTeHMAfr6xc6xpfJ y = (z) AAACxniclVFdS8MwFE3r15xfUx99CQ5BUEbbdbo9CKIvPk5wm7iVkmbZDKZpSVJ1loK/0Hcf/Sem2wTnfNmBwOHcc3Iv9wYxo1JZ1qdhLi2vrK4V1osbm1vbO6XdvbaMEoFJC0csEvcBkoRRTlqKKkbuY0FQGDDSCZ6u83rnmQhJI36nRjHxQjTkdEAxUlrySx9v8AKmvfFHXTEMvNSqWGOcWhWnUa/VXU3Oaw237mbwxbcyeFJcwG9n8NW3F8s4ecZZLFPNM1Xol8o/PjhP7Ckpgymafumr149wEhKuMENSdm0rVl6KhKKYkazYSySJEX5CQ9LVlKOQSC8dj5XBI6304SAS+nEFx+rvRIpCKUdhoJ0hUo/yby0X/6t1EzWoeynlcaIIx5NGg4RBFcH8iLBPBcGKjTRBWFA9K8SPSCCs9KlnugRhpndi/93APGk7Ffus4t665cur6XYK4AAcgmNgg3NwCW5AE7QANlzjwcBG37wxuZmYLxOraUwz+2AG5vs3e/jKcg== z = w0 + w1x1 + w2x2 + w3x3 AAAB+XicdVDLSgMxFM3UV62vqks3wSK4GmbK1La7ohuXFe0D2qFk0kwbmmSGJKOUoZ/gVtfuxK1f49I/MW1HsKIHLhzOuZd77wliRpV2nA8rt7a+sbmV3y7s7O7tHxQPj9oqSiQmLRyxSHYDpAijgrQ01Yx0Y0kQDxjpBJOrud+5J1LRSNzpaUx8jkaChhQjbaTbh4EzKJYc21kAOna5XqvUPEOqlbpniJtZJZChOSh+9ocRTjgRGjOkVM91Yu2nSGqKGZkV+okiMcITNCI9QwXiRPnp4tQZPDPKEIaRNCU0XKg/J1LElZrywHRypMfqtzcX//J6iQ5rfkpFnGgi8HJRmDCoIzj/Gw6pJFizqSEIS2puhXiMJMLapLOyJeAzk8n34/B/0i7b7oXt3XilxmWWTh6cgFNwDlxQBQ1wDZqgBTAYgUfwBJ6t1HqxXq23ZWvOymaOwQqs9y8luJSG w0 AAAB+XicdVDLSgMxFM3UV62vqks3wSK4GmbK1La7ohuXFe0D2qFk0kwbmmSGJKOUoZ/gVtfuxK1f49I/MW1HsKIHLhzOuZd77wliRpV2nA8rt7a+sbmV3y7s7O7tHxQPj9oqSiQmLRyxSHYDpAijgrQ01Yx0Y0kQDxjpBJOrud+5J1LRSNzpaUx8jkaChhQjbaTbh4E7KJYc21kAOna5XqvUPEOqlbpniJtZJZChOSh+9ocRTjgRGjOkVM91Yu2nSGqKGZkV+okiMcITNCI9QwXiRPnp4tQZPDPKEIaRNCU0XKg/J1LElZrywHRypMfqtzcX//J6iQ5rfkpFnGgi8HJRmDCoIzj/Gw6pJFizqSEIS2puhXiMJMLapLOyJeAzk8n34/B/0i7b7oXt3XilxmWWTh6cgFNwDlxQBQ1wDZqgBTAYgUfwBJ6t1HqxXq23ZWvOymaOwQqs9y8nS5SH w1 AAAB+XicdVDLSgMxFM3UV62vqks3wSK4GmbK1La7ohuXFe0D2qFk0kwbmmSGJKOUoZ/gVtfuxK1f49I/MW1HsKIHLhzOuZd77wliRpV2nA8rt7a+sbmV3y7s7O7tHxQPj9oqSiQmLRyxSHYDpAijgrQ01Yx0Y0kQDxjpBJOrud+5J1LRSNzpaUx8jkaChhQjbaTbh0F5UCw5trMAdOxyvVapeYZUK3XPEDezSiBDc1D87A8jnHAiNGZIqZ7rxNpPkdQUMzIr9BNFYoQnaER6hgrEifLTxakzeGaUIQwjaUpouFB/TqSIKzXlgenkSI/Vb28u/uX1Eh3W/JSKONFE4OWiMGFQR3D+NxxSSbBmU0MQltTcCvEYSYS1SWdlS8BnJpPvx+H/pF223Qvbu/FKjcssnTw4AafgHLigChrgGjRBC2AwAo/gCTxbqfVivVpvy9aclc0cgxVY718o3pSI w2 AAAB+XicdVDLSgMxFM34rPVVdekmWARXw0yd2nZXdOOyon1AO5RMmmlDk8yQZJQy9BPc6tqduPVrXPonpu0IVvTAhcM593LvPUHMqNKO82GtrK6tb2zmtvLbO7t7+4WDw5aKEolJE0cskp0AKcKoIE1NNSOdWBLEA0bawfhq5rfviVQ0End6EhOfo6GgIcVIG+n2oX/eLxQd25kDOnapVi1XPUMq5ZpniJtZRZCh0S989gYRTjgRGjOkVNd1Yu2nSGqKGZnme4kiMcJjNCRdQwXiRPnp/NQpPDXKAIaRNCU0nKs/J1LElZrwwHRypEfqtzcT//K6iQ6rfkpFnGgi8GJRmDCoIzj7Gw6oJFiziSEIS2puhXiEJMLapLO0JeBTk8n34/B/0irZ7oXt3XjF+mWWTg4cgxNwBlxQAXVwDRqgCTAYgkfwBJ6t1HqxXq23ReuKlc0cgSVY718qcZSJ w3 AAACA3icbVC7SgNBFJ2NrxhfUUubwSDEJuxKUMugjWUE84DsEmZnZ5Mh81hnZoWwpPQbbLW2E1s/xNI/cZJsYRIPXDiccy/ncsKEUW1c99sprK1vbG4Vt0s7u3v7B+XDo7aWqcKkhSWTqhsiTRgVpGWoYaSbKIJ4yEgnHN1O/c4TUZpK8WDGCQk4GggaU4yMlQJf0wFHVR9H0pz3yxW35s4AV4mXkwrI0eyXf/xI4pQTYTBDWvc8NzFBhpShmJFJyU81SRAeoQHpWSoQJzrIZk9P4JlVIhhLZUcYOFP/XmSIaz3mod3kyAz1sjcV//N6qYmvg4yKJDVE4HlQnDJoJJw2ACOqCDZsbAnCitpfIR4ihbCxPS2khHxiO/GWG1gl7Yuad1mr39crjZu8nSI4AaegCjxwBRrgDjRBC2DwCF7AK3hznp1358P5nK8WnPzmGCzA+foFLBmYXw== (·) When increases by , increases by AAAB+XicbVA9TwJBEJ3DL8Qv1NJmIzGxIneEoCXRxhKjfCRwIXvLHmzY3bvs7hnJhZ9gq7WdsfXXWPpPXOAKAV8yyct7M5mZF8ScaeO6305uY3Nreye/W9jbPzg8Kh6ftHSUKEKbJOKR6gRYU84kbRpmOO3EimIRcNoOxrczv/1ElWaRfDSTmPoCDyULGcHGSg/P/Uq/WHLL7hxonXgZKUGGRr/40xtEJBFUGsKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LZVYUO2n81On6MIqAxRGypY0aK7+nUix0HoiAtspsBnpVW8m/ud1ExNe+ymTcWKoJItFYcKRidDsbzRgihLDJ5Zgopi9FZERVpgYm87SlkBMbSbeagLrpFUpe7Vy9b5aqt9k6eThDM7hEjy4gjrcQQOaQGAIL/AKb07qvDsfzueiNedkM6ewBOfrF73AlEA= x2 AAAB93icbVA9SwNBEJ2LXzF+RS1tFoNgFe4kqGXQxjIB8wHJEfY2e8mS3b1jd084j/wCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzP/M4TVZpF8tGkMfUFHkkWMoKNlZrPg3LFrbpzoHXi5aQCORqD8k9/GJFEUGkIx1r3PDc2foaVYYTTaamfaBpjMsEj2rNUYkG1n80PnaILqwxRGClb0qC5+nciw0LrVAS2U2Az1qveTPzP6yUmvPUzJuPEUEkWi8KEIxOh2ddoyBQlhqeWYKKYvRWRMVaYGJvN0pZATG0m3moC66R9VfWuq7VmrVK/y9MpwhmcwyV4cAN1eIAGtIAAhRd4hTcndd6dD+dz0Vpw8plTWILz9QuRXZOd z AAAB+XicdVDLSgMxFM3UV62vqks3wSK4GmbK1La7ohuXFe0D2qFk0kwbmmSGJKOUoZ/gVtfuxK1f49I/MW1HsKIHLhzOuZd77wliRpV2nA8rt7a+sbmV3y7s7O7tHxQPj9oqSiQmLRyxSHYDpAijgrQ01Yx0Y0kQDxjpBJOrud+5J1LRSNzpaUx8jkaChhQjbaTbh0F5UCw5trMAdOxyvVapeYZUK3XPEDezSiBDc1D87A8jnHAiNGZIqZ7rxNpPkdQUMzIr9BNFYoQnaER6hgrEifLTxakzeGaUIQwjaUpouFB/TqSIKzXlgenkSI/Vb28u/uX1Eh3W/JSKONFE4OWiMGFQR3D+NxxSSbBmU0MQltTcCvEYSYS1SWdlS8BnJpPvx+H/pF223Qvbu/FKjcssnTw4AafgHLigChrgGjRBC2AwAo/gCTxbqfVivVpvy9aclc0cgxVY718o3pSI w2 Interdependency between input variables are not considered “Interaction effect” Logistic Regression/Softmax Regression (Cox, 1958) ≈ Perceptron (Rosenblatt, 1959)" It has an effect independent of the values of other variables (linearity is a naive but very strong assumption) with a softmax activation and cross entropy loss AAACA3icbVC7SgNBFL0bXzG+opY2g0GITdiVoDZC0MYygnlAsoTZyWwyZGZ3nZkV1iWl32CrtZ3Y+iGW/omTZAuTeODC4Zx7OZfjRZwpbdvfVm5ldW19I79Z2Nre2d0r7h80VRhLQhsk5KFse1hRzgLa0Exz2o4kxcLjtOWNbiZ+65FKxcLgXicRdQUeBMxnBGsjuQm6Ql3FBgKXn057xZJdsadAy8TJSAky1HvFn24/JLGggSYcK9Vx7Ei7KZaaEU7HhW6saITJCA9ox9AAC6rcdPr0GJ0YpY/8UJoJNJqqfy9SLJRKhGc2BdZDtehNxP+8Tqz9SzdlQRRrGpBZkB9zpEM0aQD1maRE88QQTCQzvyIyxBITbXqaS/HE2HTiLDawTJpnFee8Ur2rlmrXWTt5OIJjKIMDF1CDW6hDAwg8wAu8wpv1bL1bH9bnbDVnZTeHMAfr6xc6xpfJ y = (z)

Interaction detection in multivariate analysis Determination of the presence or absence of interaction Traditionally, a 'linear model + interaction term' is fitted and its significance is tested AAACqniclVHbSsNAEN3EW623qi+CL4tFEFpKkkZbX0T0QR9V7EXSEDbbbbu4ubC7UUvIT/k3PvonbmMFbX3pwMDhzJw5w4wfMyqkYXxo+tLyyupaYb24sbm1vVPa3WuLKOGYtHDEIt71kSCMhqQlqWSkG3OCAp+Rjv98Pal3XggXNAof5TgmboCGIR1QjKSivNJ7Baa9fIzDh76bGjUjj6pRs86bp01bgcbpud20M/jqpaaVZfDNM1VasFJcRGrVc6mlsr6gtG7m0nruXPRK5Z9+OA/MKSiDadx5pc9eP8JJQEKJGRLCMY1YuinikmJGsmIvESRG+BkNiaNgiAIi3DRfL4PHiunDQcRVhhLm7G9FigIhxoGvOgMkR2K2NiH/qzmJHDTdlIZxIkmIv40GCYMygpNHwT7lBEs2VgBhTtWuEI8QR1iqd/5x8YNM3cScvcA8aFs186xm39vly6vpdQrgEByBE2CCBrgEt+AOtADWDrQL7Ua71av6g/6kO9+tujbV7IM/ofe/AGC6wr0= +w12x1x2 + w23x2x3 + w31x3x1 AAACxniclVFdS8MwFE3r15xfUx99CQ5BUEbbdbo9CKIvPk5wm7iVkmbZDKZpSVJ1loK/0Hcf/Sem2wTnfNmBwOHcc3Iv9wYxo1JZ1qdhLi2vrK4V1osbm1vbO6XdvbaMEoFJC0csEvcBkoRRTlqKKkbuY0FQGDDSCZ6u83rnmQhJI36nRjHxQjTkdEAxUlrySx9v8AKmvfFHXTEMvNSqWGOcWhWnUa/VXU3Oaw237mbwxbcyeFJcwG9n8NW3F8s4ecZZLFPNM1Xol8o/PjhP7Ckpgymafumr149wEhKuMENSdm0rVl6KhKKYkazYSySJEX5CQ9LVlKOQSC8dj5XBI6304SAS+nEFx+rvRIpCKUdhoJ0hUo/yby0X/6t1EzWoeynlcaIIx5NGg4RBFcH8iLBPBcGKjTRBWFA9K8SPSCCs9KlnugRhpndi/93APGk7Ffus4t665cur6XYK4AAcgmNgg3NwCW5AE7QANlzjwcBG37wxuZmYLxOraUwz+2AG5vs3e/jKcg== z = w0 + w1x1 + w2x2 + w3x3 AAACOnicbZDLSgMxFIYz9VbrbdSlm2ARWpAy0xZ1WerGZQV7gXYYMmnahiYzQ5KRlqHv40P4DG7Vpe7ErQ9gphewrQdy+POdczjJ74WMSmVZ70ZqY3Nreye9m9nbPzg8Mo9PGjKIBCZ1HLBAtDwkCaM+qSuqGGmFgiDuMdL0hrdJvflIhKSB/6DGIXE46vu0RzFSGrlmNTdy7Us4cotJKuVhh6NQqgAu8yTZi1txQUoJzbtm1ipY04Drwp6LLJhHzTU/Ot0AR5z4CjMkZdu2QuXESCiKGZlkOpEkIcJD1CdtLX3EiXTi6V8n8EKTLuwFQh9fwSn9OxEjLuWYe7qTIzWQq7UE/ldrR6p348TUDyNFfDxb1IsY1F4kxsEuFQQrNtYCYUH1WyEeIIGw0vYubfH4RHtirzqwLhrFgn1VKN+Xs5Xq3J00OAPnIAdscA0q4A7UQB1g8ARewCt4M56NT+PL+J61poz5zClYCuPnF5txqJA= (x1, x2, x3) 7! (x1, x2, x3, x1x2, x2x3, x3x1) 3 variables 6 variables “The most naive feature engineering” • Statistically compare the likelihood of models with and without interaction terms. • Since it is linear with respect to the parameters, the calculation for fitting is the same as linear. When increases by , increases by AAAB+XicbVA9TwJBEJ3DL8Qv1NJmIzGxIneEoCXRxhKjfCRwIXvLHmzY3bvs7hnJhZ9gq7WdsfXXWPpPXOAKAV8yyct7M5mZF8ScaeO6305uY3Nreye/W9jbPzg8Kh6ftHSUKEKbJOKR6gRYU84kbRpmOO3EimIRcNoOxrczv/1ElWaRfDSTmPoCDyULGcHGSg/P/Uq/WHLL7hxonXgZKUGGRr/40xtEJBFUGsKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LZVYUO2n81On6MIqAxRGypY0aK7+nUix0HoiAtspsBnpVW8m/ud1ExNe+ymTcWKoJItFYcKRidDsbzRgihLDJ5Zgopi9FZERVpgYm87SlkBMbSbeagLrpFUpe7Vy9b5aqt9k6eThDM7hEjy4gjrcQQOaQGAIL/AKb07qvDsfzueiNedkM6ewBOfrF73AlEA= x2 AAAB93icbVA9SwNBEJ2LXzF+RS1tFoNgFe4kqGXQxjIB8wHJEfY2e8mS3b1jd084j/wCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzP/M4TVZpF8tGkMfUFHkkWMoKNlZrPg3LFrbpzoHXi5aQCORqD8k9/GJFEUGkIx1r3PDc2foaVYYTTaamfaBpjMsEj2rNUYkG1n80PnaILqwxRGClb0qC5+nciw0LrVAS2U2Az1qveTPzP6yUmvPUzJuPEUEkWi8KEIxOh2ddoyBQlhqeWYKKYvRWRMVaYGJvN0pZATG0m3moC66R9VfWuq7VmrVK/y9MpwhmcwyV4cAN1eIAGtIAAhRd4hTcndd6dD+dz0Vpw8plTWILz9QuRXZOd z AAACXniclZHLSgMxFIbTUWut1o66EdwEiyAoZWaqtd0V3bisYC/QDkMmzbShmQtJRi3DvKEv4E5fwa0uTC8L27rxQODjP39yDn/ciFEhDeMto21sbmW3czv53b3CflE/OGyLMOaYtHDIQt51kSCMBqQlqWSkG3GCfJeRjju+m/Y7T4QLGgaPchIR20fDgHoUI6kkR/eS/uyRHh+6dmKUDVXV6uUUzJphKqjXa5ZVT+GzY6XwAv7Dn1iVNIUvTgVCRy/NvYYB18FcQAksqunoH/1BiGOfBBIzJETPNCJpJ4hLihlJ8/1YkAjhMRqSnsIA+UTYyWy1FJ4pZQC9kKsTSDhTf99IkC/ExHeV00dyJFZ7U/GvXi+WXs1OaBDFkgR4PsiLGZQhnIYLB5QTLNlEAcKcql0hHiGOsFRfsDTF9VOVibmawDq0rbJZLV8/XJUat4t0cuAEnIJzYIIb0AD3oAlaAINX8Am+wHfmXctqBa04t2qZxZ0jsFTa8Q/AB691 w2 + w23x3

Interaction detection in multivariate analysis • Adding every possible factors into the model, known as 'kitchen sink' regression, was considered a flawed method that only led to overfitting https://en.wikipedia.org/wiki/Kitchen_sink_regression • In multivariate analysis, it is necessary for the analyst to carefully select the interaction terms to be included in the model based on domain knowledge An early idea toward the decision tree: AID = Automatic Interaction Detection Can't we automate this combinatorial feature selection? It naturally considers interactions like a contingency table (also known as a cross tabulation)?

Interaction detection in multivariate analysis An early idea toward the decision tree: AID = Automatic Interaction Detection AID (Morgan & Sonquist 1963) … A regression tree Theta AID (Messenger & Mandell, 1972) … A classification tree It was presented in the statistical community, but didn't receive much attention and faced quite harsh criticism...! • Overfitting issue: AID was pointed out to overfit in simulations (Einhorn, 1972). • Masking problem: When there are variables with high correlation, only one appears in the tree, leading to a spurious conclusion about the variable's importance (Doyle, 1973). • Lack of distinction between population and sample: It completely ignores the inherent 'sampling variability' in the data (Bishop+, 1975).

Brief history of decision trees 1) Concept learning 2) Interaction detection 3) Nonlinear regression CLS (Hunt+ 1966) ID3 (Quinlan 1986) C4.5 (Quinlan 1993) C5.0/See5 (Quinlan) AID (Morgan & Sonquist 1963) Breiman & Meisel 1976 Friedman 1979 THAID (Messenger & Mandell, 1972) CHAID (Kass 1980) CART (Breiman+ 1984) M5 (Quinlan 1992) M5’ (Wang & Witten 1997) MARS (Friedman 1991) Cubist (Quinlan) RETIS (Karalic & Cestnik 1991) HTL (Torgo 1997) QUEST (Loh & Shih 1997) CRUISE (Kim & Loh, 2001) OC1 (Murthy+ 1993) Quinlan 1979 Tree-shaped classification Tree-shaped regression Decision tree = A concept so simple that anyone could think of it In fact, this idea was explored in various contexts in the 1970s

Nonlinear regression One might think decision trees are for classification, it also has roots in regression. “Regression” problem • The problem of fitting a curve to points. • Linear regression = fitting a "straight line". • However, there are many situations where what we want is clearly not a straight line … → Nonlinear regression Univariate (1D) regression KNeighborsRegressor “Nearest Neighbor Estimates” “Kernel Estimates” “Partitioning Estimates” DecisionTreeRegressor KernelRegression

Nonlinear regression When solving this with polynomial regression, the optimal degree is hard to get… Polynomial Regression (4th) Polynomial Regression (10th) Polynomial Regression (30th) DecisionTreeRegressor (4 leaves) DecisionTreeRegressor (10 leaves) DecisionTreeRegressor (30 leaves) For regression trees, the degree of freedom simply controls the goodness of fit.

Nonlinear regression Let’s consider a dumbest regression model, i.e., constant regression. (No matter what the value of the input is, it returns the mean of the training data) Indeed, this produces the safest predictions, which won’t be wrong by a huge margin. But ’always returning a constant value for any x’ is neither ML nor anything else... x The idea of regression trees: What about piecewise constants? Change the constant returned locally based on the value of x. x

Nonlinear regression Text text Region averages Regression trees Find a splitting point in a certain way. For each region, it returns the region averages as predicted values. Apply this process recursively in each sub-region. We have a regression tree. (with 4 leaf nodes)

Nonlinear regression Text text ≤ > DecisionTreeRegressor (10 leaves) For univariate data

Nonlinear regression Text text https://explained.ai/decision-tree-viz/ "Diabetes" Dataset The procedure is the same for multivariate data (as will be discussed later)

Outline We will jointly learn decision trees and ensemble learning! 1. Why we should learn decision trees and ensemble learning? 2. History of Decision Trees 3. Basics of Decision Trees 4. Ensemble Learning: Additive ensembles and decision forests 5. Random forest and Regularized gradient boosted trees 6. Practical data analysis with decision tree ensembles Part 1 Part 2

Classification trees vs. Regression trees Basic learning algorithms are the same (except splitting criteria and leaf decisions) returns a categorical value (one of the prede ned categories) > ≤ ≤ > Classification trees Regression trees returns a numeric value (each leaf can have a di erent value)

The three important aspects of decision trees 1. Nested if-then rules 3. Piecewise constant prediction (i.e. histogram approximation) 2. Recursive partitioning of the input space AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGZiRVNvh8rijIXqiIrkThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOslho= X2  ✓1 yes no AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGRY0WT78eKosyFqswokThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOelho= X1  ✓2 yes no Blue AAAClnichVHLSsNAFD2N7/qquim4KZaKqzItRcWFiCK6rNZqoZWQxGkbTJOYTCta/AF/wIW4UFARP8APcOMPuOgniMsKblx4mwZExXrDZM6cuefOmbmqbeiuYKwRkLq6e3r7+geCg0PDI6OhsfFt16o6Gs9qlmE5OVVxuaGbPCt0YfCc7XClohp8R91fae3v1Ljj6pa5JY5svltRSqZe1DVFECWHxnJyMlIw+EGkIMpcKHJKDkVZnHkR+Q0SPojCj7QVekABe7CgoYoKOEwIwgYUuPTlkQCDTdwu6sQ5hHRvn+MEQdJWKYtThkLsPv1LtMr7rEnrVk3XU2t0ikHDIWUEMfbM7liTPbF79sI+/qxV92q0vBzRrLa13JZHT8OZ939VFZoFyl+qjp4Fipj3vOrk3faY1i20tr52fNbMLGzG6tPsir2S/0vWYI90A7P2pl1v8M3zDn5U8kIvRg1K/GzHb7CdjCdm46mNVHRp2W9VPyYxhRnqxxyWsI40slT/EBe4wa0UlhalVWmtnSoFfM0EvoWU/gQ6DJYd X2  ✓4 yes no Red AAAClnichVHLSsNAFD3GV62vVjeCm2JRXJWJ+MKFFEV0WVurBSshiVMNTZOYTCta/AF/wIW4UFARP8APcOMPuOgniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGGu0SK1t7R2dka5od09vX38sPrDu2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvaOVFf3+jxl3PsK01ceDwrYq6YxklQ1cFUUosXlDkRNHke4mi2OVCVaaUWJKlWBCJn0AOQRJhZOzYHYrYhg0dVVTAYUEQNqHCo28TMhgc4rZQJ84lZAT7HEeIkrZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHaX/eCj39q+qQrPA7qfqT88CJcwGXg3y7gSMfwu9qa8dnrzk5rKj9TF2wZ7J/zlrsHu6gVV71S9Xefb0Dz8aeaEXowbJ39vxE6xPpOTp1OTqZDK9ELYqgmGMYJz6MYM0VpBBnurv4wxXuJaGpHlpSVpupkotoWYQX0LKfAA5/pYd X1  ✓5 yes no Red AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4= X1  ✓6 yes no Red Blue AAAClnichVHLSsNAFD3GV3221Y3gplgUV2WiRcWFFEV0Wa3VgpWQxKmGpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxKWCGxfepAFRUW+YzJkz99w5M1dzTMMTjDVapNa29o7OSFd3T29ffzQWH1j37Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tPKCv79R465n2NaaOHD4VkXdsYySoauCKCUWLyhyomjyvURR7HKhKpNKLMlSLIjETyCHIIkwsnbsDkVsw4aOKirgsCAIm1Dh0bcJGQwOcVuoE+cSMoJ9jiN0k7ZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHSV6PJR7+1dVoVlg91P1p2eBEmYCrwZ5dwLGv4Xe1NcOT15ys6uj9TF2wZ7J/zlrsHu6gVV71S9X+OrpH3408kIvRg2Sv7fjJ1ifSMlTqfRKOpmZD1sVwTBGME79mEYGy8giT/X3cYYrXEtD0py0KC01U6WWUDOILyFlPwA1vpYb X1  ✓3 yes no Red AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykWHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm4/JcPJFJxFJLQav6MIFJzFA/kkhhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bz4+lh8= X1  ✓7 yes no Blue Blue AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSTayFoyRO3IiMAtEDUfAibYQfoQInYIACLdCAgg4csQoS2PiVQQQCJnJV6CJnIWLuPoVzCKC2hVkUMyRkm/iv46rssTqunZq2q1bwFBWHhcoIxMgLuScD8kweyCv5/LNW163heOngLA+11KyFLlZyH/+qNJw5NL5VYz1zOIVd1ytD76bLOLdQhvr22dUgl8zGuuvklryh/xvSJ094A739rtxlaLY3xo+MXvDFsEHi73aMgsJmXNyOJzKJaGrfa5UfVmENNrAfO5CCI0hD3u3DJfTgWggKW0JS2BumCj5Psww/Qjj8Ak3rksg= ✓1 AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRAooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4V4NLYTTWQS4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXtoWksD9KFTyuZhV+hHD0BVALksk= ✓2 AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSLVELR0mcuBEZBaIHouBF2gg/QgVOwAAFWqABBR04YhUksPErgwgETOSq0EXOQsTcfQrnEEBtC7MoZkjINvFfx1XZY3VcOzVtV63gKSoOC5URiJEXck8G5Jk8kFfy+WetrlvD8dLBWR5qqVkLXazkPv5VaThzaHyrxnrmcAq7rleG3k2XcW6hDPXts6tBLpmNddfJLXlD/zekT57wBnr7XbnL0GxvjB8ZveCLYYPE3+0YBYXNuLgdT2QS0dS+1yo/rMIabGA/diAFR5CGvNuHS+jBtRAUtoSksDdMFXyeZhl+hHD4BVRLkss= ✓4 AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQfIRYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixPS9wgTk1PTM95Z39y8fyEQXFzKW3rLlGlO1hXdLEqiRRWm0RxnXKFFw6SiKim0IDUP7f1Cm5oW07UT3jFoRRXrGqsxWeRIFcu8QblY3a4GwyRKnAiNgpgLwuBGSg8+QhlOQQcZWqACBQ04YgVEsPArQQwIGMhVoIuciYg5+xTOwYfaFmZRzBCRbeK/jquSy2q4tmtajlrGUxQcJipDECEv5J4MyDN5IK/k889aXaeG7aWDszTUUqMauFjJfvyrUnHm0PhWjfXMoQZ7jleG3g2HsW8hD/Xts6tBNpGJdNfJLXlD/zekT57wBlr7Xb5L00xvjB8JveCLYYNiv9sxCvKb0dhONJ6Oh5MHbqu8sAprsIH92IUkHEMKck4fLqEH14Jf2BISwv4wVfC4mmX4EcLRF1Zrksw= ✓5 AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQgoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaCwRjWfi4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BViLks0= ✓6 AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmQQooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGwnmsgkwqkDt1VeWIcN2MJ+7EIKjiENeacPl9CHa8EvxIWksD9KFTyuZhV+hHD0BVIrkso= ✓3 AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQIoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGw3Gs/Ew6kDt1VeWIcN2MJ+JCAFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BVqrks4= ✓7 AAAB+XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzQOSJcxOZpMh81hmZoWw5BO86tmbePVrPPonTpI9mMSChqKqm+6uKOHMWN//9gpr6xubW8Xt0s7u3v5B+fCoaVSqCW0QxZVuR9hQziRtWGY5bSeaYhFx2opGd1O/9Uy1YUo+2XFCQ4EHksWMYOukx3Yv6JUrftWfAa2SICcVyFHvlX+6fUVSQaUlHBvTCfzEhhnWlhFOJ6VuamiCyQgPaMdRiQU1YTY7dYLOnNJHsdKupEUz9e9EhoUxYxG5ToHt0Cx7U/E/r5Pa+CbMmExSSyWZL4pTjqxC079Rn2lKLB87golm7lZEhlhjYl06C1siMXGZBMsJrJLmRTW4ql4+XFZqt3k6RTiBUziHAK6hBvdQhwYQGMALvMKbl3nv3of3OW8tePnMMSzA+/oFiY2UHw== X1 AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5CUMugjWVE8wHJEfY2e8mS3b1jd08IR36CrdZ2YuuvsfSfuLlcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzN/c4zVZpF8slMY+oLPJIsZAQbKz12B7VBueJW3QxonXg5qUCO5qD80x9GJBFUGsKx1j3PjY2fYmUY4XRW6ieaxphM8Ij2LJVYUO2n2akzdGGVIQojZUsalKl/J1IstJ6KwHYKbMZ61ZuL/3m9xIQ3fspknBgqyWJRmHBkIjT/Gw2ZosTwqSWYKGZvRWSMFSbGprO0JRAzm4m3msA6adeq3lW1/lCvNG7zdIpwBudwCR5cQwPuoQktIDCCF3iFNyd13p0P53PRWnDymVNYgvP1C4sglCA= X2

Recursive partitioning of the input space AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSTayFoyRO3IiMAtEDUfAibYQfoQInYIACLdCAgg4csQoS2PiVQQQCJnJV6CJnIWLuPoVzCKC2hVkUMyRkm/iv46rssTqunZq2q1bwFBWHhcoIxMgLuScD8kweyCv5/LNW163heOngLA+11KyFLlZyH/+qNJw5NL5VYz1zOIVd1ytD76bLOLdQhvr22dUgl8zGuuvklryh/xvSJ094A739rtxlaLY3xo+MXvDFsEHi73aMgsJmXNyOJzKJaGrfa5UfVmENNrAfO5CCI0hD3u3DJfTgWggKW0JS2BumCj5Psww/Qjj8Ak3rksg= ✓1 AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGZiRVNvh8rijIXqiIrkThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOslho= X2  ✓1 yes no AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRAooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4V4NLYTTWQS4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXtoWksD9KFTyuZhV+hHD0BVALksk= ✓2 AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGRY0WT78eKosyFqswokThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOelho= X1  ✓2 yes no Blue AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSLVELR0mcuBEZBaIHouBF2gg/QgVOwAAFWqABBR04YhUksPErgwgETOSq0EXOQsTcfQrnEEBtC7MoZkjINvFfx1XZY3VcOzVtV63gKSoOC5URiJEXck8G5Jk8kFfy+WetrlvD8dLBWR5qqVkLXazkPv5VaThzaHyrxnrmcAq7rleG3k2XcW6hDPXts6tBLpmNddfJLXlD/zekT57wBnr7XbnL0GxvjB8ZveCLYYPE3+0YBYXNuLgdT2QS0dS+1yo/rMIabGA/diAFR5CGvNuHS+jBtRAUtoSksDdMFXyeZhl+hHD4BVRLkss= ✓4 AAAClnichVHLSsNAFD2N7/qquim4KZaKqzItRcWFiCK6rNZqoZWQxGkbTJOYTCta/AF/wIW4UFARP8APcOMPuOgniMsKblx4mwZExXrDZM6cuefOmbmqbeiuYKwRkLq6e3r7+geCg0PDI6OhsfFt16o6Gs9qlmE5OVVxuaGbPCt0YfCc7XClohp8R91fae3v1Ljj6pa5JY5svltRSqZe1DVFECWHxnJyMlIw+EGkIMpcKHJKDkVZnHkR+Q0SPojCj7QVekABe7CgoYoKOEwIwgYUuPTlkQCDTdwu6sQ5hHRvn+MEQdJWKYtThkLsPv1LtMr7rEnrVk3XU2t0ikHDIWUEMfbM7liTPbF79sI+/qxV92q0vBzRrLa13JZHT8OZ939VFZoFyl+qjp4Fipj3vOrk3faY1i20tr52fNbMLGzG6tPsir2S/0vWYI90A7P2pl1v8M3zDn5U8kIvRg1K/GzHb7CdjCdm46mNVHRp2W9VPyYxhRnqxxyWsI40slT/EBe4wa0UlhalVWmtnSoFfM0EvoWU/gQ6DJYd X2  ✓4 yes no Red AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQfIRYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixPS9wgTk1PTM95Z39y8fyEQXFzKW3rLlGlO1hXdLEqiRRWm0RxnXKFFw6SiKim0IDUP7f1Cm5oW07UT3jFoRRXrGqsxWeRIFcu8QblY3a4GwyRKnAiNgpgLwuBGSg8+QhlOQQcZWqACBQ04YgVEsPArQQwIGMhVoIuciYg5+xTOwYfaFmZRzBCRbeK/jquSy2q4tmtajlrGUxQcJipDECEv5J4MyDN5IK/k889aXaeG7aWDszTUUqMauFjJfvyrUnHm0PhWjfXMoQZ7jleG3g2HsW8hD/Xts6tBNpGJdNfJLXlD/zekT57wBlr7Xb5L00xvjB8JveCLYYNiv9sxCvKb0dhONJ6Oh5MHbqu8sAprsIH92IUkHEMKck4fLqEH14Jf2BISwv4wVfC4mmX4EcLRF1Zrksw= ✓5 AAAClnichVHLSsNAFD3GV62vVjeCm2JRXJWJ+MKFFEV0WVurBSshiVMNTZOYTCta/AF/wIW4UFARP8APcOMPuOgniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGGu0SK1t7R2dka5od09vX38sPrDu2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvaOVFf3+jxl3PsK01ceDwrYq6YxklQ1cFUUosXlDkRNHke4mi2OVCVaaUWJKlWBCJn0AOQRJhZOzYHYrYhg0dVVTAYUEQNqHCo28TMhgc4rZQJ84lZAT7HEeIkrZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHaX/eCj39q+qQrPA7qfqT88CJcwGXg3y7gSMfwu9qa8dnrzk5rKj9TF2wZ7J/zlrsHu6gVV71S9Xefb0Dz8aeaEXowbJ39vxE6xPpOTp1OTqZDK9ELYqgmGMYJz6MYM0VpBBnurv4wxXuJaGpHlpSVpupkotoWYQX0LKfAA5/pYd X1  ✓5 yes no Red AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQgoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaCwRjWfi4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BViLks0= ✓6 AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4= X1  ✓6 yes no Red Blue AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmQQooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGwnmsgkwqkDt1VeWIcN2MJ+7EIKjiENeacPl9CHa8EvxIWksD9KFTyuZhV+hHD0BVIrkso= ✓3 AAAClnichVHLSsNAFD3GV3221Y3gplgUV2WiRcWFFEV0Wa3VgpWQxKmGpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxKWCGxfepAFRUW+YzJkz99w5M1dzTMMTjDVapNa29o7OSFd3T29ffzQWH1j37Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tPKCv79R465n2NaaOHD4VkXdsYySoauCKCUWLyhyomjyvURR7HKhKpNKLMlSLIjETyCHIIkwsnbsDkVsw4aOKirgsCAIm1Dh0bcJGQwOcVuoE+cSMoJ9jiN0k7ZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHSV6PJR7+1dVoVlg91P1p2eBEmYCrwZ5dwLGv4Xe1NcOT15ys6uj9TF2wZ7J/zlrsHu6gVV71S9X+OrpH3408kIvRg2Sv7fjJ1ifSMlTqfRKOpmZD1sVwTBGME79mEYGy8giT/X3cYYrXEtD0py0KC01U6WWUDOILyFlPwA1vpYb X1  ✓3 yes no Red AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQIoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGw3Gs/Ew6kDt1VeWIcN2MJ+JCAFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BVqrks4= ✓7 AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykWHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm4/JcPJFJxFJLQav6MIFJzFA/kkhhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bz4+lh8= X1  ✓7 yes no Blue Blue if x2 ≤ θ1 then if x1 ≤ θ2 then return Blue else if x2 ≤ θ4 then return Red else if x1 ≤ θ5 then return Red else if x1 ≤ θ6 then return Blue else return Red else if x1 ≤ θ3 then if x2 ≤ θ7 then return Blue else return Blue else return Red AAAB+XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzQOSJcxOZpMh81hmZoWw5BO86tmbePVrPPonTpI9mMSChqKqm+6uKOHMWN//9gpr6xubW8Xt0s7u3v5B+fCoaVSqCW0QxZVuR9hQziRtWGY5bSeaYhFx2opGd1O/9Uy1YUo+2XFCQ4EHksWMYOukx3Yv6JUrftWfAa2SICcVyFHvlX+6fUVSQaUlHBvTCfzEhhnWlhFOJ6VuamiCyQgPaMdRiQU1YTY7dYLOnNJHsdKupEUz9e9EhoUxYxG5ToHt0Cx7U/E/r5Pa+CbMmExSSyWZL4pTjqxC079Rn2lKLB87golm7lZEhlhjYl06C1siMXGZBMsJrJLmRTW4ql4+XFZqt3k6RTiBUziHAK6hBvdQhwYQGMALvMKbl3nv3of3OW8tePnMMSzA+/oFiY2UHw== X1 AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5CUMugjWVE8wHJEfY2e8mS3b1jd08IR36CrdZ2YuuvsfSfuLlcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzN/c4zVZpF8slMY+oLPJIsZAQbKz12B7VBueJW3QxonXg5qUCO5qD80x9GJBFUGsKx1j3PjY2fYmUY4XRW6ieaxphM8Ij2LJVYUO2n2akzdGGVIQojZUsalKl/J1IstJ6KwHYKbMZ61ZuL/3m9xIQ3fspknBgqyWJRmHBkIjT/Gw2ZosTwqSWYKGZvRWSMFSbGprO0JRAzm4m3msA6adeq3lW1/lCvNG7zdIpwBudwCR5cQwPuoQktIDCCF3iFNyd13p0P53PRWnDymVNYgvP1C4sglCA= X2 AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5CUMugjWVE8wHJEfY2e8mS3b1jd08IR36CrdZ2YuuvsfSfuLlcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzN/c4zVZpF8slMY+oLPJIsZAQbKz12B7VBueJW3QxonXg5qUCO5qD80x9GJBFUGsKx1j3PjY2fYmUY4XRW6ieaxphM8Ij2LJVYUO2n2akzdGGVIQojZUsalKl/J1IstJ6KwHYKbMZ61ZuL/3m9xIQ3fspknBgqyWJRmHBkIjT/Gw2ZosTwqSWYKGZvRWSMFSbGprO0JRAzm4m3msA6adeq3lW1/lCvNG7zdIpwBudwCR5cQwPuoQktIDCCF3iFNyd13p0P53PRWnDymVNYgvP1C4sglCA= X2 AAAB+XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzQOSJcxOZpMh81hmZoWw5BO86tmbePVrPPonTpI9mMSChqKqm+6uKOHMWN//9gpr6xubW8Xt0s7u3v5B+fCoaVSqCW0QxZVuR9hQziRtWGY5bSeaYhFx2opGd1O/9Uy1YUo+2XFCQ4EHksWMYOukx3Yv6JUrftWfAa2SICcVyFHvlX+6fUVSQaUlHBvTCfzEhhnWlhFOJ6VuamiCyQgPaMdRiQU1YTY7dYLOnNJHsdKupEUz9e9EhoUxYxG5ToHt0Cx7U/E/r5Pa+CbMmExSSyWZL4pTjqxC079Rn2lKLB87golm7lZEhlhjYl06C1siMXGZBMsJrJLmRTW4ql4+XFZqt3k6RTiBUziHAK6hBvdQhwYQGMALvMKbl3nv3of3OW8tePnMMSzA+/oFiY2UHw== X1 AAAClnichVHLSsNAFD3GV323uhHcFEvFVZmKqLiQoogu66NaaEtI4lhH0yQm00ot/oA/4EJcKKiIH+AHuPEHXPgJ4lLBjQtv04BoUW+YzJkz99w5M1d3TOFJxp5alNa29o7OUFd3T29f/0A4Mrjh2WXX4BnDNm03q2seN4XFM1JIk2cdl2sl3eSb+t5CfX+zwl1P2Na6rDq8UNKKltgWhiaJUsORrCqieZPvR/Nyh0tN3VXDMZZgfkSbQTIAMQSRtsN3yGMLNgyUUQKHBUnYhAaPvhySYHCIK6BGnEtI+PscR+gmbZmyOGVoxO7Rv0irXMBatK7X9Hy1QaeYNFxSRhFnj+yGvbIHdsue2cevtWp+jbqXKs16Q8sddeB4eO39X1WJZomdL9WfniW2MeN7FeTd8Zn6LYyGvnJ48ro2uxqvjbEL9kL+z9kTu6cbWJU343KFr57+4UcnL/Ri1KDkz3Y0g42JRHIqMbkyGUvNB60KYQSjGKd+TCOFZaSRofoHOMMVrpVhZU5ZVJYaqUpLoBnCt1DSnyS9loo= Xi  ✓j yes no Learning a tree = Recursively find each split (stump)

Recursively partitioning the data Text text virginica versicolor setosa

Recursively partitioning the data Multi-dimensional data is viewed by projecting onto one dimension for each variable. virginica versicolor setosa Note: Decision trees do not look at multiple variables simultaneously! (that’s why it’s free from the curse of dimensionality…)

Recursively partitioning the data virginica versicolor setosa Multi-dimensional data is viewed by projecting onto one dimension for each variable. Note: Decision trees do not look at multiple variables simultaneously! (that’s why it’s free from the curse of dimensionality…)

Recursively partitioning the data virginica versicolor setosa Multi-dimensional data is viewed by projecting onto one dimension for each variable. Note: Decision trees do not look at multiple variables simultaneously! (that’s why it’s free from the curse of dimensionality…)

Recursively partitioning the data virginica versicolor setosa Multi-dimensional data is viewed by projecting onto one dimension for each variable. Note: Decision trees do not look at multiple variables simultaneously! (that’s why it’s free from the curse of dimensionality…) threshold Take the threshold at the midpoint between neighboring points.

Recursively partitioning the data virginica versicolor setosa Pick one of variables, and split the input space by a threshold on it threshold threshold Multi-dimensional data is viewed by projecting onto one dimension for each variable.

Recursively partitioning the data Recursive bi-partitioning Apply the same process assuming that the subregion data was given from the beginning No need for further splitting (We stop resursion) setosa We do splitting once more (We have refinements)

So how can we find the best split? Learning a stump = Finding the best split (a variable + a threshold) A toy example of two variable (petal.length & petal.width) These splits geometrically correspond to axis-parallel lines in the 2D input space = Simply examining all possible splits one by one! (Calculate the split criterion value for each split as we explain later)

Finding the best splits In the iris dataset, two best splits are found based on the split criterion value (in this example, smaller is better). Choose one of them to split. iris ( )

Finding the best splits (scikit-learn) Text text ≤ > Depth

The split criterion: Impurity scores Since decision trees make constant predictions at leaf nodes, a 'good split' is one where the values within the region are as uniform as possible (i.e., with minimal mixing). Small impurity Large impurity Small impurity Large impurity Criteria for classification sklearn.tree.DecisionTreeClassifier sklearn.tree.DecisionTreeRegressor •"gini" The Gini impurity •"entropy", "log_loss" The Shannon information gain •"squared_error" The mean squared error •"absolute_error" The mean absolute error •"poisson" Poisson deviance •"friedman_mse" (A special one for gradient boosting) Criteria for regression

Split criteria (impurities) for classification AAACJHicdVDLSgMxFM3UV62vqks3wSJU0DJT6mNTKLoR3FSwD+jUIZOmbZhkZkgyQhnmJ/wIv8Gtrt2JCzeCf2KmrWB9HAgczrmXm3PckFGpTPPNyMzNLywuZZdzK6tr6xv5za2mDCKBSQMHLBBtF0nCqE8aiipG2qEgiLuMtFzvPPVbt0RIGvjXahSSLkcDn/YpRkpLTv7A5kgNBY8pD5Niex9W4aEtI+7EXtVKbi5h6HjQZsEgJU6+YJbKR6YG/E2skjlGAUxRd/Ifdi/AESe+wgxJ2bHMUHVjJBTFjCQ5O5IkRNhDA9LR1EecyG48TpXAPa30YD8Q+vkKjtXvGzHiUo64qyfTDPKnl4p/eZ1I9U+7MfXDSBEfTw71IwZVANOKYI8KghUbaYKwoPqvEA+RQFjpImeuuDzRnXwFh/+TZrlkHZcqV5VC7WzaThbsgF1QBBY4ATVwAeqgATC4Aw/gETwZ98az8WK8TkYzxnRnG8zAeP8EUA2kiQ== imp(X) = K X k=1 pk log pk AAACIHicdVDJSgNBFOyJW4zbqEcvjYmQU5iEuCAIQS8eI5gFMiH0dHqSJj0L3W/EMJlf8CP8Bq969iYe9eaf2FkE41LQUFS9pV85oeAKLOvNSC0sLi2vpFcza+sbm1vm9k5dBZGkrEYDEcimQxQT3Gc14CBYM5SMeI5gDWdwMfYbN0wqHvjXMAxZ2yM9n7ucEtBSx8yHnQE+PbNdSWhsA7uF2M7iwMVUEKVwbpBLknjUHCUdM2sVSoeWBv5NigVrgiyaodoxP+xuQCOP+TCZ1SpaIbRjIoFTwZKMHSkWEjogPdbS1CceU+14clGCD7TSxW4g9fMBT9TvHTHxlBp6jq70CPTVT28s/uW1InBP2jH3wwiYT6eL3EhgCPA4HtzlklEQQ00IlVz/FdM+0eGADnFui+ONM/k6HP9P6qVC8ahQvipnK+ezdNJoD+2jPCqiY1RBl6iKaoiiO/SAHtGTcW88Gy/G67Q0Zcx6dtEcjPdPMKijjg== pk := # of class k |X| The ratio of samples from class k in X The Gini impurity The Shannon information gain Note: The Gini coefficient and entropy are more sensitive to probability changes than the misclassification rate, making ties less likely to occur. AAAB+XicdVDJSgNBEK1xjXGLevTSGARPw0yIy0UIehG8RDQLJEPo6fQkTbp7hu4eIQz5BK969iZe/RqP/omdRTAuDwoe71VRVS9MONPG896dhcWl5ZXV3Fp+fWNza7uws1vXcaoIrZGYx6oZYk05k7RmmOG0mSiKRchpIxxcjv3GPVWaxfLODBMaCNyTLGIEGyvdXp+XOoWi55aOPQv0m/iuN0ERZqh2Ch/tbkxSQaUhHGvd8r3EBBlWhhFOR/l2qmmCyQD3aMtSiQXVQTY5dYQOrdJFUaxsSYMm6veJDAuthyK0nQKbvv7pjcW/vFZqorMgYzJJDZVkuihKOTIxGv+NukxRYvjQEkwUs7ci0scKE2PTmdsSipHN5Otx9D+pl1z/xC3flIuVi1k6OdiHAzgCH06hAldQhRoQ6MEDPMKTkznPzovzOm1dcGYzezAH5+0TYKyUBg== K = 2 AAAB+XicdVDJSgNBEK2JW4xb1KOXxiB4GmZCoh6DXjxGNAskQ+jp9CRNunuG7h4hDPkEr3r2Jl79Go/+iZ1FMC4PCh7vVVFVL0w408bz3p3cyura+kZ+s7C1vbO7V9w/aOo4VYQ2SMxj1Q6xppxJ2jDMcNpOFMUi5LQVjq6mfuueKs1ieWfGCQ0EHkgWMYKNlW6Tnt8rljy3XPUs0G/iu94MJVig3it+dPsxSQWVhnCsdcf3EhNkWBlGOJ0UuqmmCSYjPKAdSyUWVAfZ7NQJOrFKH0WxsiUNmqnfJzIstB6L0HYKbIb6pzcV//I6qYkugozJJDVUkvmiKOXIxGj6N+ozRYnhY0swUczeisgQK0yMTWdpSygmNpOvx9H/pFl2/TO3clMp1S4X6eThCI7hFHw4hxpcQx0aQGAAD/AIT07mPDsvzuu8NecsZg5hCc7bJ89KlEw= p1 AAACP3icdVDLSgMxFM34rPVVdekmWIS6sMyU+tgUii4UuqlgH9DHkEnTNkwyMyQZoQzzSX6E3+BK0IUrd+LWnZm2gq16IHA4515uznECRqUyzSdjYXFpeWU1tZZe39jc2s7s7NalHwpMathnvmg6SBJGPVJTVDHSDARB3GGk4biXid+4I0JS37tVo4B0OBp4tE8xUlqyM1dtjtRQ8IjyIM41j2AJtmXI7cgtWXG3AgPbhTkLHidkzowqMZw63YKdyZr5wompAX8TK2+OkQVTVO3Ma7vn45ATT2GGpGxZZqA6ERKKYkbidDuUJEDYRQPS0tRDnMhONA4cw0Ot9GDfF/p5Co7VnxsR4lKOuKMnk3hy3kvEv7xWqPrnnYh6QaiIhyeH+iGDyodJe7BHBcGKjTRBWFD9V4iHSCCsdMczVxwe606+g8P/Sb2Qt07zxZtitnwxbScF9sEByAELnIEyuAZVUAMY3INH8AxejAfjzXg3PiajC8Z0Zw/MwPj8AhsprSI= imp(X) = K X k=1 pk(1 pk) = K X k=1 1 p2 k AAACDnicbVDLSsNAFJ3UV62vqODGzWAR3FiS4mtTKLpxWcE+oI1hMp20Q2aSMDMRSsw/+A1ude1O3PoLLv0Tp20WtvXAhcM593Iux4sZlcqyvo3C0vLK6lpxvbSxubW9Y+7utWSUCEyaOGKR6HhIEkZD0lRUMdKJBUHcY6TtBTdjv/1IhKRReK9GMXE4GoTUpxgpLbnmQQ3a8BT2ZMLdNKjZGYzd4KHqmmWrYk0AF4mdkzLI0XDNn14/wgknocIMSdm1rVg5KRKKYkayUi+RJEY4QAPS1TREnEgnnfyfwWOt9KEfCT2hghP170WKuJQj7ulNjtRQzntj8T+vmyj/yklpGCeKhHga5CcMqgiOy4B9KghWbKQJwoLqXyEeIoGw0pXNpHg8053Y8w0skla1Yl9Uzu/OyvXrvJ0iOARH4ATY4BLUwS1ogCbA4Am8gFfwZjwb78aH8TldLRj5zT6YgfH1C0C3mu0= = 1 X k=1 p2 k Example) Binary classi cation ( ) with and AAAB+XicbVDLTgJBEOzFF+IL9ehlIjHxRHaNryPRi0eMIiSwIbPDLEyYx2Zm1oRs+ASvevZmvPo1Hv0TB9iDgJV0UqnqTndXlHBmrO9/e4WV1bX1jeJmaWt7Z3evvH/wZFSqCW0QxZVuRdhQziRtWGY5bSWaYhFx2oyGtxO/+Uy1YUo+2lFCQ4H7ksWMYOukh6QbdMsVv+pPgZZJkJMK5Kh3yz+dniKpoNISjo1pB35iwwxrywin41InNTTBZIj7tO2oxIKaMJueOkYnTumhWGlX0qKp+nciw8KYkYhcp8B2YBa9ifif105tfB1mTCappZLMFsUpR1ahyd+oxzQllo8cwUQzdysiA6wxsS6duS2RGLtMgsUElsnTWTW4rF7cn1dqN3k6RTiCYziFAK6gBndQhwYQ6MMLvMKbl3nv3of3OWstePnMIczB+/oFr9eUOA== p1 AAAB+XicbVDLSgNBEOyNrxhfUY9eBoPgKewGX8egF48RzQOSJcxOZpMhM7PLzKwQlnyCVz17E69+jUf/xNlkDyaxoKGo6qa7K4g508Z1v53C2vrG5lZxu7Szu7d/UD48aukoUYQ2ScQj1QmwppxJ2jTMcNqJFcUi4LQdjO8yv/1MlWaRfDKTmPoCDyULGcHGSo9xv9YvV9yqOwNaJV5OKpCj0S//9AYRSQSVhnCsdddzY+OnWBlGOJ2WeommMSZjPKRdSyUWVPvp7NQpOrPKAIWRsiUNmql/J1IstJ6IwHYKbEZ62cvE/7xuYsIbP2UyTgyVZL4oTDgyEcr+RgOmKDF8YgkmitlbERlhhYmx6SxsCcTUZuItJ7BKWrWqd1W9fLio1G/zdIpwAqdwDh5cQx3uoQFNIDCEF3iFNyd13p0P53PeWnDymWNYgPP1C7FqlDk= p2 0.5 × The Shannon information gain The Gini impurity Misclassification rate AAACB3icbVDLSgMxFM34rPXRUZdugkWoCGWm+NoIRTcuK9gHtMOQSTNtaJIJSUYopR/gN7jVtTtx62e49E9M21nY1gMXDufcy7mcSDKqjed9Oyura+sbm7mt/PbO7l7B3T9o6CRVmNRxwhLVipAmjApSN9Qw0pKKIB4x0owGdxO/+USUpol4NENJAo56gsYUI2Ol0C2UZOjDMyjDCryB/mnoFr2yNwVcJn5GiiBDLXR/Ot0Ep5wIgxnSuu170gQjpAzFjIzznVQTifAA9UjbUoE40cFo+vgYnlilC+NE2REGTtW/FyPEtR7yyG5yZPp60ZuI/3nt1MTXwYgKmRoi8CwoThk0CZy0ALtUEWzY0BKEFbW/QtxHCmFju5pLifjYduIvNrBMGpWyf1m+eDgvVm+zdnLgCByDEvDBFaiCe1ADdYBBCl7AK3hznp1358P5nK2uONnNIZiD8/ULzR+XTA== (p1 + p2 = 1)

Split criteria (impurities) for regression The mean squared error (MSE) The mean absolute error (MAE) Poisson deviance (when y is a count value or frequency) AAACT3icdVDLSgMxFM3UV31XXboJFqGClhmpj40gunGpYK3QqUMmzbTBJDMkGWGI+TQ/wqVLcatrd2JaK/g8EDg5996cmxNnjCrt+w9eaWx8YnKqPD0zOze/sFhZWr5QaS4xaeKUpfIyRoowKkhTU83IZSYJ4jEjrfj6eFBv3RCpaCrOdZGRDkc9QROKkXZSVGmFHOm+5IbyzNZCU0TBZthNtdosIhHaDXgAw0QibAJrhA1VziNDDwJ75W7wtogo3Pp8gZMuRcLWTGE3bqNK1a9v7/gO8DcJ6v4QVTDCaVR5dLY450RozJBS7cDPdMcgqSlmxM6EuSIZwteoR9qOCsSJ6phhABauO6ULk1S6IzQcql8nDOJKFTx2nYNl1c/aQPyr1s51st8xVGS5JgJ/GCU5gzqFgzRhl0qCNSscQVhStyvEfeTy0i7zby4xty6Tz4/D/8nFdj3YrTfOGtXDo1E6ZbAK1kANBGAPHIITcAqaAIM78ASewYt37716b6VRa8kbkRXwDaXpd7h5tgI= imp({y1, . . . , yn }) = 1 n n X i=1 |yi median(y)| AAACTXicdVBJSwMxGM3Ufa969BIsQgtaZorbRRC9eKxgVejUIZNmNJhkhiQjDCG/zB/h2WOvevYmYrqB64PAy/v2F2eMKu37z15pYnJqemZ2bn5hcWl5pby6dqnSXGLSwilL5XWMFGFUkJammpHrTBLEY0au4vvTfvzqgUhFU3Ghi4x0OLoVNKEYaSdF5VbIkb6T3FCe2WpoiijYDrupVttFJEJbg0cwTCTCJrBG2FDlPDL0KLA37gerRUThzrgDJ0jYalGr3TSicsWvN/Z8B/ibBHV/gAoYoRmVe24ozjkRGjOkVDvwM90xSGqKGbHzYa5IhvA9uiVtRwXiRHXM4HwLt5zShUkq3RMaDtSvFQZxpQoeu8z+qupnrC/+FWvnOjnsGCqyXBOBh4OSnEGdwr6XsEslwZoVjiAsqdsV4jvk3NLO8W9TYm6dJ+PD4f/kslEP9uu757uV45ORO7NgA2yCKgjAATgGZ6AJWgCDR9ADL+DVe/LevHfvY5ha8kY16+AbSjOfVbS1EQ== imp({y1, . . . , yn }) = 1 n n X i=1 (yi mean(y))2 AAACgHicdVFdaxQxFM2Mrda26vrxJpXgIsxiXWdKtUUolPbFxwrdtrBZh0w2sxuaZIbkjjCEvPsX+9g/4G8ws92CrXogcO695+benBS1FBbS9CqKH6ysPny09nh9Y/PJ02e95y/ObNUYxkeskpW5KKjlUmg+AgGSX9SGU1VIfl5cHnf18x/cWFHpU2hrPlF0pkUpGIWQyns/iaIwN8oJVfuEuDbPtsm0Arvd5pr4AT7ApDSUucw77YltVO7EQea/hwgTyUtI2lwEVs2WwhB6d3ur4lT7pB14/AF3uvf4fgUTI2ZzGOS9fjrc+ZQG4L9JNkwX6KMlTvLeddiTNYprYJJaO87SGiaOGhBMcr9OGstryi7pjI8D1VRxO3ELxzx+FzJTXFYmHA14kf2zw1FlbauKoOz2tfdrXfJftXED5f7ECV03wDW7GVQ2EkOFO/vxVBjOQLaBUGZE2BWzOQ2+QfikO1MK5YMntw/H/ydnO8Ps83D3227/8Gjpzhp6jd6iBGVoDx2ir+gEjRBDv6JX0Vb0Jo7jJP4YZzfSOFr2vER3EH/5DUesw0I= imp({y1, . . . , yn }) = 1 n n X i=1 ✓ yi log yi mean(y) yi + mean(y) ◆ https://scikit-learn.org/stable/modules/tree.html#mathematical-formulation

Impurity improvement for a good split If the 'impurity' decreases by splitting, then it's a 'good split'. The best split maximizes the amount of improvement in impurity before and after a split . AAAChHichVG7SgNBFD1ZNcb4SNRGsAmGiIWEiQ8UCwnaWOZhHhBD2F3HuGRf7G4CMfgD2ioWVgoW4gf4ATb+gEU+QSwj2Fh4s1kQDca7zM6ZM/fcOTNXMlXFdhhr+4Sh4RH/aGAsOD4xORUKT8/kbaNuyTwnG6phFSXR5qqi85yjOCovmhYXNUnlBam2290vNLhlK4a+7zRNXtbEqq4cKbLoEJW2K+EoizM3Iv0g4YEovEgZ4Ucc4BAGZNShgUOHQ1iFCJu+EhJgMIkro0WcRUhx9zlOESRtnbI4ZYjE1uhfpVXJY3Vad2varlqmU1QaFikjiLEXds867Jk9sFf2+Wetlluj66VJs9TTcrMSOpvLfvyr0mh2cPytGujZwRE2Xa8KeTddpnsLuadvnFx1sluZWGuR3bI38n/D2uyJbqA33uW7NM9cD/AjkRd6MWpQ4nc7+kF+JZ4gnF6LJne8VgUwjwUsUT82kMQeUshRfY5zXOBS8AvLwqqw3ksVfJ5mFj9C2P4CyLeP6g== impurity before the split AAAChHichVG7SgNBFD1ZNcb4SNRGsAmGiIWEiQ8UCwnaWOZhHhBD2F3HuGRf7G4CMfgD2ioWVgoW4gf4ATb+gEU+QSwj2Fh4s1kQDca7zM6ZM/fcOTNXMlXFdhhr+4Sh4RH/aGAsOD4xORUKT8/kbaNuyTwnG6phFSXR5qqi85yjOCovmhYXNUnlBam2290vNLhlK4a+7zRNXtbEqq4cKbLoEJW2K+EoizM3Iv0g4YEovEgZ4Ucc4BAGZNShgUOHQ1iFCJu+EhJgMIkro0WcRUhx9zlOESRtnbI4ZYjE1uhfpVXJY3Vad2varlqmU1QaFikjiLEXds867Jk9sFf2+Wetlluj66VJs9TTcrMSOpvLfvyr0mh2cPytGujZwRE2Xa8KeTddpnsLuadvnFx1sluZWGuR3bI38n/D2uyJbqA33uW7NM9cD/AjkRd6MWpQ4nc7+kF+JZ4gnF6LJne8VgUwjwUsUT82kMQeUshRfY5zXOBS8AvLwqqw3ksVfJ5mFj9C2P4CyLeP6g== AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhgUIwV0caShzwSJGR3HXDDvrK7kCDxB0xspbDSxML4AX6AjT9gwScYS0xsLLwsmxgl4t3Mzpkz99w5M1cyVcV2GOv5hInJqekZ/2xgbn5hMRhaWi7YRtOSeV42VMMqSaLNVUXneUdxVF4yLS5qksqLUuNgsF9scctWDP3IaZu8ool1XakpsugQldOr2WoowmLMjfAoiHsgAi/SRugRxziBARlNaODQ4RBWIcKmr4w4GEziKugQZxFS3H2OcwRI26QsThkisQ3612lV9lid1oOatquW6RSVhkXKMKLshd2zPntmD+yVff5Zq+PWGHhp0ywNtdysBi9Wcx//qjSaHZx+q8Z6dlDDrutVIe+mywxuIQ/1rbNuP7eXjXY22C17I/83rMee6AZ6612+y/Ds9Rg/EnmhF6MGxX+3YxQUtmLxnVgik4ik9r1W+bGGdWxSP5JI4RBp5Kl+HZe4QlfwCzFhW0gOUwWfp1nBjxBSX4V+kLM= nR AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhgUIwV0cbCgoc8EiRkdx1xw76yu5Ag8QdMbKWw0sTC+AF+gI0/YMEnGEtMbCy8LJsYJeLdzM6ZM/fcOTNXMlXFdhjr+oSx8YnJKf90YGZ2bj4YWlgs2EbDknleNlTDKkmizVVF53lHcVReMi0uapLKi1J9r79fbHLLVgz90GmZvKKJNV05UWTRISqnVw+qoQiLMTfCwyDugQi8SBuhRxzhGAZkNKCBQ4dDWIUIm74y4mAwiaugTZxFSHH3Oc4RIG2DsjhliMTW6V+jVdljdVr3a9quWqZTVBoWKcOIshd2z3rsmT2wV/b5Z622W6PvpUWzNNBysxq8WM59/KvSaHZw+q0a6dnBCbZdrwp5N12mfwt5oG+edXq5nWy0vcZu2Rv5v2Fd9kQ30Jvv8l2GZ69H+JHIC70YNSj+ux3DoLARi2/FEplEJLXrtcqPFaxinfqRRAr7SCNP9Wu4xBU6gl+ICZtCcpAq+DzNEn6EkPoCeL6QrQ== nL AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqqZUCIRZhTgT7QdQFIbiR0AOPOMQRdMioowYODTZhFSIs+gqIgsEgrogWcSYhxdnnOIWPtHXK4pQhElulf4VWBZfVaN2taTlqmU5RaZikDCLMXtg967Bn9sBe2eeftVpOja6XJs1ST8uNkv9sPv3xr6pGs43jb9VAzzbK2HK8KuTdcJjuLeSevnFy1Ulvp8KtJXbL3sj/DWuzJ7qB1niX75I8dT3Aj0Re6MWoQdHf7egH2dVIdCMSS8ZC8V23VV4sYBHL1I9NxLGPBDJUn+McF7gURoUVYU1Y76UKHlczhx8h7HwBwQOP7g== n AAAChHichVG7SgNBFD1ZNcZ31EawCQbFQsJE4wMLCdpYWJjERCGK7K6TOLjZXXYngRj8AW0VCysFC/ED/AAbf8AinyCWEWwsvNksiAb1LrNz5sw9d87M1WxDuJKxekDp6OwKdod6evv6BwaHwsMjOdcqOzrP6pZhOTua6nJDmDwrhTT4ju1wtaQZfFs7Wmvub1e44wrL3JJVm++V1KIpCkJXJVGpjf1wlMWYF5F2EPdBFH5sWuEH7OIAFnSUUQKHCUnYgAqXvjziYLCJ20ONOIeQ8PY5TtBL2jJlccpQiT2if5FWeZ81ad2s6XpqnU4xaDikjGCSPbM71mBP7J69sI9fa9W8Gk0vVZq1lpbb+0OnY5n3f1UlmiUOv1R/epYoYMnzKsi77THNW+gtfeX4spFZTk/WptgNeyX/16zOHukGZuVNv03x9NUffjTyQi9GDYr/bEc7yM3G4guxRCoRTa76rQphHBOYpn4sIol1bCJL9TnOcI4LJajMKHPKfCtVCfiaUXwLZeUTeMOPzA== L AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqmSoFQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkChYOP0g== R AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqmS8FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCkkOP2A== X samples yes no AAAC93ichVFPSxtBFH9ZW01t1aiXgpelwZJFDBMRFaEg1YMHDxobDbgSZsdJMjj7h9lJIF3yBfwCHjwUCxWlJ09+AC9+gYJ+BOnRQi+l9GWzRVTUt8zOm9/8fm9+M88JpAg1IVcpo+fFy96+9Kv+128GBocywyMbod9QjJeYL31VdmjIpfB4SQsteTlQnLqO5JvO7mJnf7PJVSh875NuBXzbpTVPVAWjGqFKRtlLXGqaCy3zg2m7VNeVGwk3aOfKljlp2pJXdc6uKsoir7LSjry2zXZ8fZe6YpkT5n9S8RFS0bKVqNW1VclkSZ7EYT5MCkmShSRW/cwZ2LADPjBogAscPNCYS6AQ4rcFBSAQILYNEWIKMxHvc2hDP2obyOLIoIju4r+Gq60E9XDdqRnGaoanSBwKlSaMkx/khNyQC/KdXJM/j9aK4hodLy2cna6WB5Whvbfrv59VuThrqN+qnvSsoQpzsVeB3oMY6dyCdfXNz/s36/PF8eg9+Up+ov9DckXO8QZe8xf7tsaLB0/4cdALvhg2qHC/HQ+Tjal8YSY/vTadXfiYtCoNY/AOctiPWViAZViFEta/hL+pvlTaaBlfjCPjuEs1UolmFO6EcfoPe5i70A== (s) = imp(X) ⇣nL n · imp(L) + nR n · imp(R) ⌘ samples samples Split impurity after the split (the weighted mean of two impurities) AAACunichVG7SgNBFD2u73fURrBZDIoihIn4QixEGwsLjUYDiSy7m4kO2Re7k4AGf8AfsLBSsBA/wFpsRGsLP0EsFWwsvNksiK94l905c+49d8/MNTxLBJKxxwalsam5pbWtvaOzq7unN9bXvxW4Jd/kadO1XD9j6AG3hMPTUkiLZzyf67Zh8W2juFzNb5e5HwjX2ZT7Ht+x9V1HFISpS6K02LSjrao5M+9KNWfrcs+3K8L2DsdWx9UJ1dFSv+VS41oszhIsDPUnSEYgjijW3NgVcsjDhYkSbHA4kIQt6AjoySIJBo+4HVSI8wmJMM9xiA7SlqiKU4VObJG+u7TLRqxD+2rPIFSb9BeLXp+UKkbYA7tgL+yWXbIn9v5nr0rYo+pln1ajpuWe1ns0uPH2r8qmVWLvU1XXs0QBc6FXQd69kKmewqzpywfHLxvzqZHKKDtjz+T/lD2yGzqBU341z9d56qSOH4O80I3RgJLfx/ETbE0mkjOJqfWp+OJSNKo2DGEYYzSPWSxiBWtIU/8TXOMO98qCYihCKdZKlYZIM4AvocgPt/Okbg== nL · imp(L) + nR · imp(R) AAACE3icdVDLSgMxFM34rPU16rKbYBHaTZkp9bERirpwWcE+oFNKJs20oclkSDLiMHThR/gNbnXtTtz6AS79E9OHYH0cuHA4517uvcePGFXacd6thcWl5ZXVzFp2fWNza9ve2W0oEUtM6lgwIVs+UoTRkNQ11Yy0IkkQ9xlp+sPzsd+8IVJREV7rJCIdjvohDShG2khdO+dxpAeSp5EUt8mo4F0QplFBFYvwtGvnnVL50DGAv4lbcibIgxlqXfvD6wkccxJqzJBSbdeJdCdFUlPMyCjrxYpECA9Rn7QNDREnqpNOnhjBA6P0YCCkqVDDifp9IkVcqYT7pnN8svrpjcW/vHasg5NOSsMo1iTE00VBzKAWcJwI7FFJsGaJIQhLam6FeIAkwtrkNrfF5yOTydfj8H/SKJfco1LlqpKvns3SyYAc2AcF4IJjUAWXoAbqAIM78AAewZN1bz1bL9brtHXBms3sgTlYb588FJ5E proxy( (s)) = Proxy minimization No matter what the split is, the values of and remain unchanged. So we need to minimize the following proxy criterion. AAACB3icbVDLSsNAFL2pr1ofjbp0M1iEuimJ+FoW3bisYB/QhjKZTtqhM0mYmQgl9AP8Bre6didu/QyX/omTNgvbeuDC4Zx7uYfjx5wp7TjfVmFtfWNzq7hd2tnd2y/bB4ctFSWS0CaJeCQ7PlaUs5A2NdOcdmJJsfA5bfvju8xvP1GpWBQ+6klMPYGHIQsYwdpIfbvcE1iPpEiZiKfVzlnfrjg1Zwa0StycVCBHo2//9AYRSQQNNeFYqa7rxNpLsdSMcDot9RJFY0zGeEi7hoZYUOWls+BTdGqUAQoiaSbUaKb+vUixUGoifLOZxVTLXib+53UTHdx4KQvjRNOQzB8FCUc6QlkLaMAkJZpPDMFEMpMVkRGWmGjT1cIXX0xNJ+5yA6ukdV5zr2qXDxeV+m3eThGO4QSq4MI11OEeGtAEAgm8wCu8Wc/Wu/Vhfc5XC1Z+cwQLsL5+AZ1Wma0= imp(X) AAAB93icbVDLSgNBEOyNrxhfUY9eFoPgKeyKr2PQi8cEzAOSJcxOepMhM7PLzKywhHyBVz17E69+jkf/xEmyBxMtaCiquunuChPOtPG8L6ewtr6xuVXcLu3s7u0flA+PWjpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2O72d++wmVZrF8NFmCgSBDySJGibFSQ/bLFa/qzeH+JX5OKpCj3i9/9wYxTQVKQznRuut7iQkmRBlGOU5LvVRjQuiYDLFrqSQCdTCZHzp1z6wycKNY2ZLGnau/JyZEaJ2J0HYKYkZ61ZuJ/3nd1ES3wYTJJDUo6WJRlHLXxO7sa3fAFFLDM0sIVcze6tIRUYQam83SllBMbSb+agJ/Seui6l9XrxqXldpdnk4RTuAUzsGHG6jBA9ShCRQQnuEFXp3MeXPenY9Fa8HJZ45hCc7nD37Lk5I= n

Example 1: Calculating impurity improvement AAADIXichZHBTttAEIbHhhYaSgn0UqkXqxEVBxTNOo4TIyGh9tIDBwgNIIUost0ltXBsy3YigZUX6AtwaC8FcUB9AB6AS3urkDggnqDqkUpcOHTWcVVBVLore3/Pzjf7r8cKXCeKES8keWT0wcOx8Ue5iceTT6by0zPrkd8NbV63fdcPNy0z4q7j8XrsxC7fDEJudiyXb1g7r8X+Ro+HkeN7b+PdgDc7Zttzth3bjCnUyn9KttIijbBtNRMsYjrmsagaVZ0JUSkbpYrRV7y+sqgw2lNydxijStOg1DIiKzMSWFJ1o0RMa1lQJTY/xGCVqZo4R9eYJs4pG4xVWF8hqCYg3ci18oU/jpRhwTJRgGys+PkT2IJ34IMNXegABw9i0i6YENFsAAOEgGJNSCgWknLSfQ59yBHbpSxOGSZFd+jdpq9GFvXoW9SMUtqmU1x6QiIVmMVzPMYr/Ipf8Afe/LNWktYQXnZptQYsD1pTH56tXf+X6tAaw/u/1L2eY9iGaurVIe9BGhG3sAd8b2//am2hNpu8xAP8Sf4/4wWe0g283i/7aJXXPt7jxyIv9MeoQexuO4bFulpkelFb1QpLr7JWjcNzeAFz1I8KLMEbWIE62JIszUlMUuVD+Zv8XT4bpMpSxjyFW0O+/A3hp7rE n = 100, nL = 31, nR = 69 AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqmS8FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCkkOP2A== X AAAChHichVG7SgNBFD1ZNcZ31EawCQbFQsJE4wMLCdpYWJjERCGK7K6TOLjZXXYngRj8AW0VCysFC/ED/AAbf8AinyCWEWwsvNksiAb1LrNz5sw9d87M1WxDuJKxekDp6OwKdod6evv6BwaHwsMjOdcqOzrP6pZhOTua6nJDmDwrhTT4ju1wtaQZfFs7Wmvub1e44wrL3JJVm++V1KIpCkJXJVGpjf1wlMWYF5F2EPdBFH5sWuEH7OIAFnSUUQKHCUnYgAqXvjziYLCJ20ONOIeQ8PY5TtBL2jJlccpQiT2if5FWeZ81ad2s6XpqnU4xaDikjGCSPbM71mBP7J69sI9fa9W8Gk0vVZq1lpbb+0OnY5n3f1UlmiUOv1R/epYoYMnzKsi77THNW+gtfeX4spFZTk/WptgNeyX/16zOHukGZuVNv03x9NUffjTyQi9GDYr/bEc7yM3G4guxRCoRTa76rQphHBOYpn4sIol1bCJL9TnOcI4LJajMKHPKfCtVCfiaUXwLZeUTeMOPzA== L AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqmSoFQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkChYOP0g== R 31 samples (o 31, o 0, o 0) gini = 0.0 69 samples (o 0, o 36, o 33) gini = 0.499 100 samples (value o 31, o 36, o 33) gini = 0.665 AAAC53ichVHLShxBFL22mqgxcaIbwU3jYBghGW7P+BohIHHjIgtfo8L0ZOhua2YK+0V1zYA2/QNu3EXBlQEX4gfkA9z4A1n4CSJkY0AXWXi7p0ESiblFVZ06dc/tU31N3+aBRLzqUrp7el+87OsfeDX4+s1Q5u3wRuC1hMXKlmd7Yss0AmZzl5Ullzbb8gUzHNNmm+bOYny/2WYi4J67Lnd9VnWMhsvr3DIkUbWMqTuGbAon5I4f5UI9qVgRDbMaYr40R6P0HvPTiNq0RgCLhZlSMVI/R5PqR1VTP6i6zeoyp9eFYYVFLYqnLnijKSe/FCiFJLVMltYk1KdAS0EW0lj2Mt9Bh23wwIIWOMDABUnYBgMCGhXQAMEnrgohcYIQT+4ZRDBA2hZlMcowiN2htUGnSsq6dI5rBonaoq/YNAUpVZjAH3iGt3iJ53iNv/9ZK0xqxF52aTc7WubXhvZH1+7+q3Jol9B8VD3rWUId5hKvnLz7CRO/wuro23uHt2vzqxPhO/yGN+T/BK/wgl7gtn9Zpyts9fgZPyZ5oT9GDdL+bsdTsFHIazP5qZWp7MKntFV9MAbjkKN+zMICLMEylKn+JfyEO7hXuHKgfFWOOqlKV6oZgT9COXkAqJyxGg== imp(L) = 1 ✓ 31 31 ◆2 = 0.0 AAADOnichVE7b9NQFD42rxIeDbAgdbGIilIB1nHSOg1SpQILgqUP0kaK08g2N8lV/dL1TaRi+Q/wBxiYQGJAzIiFjYU/wNCVrerYSiwMHDtGqK1oj2Wf7373fMffvceJPB5LxF1FPXf+wsVLU5dLV65euz5dvnFzIw5HwmUtN/RC0XbsmHk8YC3JpcfakWC273hs09l+ku1vjpmIeRi8kDsR6/r2IOB97tqSqF75s+Xbcij8hPtRWk2svGNHDJxugjrmcR/1WnPRNDLQWGjWG81Ua6dz2pJmaA80y2N9WS1NktUXtpvUjTQxEFNL8MFQzm3VtHva0QLzrIL68YJSAZdQN82FXrny1512EhgFqEARK2H5C1jwEkJwYQQ+MAhAEvbAhpieDhiAEBHXhYQ4QYjn+wxSKJF2RFWMKmxit+k7oFWnYANaZz3jXO3SXzx6BSk1mMUf+BEP8Dt+wj38/d9eSd4j87JD2ZloWdSbfn17/deZKp+yhOE/1ameJfRhMffKyXuUM9kp3Il+/OrNwfrDtdnkLr7HffL/DnfxG50gGB+6H1bZ2ttT/DjkhW6MBmQcH8dJsFHTDVOfX52vLD8uRjUFM3AHqjSPBizDU1iBFrjKjPJIeaY8V7+qP9U9dX9SqiqF5hYcCfXwD3gkyyY= imp(X) = 1 ✓ 31 100 ◆2 + ✓ 36 100 ◆2 + ✓ 33 100 ◆2 ! = 0.665 AAADGHichVFNaxNRFL0Zv2r8aNSN4MKHoZKgDnfSMe0IhaIbl21q2kImhpnxJXl0vnjzEqjD/AG3LgRdKbgQf4BbwYV2W3DRnyAuK4jgwpvJgLTF+oaZe96599w571039kWiEPdK2omTp06fmTlbPnf+wsXZyqXL60k0kh5ve5EfyU3XSbgvQt5WQvl8M5bcCVyfb7hbDyb5jTGXiYjCR2o75t3AGYSiLzxHEdWrPLcDRw1lkIogzmqpnXfsyIHbTVG3cNFomLdRbzRNw0QCdy3DWDAy1srqbIkxg91hts/7qlaeBrsvHS+db2Zp08psKQZDVX/cYLfYwfz8oXy5gEuom5bVq1RRx3yxo8AoQBWKtRJVPoANTyACD0YQAIcQFGEfHEjo6YABCDFxXUiJk4REnueQQZm0I6riVOEQu0XfAe06BRvSftIzydUe/cWnV5KSwRx+xXe4j1/wPX7D3//sleY9Jl62KbpTLY97s8+urv38ryqgqGD4V3WsZwV9WMy9CvIe58zkFN5UP376Yn/tXmsuvYlv8Dv5f417+IlOEI5/eG9XeevVMX5c8kI3RgMyDo/jKFhv6EZTN1fN6vL9YlQzcA1uQI3msQDL8BBWoE39f5Wul2qluvZS+6h91nampVqp0FyBA0vb/QOas7/3 imp(R) = 1 ✓ 36 69 ◆2 + ✓ 33 69 ◆2 ! = 0.499 (0.665 → 0.344) Impurity Improvement = 0.321 AAACH3icbVDLSgMxFM34rPVVdekmWERdWGYKPhAKRRe6VLAP6IxDJs20oUlmSDJCGeYHBBd+gt/gVtfuRJcu/RExfSx8HbhwOOde7r0niBlV2rbfrYnJqemZ2dxcfn5hcWm5sLJaV1EiManhiEWyGSBFGBWkpqlmpBlLgnjASCPonQz8xjWRikbiUvdj4nHUETSkGGkj+YUtlyPdlTw9pYJm280deFSBzi50VcL9tFdxMhj7vauyXyjaJXsI+Jc4Y1Ksnt6cfLp3b+d+4cNtRzjhRGjMkFItx461lyKpKWYky7uJIjHCPdQhLUMF4kR56fChDG4apQ3DSJoSGg7V7xMp4kr1eWA6B+er395A/M9rJTo89FIq4kQTgUeLwoRBHcFBOrBNJcGa9Q1BWFJzK8RdJBHWJsMfWwKemUyc3wn8JfVyydkv7V2YcI7BCDmwDjbANnDAAaiCM3AOagCDW/AAHsGTdW89Wy/W66h1whrPrIEfsN6/AA6Ppeo= Gini(X) := 1 X k=1 p2 k The Gini impurity (31/100 * 0.0 + 69/100 * 0.499) = 0.344 AAAEH3icrVHLbtNAFL2xeZTyaAobJDYWUatEQJhJnIeRkCpgwaKLNiVtpDiKbHeSWPVL9iRSseYH+AGEWBWJBeID+AAWsEJsWOQTEMsisWHBtROESEnYdEb2nLlzzz1n5pqBY0eckHFGks+cPXd+6cLyxUuXr6xkV6/uRv4wtFjT8h0/bJlGxBzbY01uc4e1gpAZrumwPfPgYXK+N2JhZPveE34YsI5r9D27Z1sGx1B3NfNCf8QcbuSjgnJf0V2DD0I3tt1A5GM9Ld8O+2YnJkWSjtukWNLqVZqAWkUr1zShtERBuaPoDuvxvN4LDSueoWp1nBoyKoTQCkVAyqWqVhaK190UYjZ9vpInhG7t+3yh0flqm2j0lvJvi6ROS2qiWVWpmmhWNEprVCjosXH6HufKNURBD+3+gBe62dxvHeUkoFOQg+nY8rPvQId98MGCIbjAwAOO2AEDIpxtoEAgwFgHYoyFiOz0nIGAZeQOMYthhoHRA/z3cdeeRj3cJzWjlG2hioNfiEwF1sgX8oYck4/kLflKfs6tFac1Ei+HuJoTLgu6K8+u7/z4L8vFlcPgD2uhZw49qKdebfQepJHkFtaEP3r6/HjnXmMtXievyDf0f0TG5D3ewBt9t15vs8bLBX5M9IIvhg2is+04CXZLRVotqttqbuPBtFVLcANuQh77UYMNeAxb0ARLAmlduisR+Uj+IH+SP09SpcyUcw3+GvL4F7rTDME= (s) = imp(X) ⇣nL n · imp(L) + nR n · imp(R) ⌘ 0.665 Petal.Length o o o o o o o o o o o oo o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o oo o o o o o o o o o o o o o o o oo o 1 2 3 4 5 6 setosa versicolor virginica

Example 2: Changes in proxy for Iris dataset proxy of the impurity improvement impurity improvement

Class probabilities Text text Random Forest Gaussian Process Classifier Logistic Regression Output a probability between and (predict_proba) P(class=red) P(class=blue) = 1 - P(class=red) Class probability in classification

How to get final predicted values at leaf nodes Decision tree (class probability) = Histogram approximation by data-dependent partitions P(class=red) P(class=blue) = 1 - P(class=red) predict_proba(x) red blue 8/27 = 0.296 19/27 = 0.704 0.0 1.0 8/27 19/27 19 8 35 2

How to get final predicted values at leaf nodes predict_proba(x) / / / / Class probabilities = [ . , . , . . . ] The distribution of y for the training samples in the region where x fell Decision tree (class probability) = Histogram approximation by data-dependent partitions

How to get final predicted values at leaf nodes The sample average of the local region • A very conservative prediction (constant prediction = histogram approximation) based on very sloppy/rough partitions • In grid partitions, we have many regions that do not contain sample points. But, with data- dependent splitting, each region always contains sample points even in high dimensions! Regression tree = Histogram approximation by data-dependent partitions

How to get final predicted values at leaf nodes ≤ > Return the average y of the training samples in the region where x fell as the predicted value Regression tree = Histogram approximation by data-dependent partitions

Stopping condition: When should the recursion be stopped? Stopping condition for recursion • When the depth of the tree reaches the specified limit (max_depth). • When the number of leaf nodes reaches the specified limit (max_leaf_nodes). • When the improvement in impurity after splitting is sufficiently small (min_impurity_decrease). • When the impurity of the region is sufficiently low (min_impurity_split). • When the number of samples in the region is sufficiently small (min_samples_split). • When the number of samples in the leaf node reaches the lower limit (min_samples_leaf). • When the sample size ratio of the node after splitting reaches the lower limit (min_weight_fraction_leaf). Or stop to prevent the partitions from becoming too fine or unbalanced Don't stop (fully grown trees) ➡ Recursion continues until the training error becomes zero. In the worst case, it stops at a leaf node with a sample size of 1

Stop based on the size of the tree. • max_depth • max_leaf_nodes • min_impurity_decrease • min_impurity_split • min_samples_leaf • min_samples_split • min_weight_fraction_leaf AAAClnichVHLSsNAFD2N73fUjeAmWCquZCKC4kJEEV22am2hLSGJow3Ny2Ra0eIP+AMuxIWCivgBfoAbf8CFnyAuFdy48DYNiIp6w2TOnLnnzpm5hm9boWDsMSG1tLa1d3R2dff09vUPyINDm6FXDUyeNT3bC/KGHnLbcnlWWMLmeT/gumPYPGdUlhr7uRoPQstzN8S+z0uOvuNa25apC6I0eTCvqUrR5rtKUZS50DVVk5NskkWh/ARqDJKII+3JtyhiCx5MVOGAw4UgbENHSF8BKhh84kqoExcQsqJ9jkN0k7ZKWZwydGIr9N+hVSFmXVo3aoaR2qRTbBoBKRWk2AO7Zi/snt2wJ/b+a616VKPhZZ9mo6nlvjZwNLL+9q/KoVmg/Kn607PANmYjrxZ59yOmcQuzqa8dHL+sz62l6uPsnD2T/zP2yO7oBm7t1bzI8LWTP/wY5IVejBqkfm/HT7A5NakSzkwnFxbjVnViFGOYoH7MYAGrSCNL9fdwiktcSSPSvLQsrTRTpUSsGcaXkNIfLpKWEA== AAAClnichVHLSsNAFD3GV62vVjeCm2JRXMmkCooLKYrosg+rBSshiWMbTJOYTCu1+AP+gAtxoaAifoAf4MYfcOEniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGHtsk9o7Oru6Iz3R3r7+gcFYfGjds2uuzgu6bdpuUVM9bhoWLwhDmLzouFytaibf0HaX/P2NOnc9w7bWRMPhW1W1bBk7hq4KopRYvKikEiWT7yVKosKFqkwrsSSbYkEkfgI5BEmEkbFjtyhhGzZ01FAFhwVB2IQKj75NyGBwiNtCkziXkBHscxwiStoaZXHKUIndpX+ZVpsha9Har+kFap1OMWm4pExgnD2wa/bC7tkNe2Lvv9ZqBjV8Lw2atZaWO8rg0Uj+7V9VlWaByqfqT88CO5gLvBrk3QkY/xZ6S18/OH7Jz+fGmxPsnD2T/zP2yO7oBlb9Vb/I8tzJH3408kIvRg2Sv7fjJ1hPTcmEszPJ9GLYqghGMYZJ6scs0lhFBgWqv49TXOJKGpEWpGVppZUqtYWaYXwJKfMBNQCWEw== AAAClnichVHLSsNAFD2Nr1pfrW4KbopFcSXTIigupCiiyz6sFqyEJE7bYF4m00ot/oA/4EJcKKiIH+AHuPEHXPgJ4lLBjQtv04CoqDdM5syZe+6cmas6hu4Jxh5DUld3T29fuD8yMDg0PBKNjW54dt3VeFGzDdstqYrHDd3iRaELg5cclyumavBNdXe5vb/Z4K6n29a6aDp821Sqll7RNUUQJUdjJTmdKBt8L1EWNS4UOS1Hk2yG+ZH4CVIBSCKIrB29RRk7sKGhDhMcFgRhAwo8+raQAoND3DZaxLmEdH+f4xAR0tYpi1OGQuwu/au02gpYi9btmp6v1ugUg4ZLygQm2QO7Zi/snt2wJ/b+a62WX6PtpUmz2tFyRx45ihfe/lWZNAvUPlV/ehaoYN73qpN3x2fat9A6+sbB8UthIT/ZmmLn7Jn8n7FHdkc3sBqv2kWO50/+8KOSF3oxalDqezt+go30TIpwbjaZWQpaFcY4JjBN/ZhDBmvIokj193GKS1xJcWlRWpFWO6lSKNCM4UtI2Q8y4JYS AAAClnichVHLSsNAFD3GV323uim4KZaKK5lIQXEhRRFdtmproS0hiaOGpklMppVa/AF/wIW4UFARP8APcOMPuOgniEsFNy68TQOiRb1hMmfO3HPnzFzNMQ1PMNbskrp7evv6QwODQ8Mjo2PhyHjOs6uuzrO6bdpuXlM9bhoWzwpDmDzvuFytaCbf1sorrf3tGnc9w7a2RN3hpYq6Zxm7hq4KopRwJK/IsaLJD2JFsc+FqiSVcJzNMj9inUAOQBxBpO3wPYrYgQ0dVVTAYUEQNqHCo68AGQwOcSU0iHMJGf4+xzEGSVulLE4ZKrFl+u/RqhCwFq1bNT1frdMpJg2XlDEk2BO7Za/skd2xZ/bxa62GX6PlpU6z1tZyRxk7iW6+/6uq0Cyw/6X607PALhZ8rwZ5d3ymdQu9ra8dnb5uLm4kGtPskr2Q/wvWZA90A6v2pl9l+MbZH3408kIvRg2Sf7ajE+TmZmXCmWQ8tRy0KoRJTGGG+jGPFNaRRpbqH+Ic17iRotKStCqttVOlrkAzgW8hpT8BNPKWEw== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhVmRYlYBW0s8zAPiCHsrpO4ZF/sbgIx+AOCrSmsFCzED/ADbPwBi3yCWEawsfBmsyAajHeZnTNn7rlzZq5saarjMtYLCBOTU9MzwdnQ3PzCYjiytFxwzKat8LxiaqZdkiWHa6rB867qarxk2VzSZY0X5cbBYL/Y4rajmsaR27Z4RZfqhlpTFcklKqdVxWokxhLMi+goEH0Qgx9pM/KIY5zAhIImdHAYcAlrkODQV4YIBou4CjrE2YRUb5/jHCHSNimLU4ZEbIP+dVqVfdag9aCm46kVOkWjYZMyijh7Yfesz57ZA3tln3/W6ng1Bl7aNMtDLbeq4YvV3Me/Kp1mF6ffqrGeXdSw63lVybvlMYNbKEN966zbz+1l450NdsveyP8N67EnuoHRelfuMjx7PcaPTF7oxahB4u92jILCVkIknNmOpfb9VgWxhnVsUj+SSOEQaeSpfh2XuEJXCAoJYUdIDlOFgK9ZwY8QUl84LpCH AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmQYlYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKqUSScG4rltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBOk6QiA== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmPohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYLCGPjE5NTwenQzOzcfDiysFi0jaYl84JsqIZVlkSbq4rOC47iqLxsWlzUJJWXpMa+u19qcctWDP3QaZu8qol1XTlRZNEhKq/WNmuRGEswL6LDIOmDGPzIGJFHHOEYBmQ0oYFDh0NYhQibvgqSYDCJq6JDnEVI8fY5zhEibZOyOGWIxDboX6dVxWd1Wrs1bU8t0ykqDYuUUcTZC7tnffbMHtgr+/yzVser4Xpp0ywNtNyshS+W8x//qjSaHZx+q0Z6dnCCHc+rQt5Nj3FvIQ/0rbNuP7+bi3fW2C17I/83rMee6AZ6612+y/Lc9Qg/EnmhF6MGJX+3YxgUNxJJwtmtWHrPb1UQK1jFOvUjhTQOkEGB6tdxiSt0haCQELaF1CBVCPiaJfwIIf0FPG6QiQ== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmEolYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4VwqltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBPo6Qig== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmYohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4dx2LLPntyqINaxjk/qRRgYHyKJI9Ru4xBV6QlBICCkhPUwVAr5mBT9CyHwBQK6Qiw== AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmjaAEi6AIlnmYByQh7K6TZMi+2J0EYvAHLG0sYqNgIX6AH2DjD1jkE8Qygo2FdzcLosF4l9k5c+aeO2fmyqbKbE7IwCdMTc/MzvnnAwuLwaVQeHmlYBttS6F5xVANqyRLNlWZTvOccZWWTItKmqzSotw6dPaLHWrZzNBPeNekVU1q6KzOFIkjVarwJuVSLVELR0mcuBEZB6IHouBF2gg/QgVOwQAF2qABBR04YhUksPErgwgETOSq0EPOQsTcfQrnEEBtG7MoZkjItvDfwFXZY3VcOzVtV63gKSoOC5URiJEXck+G5Jk8kFfy+WetnlvD8dLFWR5pqVkLXazlPv5VaThzaH6rJnrmUIc91ytD76bLOLdQRvrO2dUwl8zGepvklryh/xsyIE94A73zrtxlaLY/wY+MXvDFsEHi73aMg8J2XESc2YmmDrxW+WEdNmAL+7ELKTiGNOTdPlxCH66FoJAQksL+KFXweZpV+BHC0RdPP5LB AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhXuqqAEi6AIlnmYByQh7K4Ts2Rf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCm82CaDDeZXbOnLnnzpm5sqWpDkfs+YSJyanpGf9sYG4+uBAKLy7lHbNpKyynmJppF2XJYZpqsBxXucaKls0kXdZYQW4cDPYLLWY7qmkc87bFKrp0aqg1VZE4UcUyrzMuVcVqOIpxdCMyCkQPRMGLlBl+hDKcgAkKNEEHBgZwwhpI4NBXAhEQLOIq0CHOJqS6+wzOIUDaJmUxypCIbdD/lFYljzVoPajpuGqFTtFo2KSMQAxf8B77+IwP+Iqff9bquDUGXto0y0Mts6qhi5Xsx78qnWYO9W/VWM8carDrelXJu+Uyg1soQ33r7KqfTWRinXW8xTfyf4M9fKIbGK135S7NMt0xfmTyQi9GDRJ/t2MU5DfjIuH0djS577XKD6uwBhvUjx1IwhGkIOf24RK6cC0EhS0hIewNUwWfp1mGHyEcfgFK/5K/ AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmo6AEi6AIlnmYByQh7K6TZMi+2J0EYvAHLG0sYqNgIX6AH2DjD1jkE8Qygo2FdzcLosF4l9k5c+aeO2fmyqbKbE7IwCdMTc/MzvnnAwuLwaVQeHmlYBttS6F5xVANqyRLNlWZTvOccZWWTItKmqzSotw6dPaLHWrZzNBPeNekVU1q6KzOFIkjVarwJuVSLVELR0mcuBEZB6IHouBF2gg/QgVOwQAF2qABBR04YhUksPErgwgETOSq0EPOQsTcfQrnEEBtG7MoZkjItvDfwFXZY3VcOzVtV63gKSoOC5URiJEXck+G5Jk8kFfy+WetnlvD8dLFWR5pqVkLXazlPv5VaThzaH6rJnrmUIc91ytD76bLOLdQRvrO2dUwl8zGepvklryh/xsyIE94A73zrtxlaLY/wY+MXvDFsEHi73aMg0IiLiLO7ERTB16r/LAOG7CF/diFFBxDGvJuHy6hD9dCUNgWksL+KFXweZpV+BHC0RdNH5LA AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNaAEi6AIlnmYByQh7K6TZMi+2J0EYvAHLG0sYqNgIX6AH2DjD1jkE8Qygo2FdzcLosF4l9k5c+aeO2fmyqbKbE7IwCdMTc/MzvnnAwuLwaVQeHmlYBttS6F5xVANqyRLNlWZTvOccZWWTItKmqzSotw6dPaLHWrZzNBPeNekVU1q6KzOFIkjVarwJuVSLVELR0mcuBEZB6IHouBF2gg/QgVOwQAF2qABBR04YhUksPErgwgETOSq0EPOQsTcfQrnEEBtG7MoZkjItvDfwFXZY3VcOzVtV63gKSoOC5URiJEXck+G5Jk8kFfy+WetnlvD8dLFWR5pqVkLXazlPv5VaThzaH6rJnrmUIc91ytD76bLOLdQRvrO2dUwl8zGepvklryh/xsyIE94A73zrtxlaLY/wY+MXvDFsEHi73aMg8J2XEScSURTB16r/LAOG7CF/diFFBxDGvJuHy6hD9dCUNgRksL+KFXweZpV+BHC0RdRX5LC AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhVmRYlYBW0s8zAPiCHsrpO4ZF/sbgIx+AOCrSmsFCzED/ADbPwBi3yCWEawsfBmsyAajHeZnTNn7rlzZq5saarjMtYLCBOTU9MzwdnQ3PzCYjiytFxwzKat8LxiaqZdkiWHa6rB867qarxk2VzSZY0X5cbBYL/Y4rajmsaR27Z4RZfqhlpTFcklKqdVxWokxhLMi+goEH0Qgx9pM/KIY5zAhIImdHAYcAlrkODQV4YIBou4CjrE2YRUb5/jHCHSNimLU4ZEbIP+dVqVfdag9aCm46kVOkWjYZMyijh7Yfesz57ZA3tln3/W6ng1Bl7aNMtDLbeq4YvV3Me/Kp1mF6ffqrGeXdSw63lVybvlMYNbKEN966zbz+1l450NdsveyP8N67EnuoHRelfuMjx7PcaPTF7oxahB4u92jILCVkIknNmOpfb9VgWxhnVsUj+SSOEQaeSpfh2XuEJXCAoJYUdIDlOFgK9ZwY8QUl84LpCH AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmQYlYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKqUSScG4rltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBOk6QiA== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmPohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYLCGPjE5NTwenQzOzcfDiysFi0jaYl84JsqIZVlkSbq4rOC47iqLxsWlzUJJWXpMa+u19qcctWDP3QaZu8qol1XTlRZNEhKq/WNmuRGEswL6LDIOmDGPzIGJFHHOEYBmQ0oYFDh0NYhQibvgqSYDCJq6JDnEVI8fY5zhEibZOyOGWIxDboX6dVxWd1Wrs1bU8t0ykqDYuUUcTZC7tnffbMHtgr+/yzVser4Xpp0ywNtNyshS+W8x//qjSaHZx+q0Z6dnCCHc+rQt5Nj3FvIQ/0rbNuP7+bi3fW2C17I/83rMee6AZ6612+y/Lc9Qg/EnmhF6MGJX+3YxgUNxJJwtmtWHrPb1UQK1jFOvUjhTQOkEGB6tdxiSt0haCQELaF1CBVCPiaJfwIIf0FPG6QiQ== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmEolYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4VwqltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBPo6Qig== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmYohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4dx2LLPntyqINaxjk/qRRgYHyKJI9Ru4xBV6QlBICCkhPUwVAr5mBT9CyHwBQK6Qiw== depth leaf yes no yes no yes no yes no Stopping condition: When should the recursion be stopped? Stop based on impurity Stop based on the number of samples in the region

• max_depth • max_leaf_nodes • min_samples_leaf • min_samples_split • min_weight_fraction_leaf How many regions (leaf nodes) should it be divided into? How many sample points should each region (leaf node) contain at a minimum? Stopping condition: When should the recursion be stopped? Stop based on the size of the tree Stop based on impurity Stop based on the number of samples in the region • min_impurity_decrease • min_impurity_split

Scikit-learn's default setting is to let the tree grow fully. from sklearn.tree import DecisionTreeClassifier DecisionTreeClassifier().get_params() from sklearn.tree import DecisionTreeRegressor DecisionTreeRegressor().get_params() {'ccp_alpha': 0.0, 'class_weight': None, 'criterion': 'gini', 'max_depth': None, 'max_features': None, 'max_leaf_nodes': None, 'min_impurity_decrease': 0.0, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'random_state': None, 'splitter': 'best'} {'ccp_alpha': 0.0, 'criterion': 'squared_error', 'max_depth': None, 'max_features': None, 'max_leaf_nodes': None, 'min_impurity_decrease': 0.0, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'random_state': None, 'splitter': 'best'} Note: In Scikit-learn, if max_leaf_nodes is set, it expands child nodes in a best-first manner. If not set, it expands in a depth-first manner. Stopping condition: When should the recursion be stopped?

In other words, be careful because even for completely random training data, we can (easily) have the zero training error! (Of course, it’s overfitting though) from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import accuracy_score import numpy as np X = np.random.rand(100, 3) y = np.random.randint(2, size=100) model = DecisionTreeClassifier() model.fit(X, y) y_pred = model.predict(X) from sklearn.tree import DecisionTreeRegressor from sklearn.metrics import mean_squared_error import numpy as np X = np.random.rand(100, 3) y = np.random.rand(100) model = DecisionTreeRegressor() model.fit(X, y) y_pred = model.predict(X) accuracy_score(y, y_pred) model.get_n_leaves() mean_squared_error(y, y_pred) model.get_n_leaves() 0.0 100 1.0 36 model.get_depth() 20 model.get_depth() 10 Stopping condition: When should the recursion be stopped?

Optimality concern: Is the greedy method OK? TDIDT (Top-Down Induction of Decision Trees) plus pruning is the mainstream for learning a single decision tree (to prevent the horizon effect) • TDIDT is a greedy method: The best split is chosen at each stage of recursion, and the final solution is a combination of these partial solutions. → It does not consider global optimality • But learning an optimal tree out of all possible trees is known to be NP-Hard • Since the optimal tree for the training data is almost never optimal for the test data, the actual benefit obtained by working hard on costly combinatorial optimization is small This is why classic TDIDT (+post-pruning) is still the standard approach: starting with building a larger tree, and then prune unnecessary branches to shape the tree

Controlling the model complexity of decision trees If the model complexity is too low, it results in underfitting. If the model complexity is too high, it results in overfitting. Underfitting Overfitting The basics of decision trees: Control by pruning the overgrown decision tree. Caveat: However, when used in ensemble learning, pruning is often not performed for each individual decision tree. Controlling the overall complexity of the ensemble is important.

Cost-Complexity Pruning (CCP) AAACBHicbVC7SgNBFJ31GeMramkzGITYhF0JaiME01hGyAuSNdydTJIhM7PLzKwQlrR+g63WdmLrf1j6J06SLUzigQuHc+7lXE4QcaaN6347a+sbm1vbmZ3s7t7+wWHu6Lihw1gRWichD1UrAE05k7RumOG0FSkKIuC0GYwqU7/5RJVmoayZcUR9AQPJ+oyAsdJjpdsBHg2hULvAt7iby7tFdwa8SryU5FGKajf30+mFJBZUGsJB67bnRsZPQBlGOJ1kO7GmEZARDGjbUgmCaj+ZfT3B51bp4X6o7EiDZ+rfiwSE1mMR2E0BZqiXvan4n9eOTf/GT5iMYkMlmQf1Y45NiKcV4B5TlBg+tgSIYvZXTIaggBhb1EJKICa2E2+5gVXSuCx6V8XSQylfvkvbyaBTdIYKyEPXqIzuURXVEUEKvaBX9OY8O+/Oh/M5X11z0psTtADn6xdZn5fL C↵(T) = ccp_alpha in scikit-learn Pruning of decision trees Q Q Q Yes No Yes No Yes No Q Q Yes No Yes No ⾐ https://scikit-learn.org/stable/auto_examples/tree/plot_cost_complexity_pruning.html Prediction Error + α × # of leaf nodes Minimizing a penalized loss to prevent overfitting: 4 leaf nodes 3 leaf nodes Just revert to the state before that split

Weakest link pruning in CART • Determine the value of α for regularization by cross-validation • We can guarantee that for each α, we can always find a tree that minimizes in the following sequence of trees. Prune the leaf where the increase in error is the smallest a single-split tree (decision stump) AAACAXicbVC7SgNBFL3rM8ZX1NJmMAixCbviqwymsYyQF2yWMDuZJENmZ5aZWSEsqfwGW63txNYvsfRPnCRbmMQDFw7n3Mu994QxZ9q47reztr6xubWd28nv7u0fHBaOjptaJorQBpFcqnaINeVM0IZhhtN2rCiOQk5b4ag69VtPVGkmRd2MYxpEeCBYnxFsrORXux3M4yEu1S+6haJbdmdAq8TLSBEy1LqFn05PkiSiwhCOtfY9NzZBipVhhNNJvpNoGmMywgPqWypwRHWQzk6eoHOr9FBfKlvCoJn6dyLFkdbjKLSdETZDvexNxf88PzH9uyBlIk4MFWS+qJ9wZCSa/o96TFFi+NgSTBSztyIyxAoTY1Na2BJGE5uJt5zAKmlelr2b8vXjVbFyn6WTg1M4gxJ4cAsVeIAaNICAhBd4hTfn2Xl3PpzPeeuak82cwAKcr18aiJcx C↵(T)

Generalizing decision trees? x1 AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuK9gHtUDJppg1NMkOSEcvYT3Cre3fi1p9x65eYtrPQ1gMXDufcy7mcMBHcWN//8gorq2vrG8XN0tb2zu5eef+gaeJUU9agsYh1OySGCa5Yw3IrWDvRjMhQsFY4up76rQemDY/VvR0nLJBkoHjEKbFOunvs4V654lf9GdAywTmpQI56r/zd7cc0lUxZKogxHewnNsiItpwKNil1U8MSQkdkwDqOKiKZCbLZqxN04pQ+imLtRlk0U39fZEQaM5ah25TEDs2iNxX/9UK5kGyjyyDjKkktU3QeHKUC2RhNe0B9rhm1YuwIoZq73xEdEk2odW2VXCl4sYJl0jyrYr+Kb88rtau8niIcwTGcAoYLqMEN1KEBFAbwDC/w6j15b9679zFfLXj5zSH8gff5A0z4lEY= AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuK9gHtUDJppg1NMkOSEcvYT3Cre3fi1p9x65eYtrPQ1gMXDufcy7mcMBHcWN//8gorq2vrG8XN0tb2zu5eef+gaeJUU9agsYh1OySGCa5Yw3IrWDvRjMhQsFY4up76rQemDY/VvR0nLJBkoHjEKbFOunvs4V654lf9GdAywTmpQI56r/zd7cc0lUxZKogxHewnNsiItpwKNil1U8MSQkdkwDqOKiKZCbLZqxN04pQ+imLtRlk0U39fZEQaM5ah25TEDs2iNxX/9UK5kGyjyyDjKkktU3QeHKUC2RhNe0B9rhm1YuwIoZq73xEdEk2odW2VXCl4sYJl0jyrYr+Kb88rtau8niIcwTGcAoYLqMEN1KEBFAbwDC/w6j15b9679zFfLXj5zSH8gff5A0z4lEY= AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuK9gHtUDJppg1NMkOSEcvYT3Cre3fi1p9x65eYtrPQ1gMXDufcy7mcMBHcWN//8gorq2vrG8XN0tb2zu5eef+gaeJUU9agsYh1OySGCa5Yw3IrWDvRjMhQsFY4up76rQemDY/VvR0nLJBkoHjEKbFOunvs4V654lf9GdAywTmpQI56r/zd7cc0lUxZKogxHewnNsiItpwKNil1U8MSQkdkwDqOKiKZCbLZqxN04pQ+imLtRlk0U39fZEQaM5ah25TEDs2iNxX/9UK5kGyjyyDjKkktU3QeHKUC2RhNe0B9rhm1YuwIoZq73xEdEk2odW2VXCl4sYJl0jyrYr+Kb88rtau8niIcwTGcAoYLqMEN1KEBFAbwDC/w6j15b9679zFfLXj5zSH8gff5A0z4lEY= AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuK9gHtUDJppg1NMkOSEcvYT3Cre3fi1p9x65eYtrPQ1gMXDufcy7mcMBHcWN//8gorq2vrG8XN0tb2zu5eef+gaeJUU9agsYh1OySGCa5Yw3IrWDvRjMhQsFY4up76rQemDY/VvR0nLJBkoHjEKbFOunvs4V654lf9GdAywTmpQI56r/zd7cc0lUxZKogxHewnNsiItpwKNil1U8MSQkdkwDqOKiKZCbLZqxN04pQ+imLtRlk0U39fZEQaM5ah25TEDs2iNxX/9UK5kGyjyyDjKkktU3QeHKUC2RhNe0B9rhm1YuwIoZq73xEdEk2odW2VXCl4sYJl0jyrYr+Kb88rtau8niIcwTGcAoYLqMEN1KEBFAbwDC/w6j15b9679zFfLXj5zSH8gff5A0z4lEY= x2 AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047KifUA7lEyaaUOTzJBkxDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31vO+0Mrq2vrGZmGruL2zu7dfOjhsmjjVlDVoLGLdDolhgivWsNwK1k40IzIUrBWOrqd+64Fpw2N1b8cJCyQZKB5xSqyT7h571V6p7FW8GfAy8XNShhz1Xum7249pKpmyVBBjOr6X2CAj2nIq2KTYTQ1LCB2RAes4qohkJshmr07wqVP6OIq1G2XxTP19kRFpzFiGblMSOzSL3lT81wvlQrKNLoOMqyS1TNF5cJQKbGM87QH3uWbUirEjhGrufsd0SDSh1rVVdKX4ixUsk2a14nsV//a8XLvK6ynAMZzAGfhwATW4gTo0gMIAnuEFXtETekPv6GO+uoLymyP4A/T5A06MlEc= AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047KifUA7lEyaaUOTzJBkxDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31vO+0Mrq2vrGZmGruL2zu7dfOjhsmjjVlDVoLGLdDolhgivWsNwK1k40IzIUrBWOrqd+64Fpw2N1b8cJCyQZKB5xSqyT7h571V6p7FW8GfAy8XNShhz1Xum7249pKpmyVBBjOr6X2CAj2nIq2KTYTQ1LCB2RAes4qohkJshmr07wqVP6OIq1G2XxTP19kRFpzFiGblMSOzSL3lT81wvlQrKNLoOMqyS1TNF5cJQKbGM87QH3uWbUirEjhGrufsd0SDSh1rVVdKX4ixUsk2a14nsV//a8XLvK6ynAMZzAGfhwATW4gTo0gMIAnuEFXtETekPv6GO+uoLymyP4A/T5A06MlEc= AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047KifUA7lEyaaUOTzJBkxDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31vO+0Mrq2vrGZmGruL2zu7dfOjhsmjjVlDVoLGLdDolhgivWsNwK1k40IzIUrBWOrqd+64Fpw2N1b8cJCyQZKB5xSqyT7h571V6p7FW8GfAy8XNShhz1Xum7249pKpmyVBBjOr6X2CAj2nIq2KTYTQ1LCB2RAes4qohkJshmr07wqVP6OIq1G2XxTP19kRFpzFiGblMSOzSL3lT81wvlQrKNLoOMqyS1TNF5cJQKbGM87QH3uWbUirEjhGrufsd0SDSh1rVVdKX4ixUsk2a14nsV//a8XLvK6ynAMZzAGfhwATW4gTo0gMIAnuEFXtETekPv6GO+uoLymyP4A/T5A06MlEc= AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047KifUA7lEyaaUOTzJBkxDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31vO+0Mrq2vrGZmGruL2zu7dfOjhsmjjVlDVoLGLdDolhgivWsNwK1k40IzIUrBWOrqd+64Fpw2N1b8cJCyQZKB5xSqyT7h571V6p7FW8GfAy8XNShhz1Xum7249pKpmyVBBjOr6X2CAj2nIq2KTYTQ1LCB2RAes4qohkJshmr07wqVP6OIq1G2XxTP19kRFpzFiGblMSOzSL3lT81wvlQrKNLoOMqyS1TNF5cJQKbGM87QH3uWbUirEjhGrufsd0SDSh1rVVdKX4ixUsk2a14nsV//a8XLvK6ynAMZzAGfhwATW4gTo0gMIAnuEFXtETekPv6GO+uoLymyP4A/T5A06MlEc= n0 AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubd8mS3b1jd08IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSq4sb7/5RVWVtfWN4qbpa3tnd298v5B0ySZZthgiUh0K6IGBVfYsNwKbKUaqYwEPkTD66n/8Ija8ETd21GKoaR9xWPOqHXSner63XLFr/ozkGUS5KQCOerd8nenl7BMorJMUGPagZ/acEy15UzgpNTJDKaUDWkf244qKtGE49mrE3LilB6JE+1GWTJTf1+MqTRmJCO3KakdmEVvKv7rRXIh2caX4ZirNLOo2Dw4zgSxCZn2QHpcI7Ni5AhlmrvfCRtQTZl1bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/O4iUOw== AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubd8mS3b1jd08IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSq4sb7/5RVWVtfWN4qbpa3tnd298v5B0ySZZthgiUh0K6IGBVfYsNwKbKUaqYwEPkTD66n/8Ija8ETd21GKoaR9xWPOqHXSner63XLFr/ozkGUS5KQCOerd8nenl7BMorJMUGPagZ/acEy15UzgpNTJDKaUDWkf244qKtGE49mrE3LilB6JE+1GWTJTf1+MqTRmJCO3KakdmEVvKv7rRXIh2caX4ZirNLOo2Dw4zgSxCZn2QHpcI7Ni5AhlmrvfCRtQTZl1bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/O4iUOw== AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubd8mS3b1jd08IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSq4sb7/5RVWVtfWN4qbpa3tnd298v5B0ySZZthgiUh0K6IGBVfYsNwKbKUaqYwEPkTD66n/8Ija8ETd21GKoaR9xWPOqHXSner63XLFr/ozkGUS5KQCOerd8nenl7BMorJMUGPagZ/acEy15UzgpNTJDKaUDWkf244qKtGE49mrE3LilB6JE+1GWTJTf1+MqTRmJCO3KakdmEVvKv7rRXIh2caX4ZirNLOo2Dw4zgSxCZn2QHpcI7Ni5AhlmrvfCRtQTZl1bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/O4iUOw== AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubd8mS3b1jd08IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSq4sb7/5RVWVtfWN4qbpa3tnd298v5B0ySZZthgiUh0K6IGBVfYsNwKbKUaqYwEPkTD66n/8Ija8ETd21GKoaR9xWPOqHXSner63XLFr/ozkGUS5KQCOerd8nenl7BMorJMUGPagZ/acEy15UzgpNTJDKaUDWkf244qKtGE49mrE3LilB6JE+1GWTJTf1+MqTRmJCO3KakdmEVvKv7rRXIh2caX4ZirNLOo2Dw4zgSxCZn2QHpcI7Ni5AhlmrvfCRtQTZl1bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/O4iUOw== dn0 AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XPpr65QqyGw1UrdYgsmvIdd26IejCrTcc6NhojgpYouWXPwdhQrKYCk04VqrvoFR7OZaaEU6npUGmaIrJGA9p31CBY6q8fH7uFJ4ZJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuZUL4/h/+Tjms7yHZuq5Xm1TKeIjgBp+AcOOASNMENaIE2IGAMHsETeLYerBfr1XpbtBas5cwx+AXr/QsAXZZn AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XPpr65QqyGw1UrdYgsmvIdd26IejCrTcc6NhojgpYouWXPwdhQrKYCk04VqrvoFR7OZaaEU6npUGmaIrJGA9p31CBY6q8fH7uFJ4ZJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuZUL4/h/+Tjms7yHZuq5Xm1TKeIjgBp+AcOOASNMENaIE2IGAMHsETeLYerBfr1XpbtBas5cwx+AXr/QsAXZZn AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XPpr65QqyGw1UrdYgsmvIdd26IejCrTcc6NhojgpYouWXPwdhQrKYCk04VqrvoFR7OZaaEU6npUGmaIrJGA9p31CBY6q8fH7uFJ4ZJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuZUL4/h/+Tjms7yHZuq5Xm1TKeIjgBp+AcOOASNMENaIE2IGAMHsETeLYerBfr1XpbtBas5cwx+AXr/QsAXZZn AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XPpr65QqyGw1UrdYgsmvIdd26IejCrTcc6NhojgpYouWXPwdhQrKYCk04VqrvoFR7OZaaEU6npUGmaIrJGA9p31CBY6q8fH7uFJ4ZJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuZUL4/h/+Tjms7yHZuq5Xm1TKeIjgBp+AcOOASNMENaIE2IGAMHsETeLYerBfr1XpbtBas5cwx+AXr/QsAXZZn `5 AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKosegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AI70lYk= AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKosegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AI70lYk= AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKosegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AI70lYk= AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKosegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AI70lYk= `6 AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AJCIlYo= AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AJCIlYo= AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AJCIlYo= AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AJCIlYo= n4 AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWakoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSeui6ntV/65WqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AEHYlD8= AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWakoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSeui6ntV/65WqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AEHYlD8= AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWakoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSeui6ntV/65WqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AEHYlD8= AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWakoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSeui6ntV/65WqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AEHYlD8= dn4 AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSYxxXZXdOOygn1AG8JkMm2HTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+YcqY0Qh9WYW19Y3OruF3a2d3bPygfHrVVkklCWyThieyGWFHOBG1ppjntppLiOOS0E46vZ37nnkrFEnGnJyn1YzwUbMAI1kbqREEuAm8alCvIrteR51UhsqvIdd2aIejCrdUd6NhojgpYohmUP/tRQrKYCk04VqrnoFT7OZaaEU6npX6maIrJGA9pz1CBY6r8fH7uFJ4ZJYKDRJoSGs7VnxM5jpWaxKHpjLEeqVVvJv7phfHKZj2o+TkTaaapIIvFg4xDncBZFjBikhLNJ4ZgIpm5HZIRlphok1jJhPL9OfyftF3bQbZz61UaV8t4iuAEnIJz4IBL0AA3oAlagIAxeARP4Nl6sF6sV+tt0VqwljPH4Bes9y8GsZZr AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSYxxXZXdOOygn1AG8JkMm2HTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+YcqY0Qh9WYW19Y3OruF3a2d3bPygfHrVVkklCWyThieyGWFHOBG1ppjntppLiOOS0E46vZ37nnkrFEnGnJyn1YzwUbMAI1kbqREEuAm8alCvIrteR51UhsqvIdd2aIejCrdUd6NhojgpYohmUP/tRQrKYCk04VqrnoFT7OZaaEU6npX6maIrJGA9pz1CBY6r8fH7uFJ4ZJYKDRJoSGs7VnxM5jpWaxKHpjLEeqVVvJv7phfHKZj2o+TkTaaapIIvFg4xDncBZFjBikhLNJ4ZgIpm5HZIRlphok1jJhPL9OfyftF3bQbZz61UaV8t4iuAEnIJz4IBL0AA3oAlagIAxeARP4Nl6sF6sV+tt0VqwljPH4Bes9y8GsZZr AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSYxxXZXdOOygn1AG8JkMm2HTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+YcqY0Qh9WYW19Y3OruF3a2d3bPygfHrVVkklCWyThieyGWFHOBG1ppjntppLiOOS0E46vZ37nnkrFEnGnJyn1YzwUbMAI1kbqREEuAm8alCvIrteR51UhsqvIdd2aIejCrdUd6NhojgpYohmUP/tRQrKYCk04VqrnoFT7OZaaEU6npX6maIrJGA9pz1CBY6r8fH7uFJ4ZJYKDRJoSGs7VnxM5jpWaxKHpjLEeqVVvJv7phfHKZj2o+TkTaaapIIvFg4xDncBZFjBikhLNJ4ZgIpm5HZIRlphok1jJhPL9OfyftF3bQbZz61UaV8t4iuAEnIJz4IBL0AA3oAlagIAxeARP4Nl6sF6sV+tt0VqwljPH4Bes9y8GsZZr AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSYxxXZXdOOygn1AG8JkMm2HTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+YcqY0Qh9WYW19Y3OruF3a2d3bPygfHrVVkklCWyThieyGWFHOBG1ppjntppLiOOS0E46vZ37nnkrFEnGnJyn1YzwUbMAI1kbqREEuAm8alCvIrteR51UhsqvIdd2aIejCrdUd6NhojgpYohmUP/tRQrKYCk04VqrnoFT7OZaaEU6npX6maIrJGA9pz1CBY6r8fH7uFJ4ZJYKDRJoSGs7VnxM5jpWaxKHpjLEeqVVvJv7phfHKZj2o+TkTaaapIIvFg4xDncBZFjBikhLNJ4ZgIpm5HZIRlphok1jJhPL9OfyftF3bQbZz61UaV8t4iuAEnIJz4IBL0AA3oAlagIAxeARP4Nl6sF6sV+tt0VqwljPH4Bes9y8GsZZr n1 AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubvWTJ7t6x+04IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSqFRd//8gorq2vrG8XN0tb2zu5eef+gaZPMMN5giUxMK6KWS6F5AwVK3koNpyqS/CEaXk/9h0durEj0PY5SHira1yIWjKKT7nQ36JYrftWfgSyTICcVyFHvlr87vYRlimtkklrbDvwUwzE1KJjkk1InszylbEj7vO2oporbcDx7dUJOnNIjcWLcaCQz9ffFmCprRypym4riwC56U/FfL1ILyRhfhmOh0wy5ZvPgOJMEEzLtgfSE4QzlyBHKjHC/EzaghjJ0bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/PRyUPA== AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubvWTJ7t6x+04IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSqFRd//8gorq2vrG8XN0tb2zu5eef+gaZPMMN5giUxMK6KWS6F5AwVK3koNpyqS/CEaXk/9h0durEj0PY5SHira1yIWjKKT7nQ36JYrftWfgSyTICcVyFHvlr87vYRlimtkklrbDvwUwzE1KJjkk1InszylbEj7vO2oporbcDx7dUJOnNIjcWLcaCQz9ffFmCprRypym4riwC56U/FfL1ILyRhfhmOh0wy5ZvPgOJMEEzLtgfSE4QzlyBHKjHC/EzaghjJ0bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/PRyUPA== AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubvWTJ7t6x+04IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSqFRd//8gorq2vrG8XN0tb2zu5eef+gaZPMMN5giUxMK6KWS6F5AwVK3koNpyqS/CEaXk/9h0durEj0PY5SHira1yIWjKKT7nQ36JYrftWfgSyTICcVyFHvlr87vYRlimtkklrbDvwUwzE1KJjkk1InszylbEj7vO2oporbcDx7dUJOnNIjcWLcaCQz9ffFmCprRypym4riwC56U/FfL1ILyRhfhmOh0wy5ZvPgOJMEEzLtgfSE4QzlyBHKjHC/EzaghjJ0bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/PRyUPA== AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubvWTJ7t6x+04IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSqFRd//8gorq2vrG8XN0tb2zu5eef+gaZPMMN5giUxMK6KWS6F5AwVK3koNpyqS/CEaXk/9h0durEj0PY5SHira1yIWjKKT7nQ36JYrftWfgSyTICcVyFHvlr87vYRlimtkklrbDvwUwzE1KJjkk1InszylbEj7vO2oporbcDx7dUJOnNIjcWLcaCQz9ffFmCprRypym4riwC56U/FfL1ILyRhfhmOh0wy5ZvPgOJMEEzLtgfSE4QzlyBHKjHC/EzaghjJ0bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/PRyUPA== dn1 AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XvjP1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcB8pZo AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XvjP1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcB8pZo AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XvjP1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcB8pZo AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XvjP1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcB8pZo `1 AAAB/XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNlJJxkzM7vMzAphCf6CV717E69+i1e/xEmyB00saCiquqmmokRwY33/y1tZXVvf2CxsFbd3dvf2SweHDROnmmGdxSLWrYgaFFxh3XIrsJVopDIS2IxGN1O/+Yja8Fjd23GCoaQDxfucUeukRgeF6AbdUtmv+DOQZRLkpAw5at3Sd6cXs1SiskxQY9qBn9gwo9pyJnBS7KQGE8pGdIBtRxWVaMJs9u2EnDqlR/qxdqMsmam/LzIqjRnLyG1Kaodm0ZuK/3qRXEi2/asw4ypJLSo2D+6ngtiYTKsgPa6RWTF2hDLN3e+EDammzLrCiq6UYLGCZdI4rwR+Jbi7KFev83oKcAwncAYBXEIVbqEGdWDwAM/wAq/ek/fmvXsf89UVL785gj/wPn8AiKSVhQ== AAAB/XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNlJJxkzM7vMzAphCf6CV717E69+i1e/xEmyB00saCiquqmmokRwY33/y1tZXVvf2CxsFbd3dvf2SweHDROnmmGdxSLWrYgaFFxh3XIrsJVopDIS2IxGN1O/+Yja8Fjd23GCoaQDxfucUeukRgeF6AbdUtmv+DOQZRLkpAw5at3Sd6cXs1SiskxQY9qBn9gwo9pyJnBS7KQGE8pGdIBtRxWVaMJs9u2EnDqlR/qxdqMsmam/LzIqjRnLyG1Kaodm0ZuK/3qRXEi2/asw4ypJLSo2D+6ngtiYTKsgPa6RWTF2hDLN3e+EDammzLrCiq6UYLGCZdI4rwR+Jbi7KFev83oKcAwncAYBXEIVbqEGdWDwAM/wAq/ek/fmvXsf89UVL785gj/wPn8AiKSVhQ== AAAB/XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNlJJxkzM7vMzAphCf6CV717E69+i1e/xEmyB00saCiquqmmokRwY33/y1tZXVvf2CxsFbd3dvf2SweHDROnmmGdxSLWrYgaFFxh3XIrsJVopDIS2IxGN1O/+Yja8Fjd23GCoaQDxfucUeukRgeF6AbdUtmv+DOQZRLkpAw5at3Sd6cXs1SiskxQY9qBn9gwo9pyJnBS7KQGE8pGdIBtRxWVaMJs9u2EnDqlR/qxdqMsmam/LzIqjRnLyG1Kaodm0ZuK/3qRXEi2/asw4ypJLSo2D+6ngtiYTKsgPa6RWTF2hDLN3e+EDammzLrCiq6UYLGCZdI4rwR+Jbi7KFev83oKcAwncAYBXEIVbqEGdWDwAM/wAq/ek/fmvXsf89UVL785gj/wPn8AiKSVhQ== AAAB/XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNlJJxkzM7vMzAphCf6CV717E69+i1e/xEmyB00saCiquqmmokRwY33/y1tZXVvf2CxsFbd3dvf2SweHDROnmmGdxSLWrYgaFFxh3XIrsJVopDIS2IxGN1O/+Yja8Fjd23GCoaQDxfucUeukRgeF6AbdUtmv+DOQZRLkpAw5at3Sd6cXs1SiskxQY9qBn9gwo9pyJnBS7KQGE8pGdIBtRxWVaMJs9u2EnDqlR/qxdqMsmam/LzIqjRnLyG1Kaodm0ZuK/3qRXEi2/asw4ypJLSo2D+6ngtiYTKsgPa6RWTF2hDLN3e+EDammzLrCiq6UYLGCZdI4rwR+Jbi7KFev83oKcAwncAYBXEIVbqEGdWDwAM/wAq/ek/fmvXsf89UVL785gj/wPn8AiKSVhQ== `2 AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukVat6btW7u6jUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/ijiVhg== AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukVat6btW7u6jUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/ijiVhg== AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukVat6btW7u6jUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/ijiVhg== AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukVat6btW7u6jUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/ijiVhg== n2 AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWaKoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSatW9b2qf3dRqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AD6wlD0= AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWaKoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSatW9b2qf3dRqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AD6wlD0= AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWaKoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSatW9b2qf3dRqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AD6wlD0= AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWaKoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSatW9b2qf3dRqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AD6wlD0= dn2 AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8Xvjv1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcDh5Zp AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8Xvjv1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcDh5Zp AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8Xvjv1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcDh5Zp AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8Xvjv1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcDh5Zp `3 AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL3zXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d1GpXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AIvMlYc= AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL3zXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d1GpXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AIvMlYc= AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL3zXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d1GpXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AIvMlYc= AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL3zXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d1GpXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AIvMlYc= `4 AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukdVH13Kp3V6vUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/jWCViA== AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukdVH13Kp3V6vUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/jWCViA== AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukdVH13Kp3V6vUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/jWCViA== AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukdVH13Kp3V6vUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/jWCViA== n3 AAAB+nicbVC7SgNBFL0bXzG+opY2g0GwCrsqaBm0sYxoHpAsYXYymwyZxzIzK4Q1n2CrvZ3Y+jO2fomTZAtNPHDhcM69nMuJEs6M9f0vr7Cyura+UdwsbW3v7O6V9w+aRqWa0AZRXOl2hA3lTNKGZZbTdqIpFhGnrWh0M/Vbj1QbpuSDHSc0FHggWcwItk66l73zXrniV/0Z0DIJclKBHPVe+bvbVyQVVFrCsTGdwE9smGFtGeF0UuqmhiaYjPCAdhyVWFATZrNXJ+jEKX0UK+1GWjRTf19kWBgzFpHbFNgOzaI3Ff/1IrGQbOOrMGMySS2VZB4cpxxZhaY9oD7TlFg+dgQTzdzviAyxxsS6tkqulGCxgmXSPKsGfjW4u6jUrvN6inAEx3AKAVxCDW6hDg0gMIBneIFX78l78969j/lqwctvDuEPvM8fQESUPg== AAAB+nicbVC7SgNBFL0bXzG+opY2g0GwCrsqaBm0sYxoHpAsYXYymwyZxzIzK4Q1n2CrvZ3Y+jO2fomTZAtNPHDhcM69nMuJEs6M9f0vr7Cyura+UdwsbW3v7O6V9w+aRqWa0AZRXOl2hA3lTNKGZZbTdqIpFhGnrWh0M/Vbj1QbpuSDHSc0FHggWcwItk66l73zXrniV/0Z0DIJclKBHPVe+bvbVyQVVFrCsTGdwE9smGFtGeF0UuqmhiaYjPCAdhyVWFATZrNXJ+jEKX0UK+1GWjRTf19kWBgzFpHbFNgOzaI3Ff/1IrGQbOOrMGMySS2VZB4cpxxZhaY9oD7TlFg+dgQTzdzviAyxxsS6tkqulGCxgmXSPKsGfjW4u6jUrvN6inAEx3AKAVxCDW6hDg0gMIBneIFX78l78969j/lqwctvDuEPvM8fQESUPg== AAAB+nicbVC7SgNBFL0bXzG+opY2g0GwCrsqaBm0sYxoHpAsYXYymwyZxzIzK4Q1n2CrvZ3Y+jO2fomTZAtNPHDhcM69nMuJEs6M9f0vr7Cyura+UdwsbW3v7O6V9w+aRqWa0AZRXOl2hA3lTNKGZZbTdqIpFhGnrWh0M/Vbj1QbpuSDHSc0FHggWcwItk66l73zXrniV/0Z0DIJclKBHPVe+bvbVyQVVFrCsTGdwE9smGFtGeF0UuqmhiaYjPCAdhyVWFATZrNXJ+jEKX0UK+1GWjRTf19kWBgzFpHbFNgOzaI3Ff/1IrGQbOOrMGMySS2VZB4cpxxZhaY9oD7TlFg+dgQTzdzviAyxxsS6tkqulGCxgmXSPKsGfjW4u6jUrvN6inAEx3AKAVxCDW6hDg0gMIBneIFX78l78969j/lqwctvDuEPvM8fQESUPg== AAAB+nicbVC7SgNBFL0bXzG+opY2g0GwCrsqaBm0sYxoHpAsYXYymwyZxzIzK4Q1n2CrvZ3Y+jO2fomTZAtNPHDhcM69nMuJEs6M9f0vr7Cyura+UdwsbW3v7O6V9w+aRqWa0AZRXOl2hA3lTNKGZZbTdqIpFhGnrWh0M/Vbj1QbpuSDHSc0FHggWcwItk66l73zXrniV/0Z0DIJclKBHPVe+bvbVyQVVFrCsTGdwE9smGFtGeF0UuqmhiaYjPCAdhyVWFATZrNXJ+jEKX0UK+1GWjRTf19kWBgzFpHbFNgOzaI3Ff/1IrGQbOOrMGMySS2VZB4cpxxZhaY9oD7TlFg+dgQTzdzviAyxxsS6tkqulGCxgmXSPKsGfjW4u6jUrvN6inAEx3AKAVxCDW6hDg0gMIBneIFX78l78969j/lqwctvDuEPvM8fQESUPg== dn3 AAAB/nicdVDLSsNAFJ34rPVVdelmsAiuwiRtsd0V3bisYB/QhjCZTNqhk0mYmQglFPwFt7p3J279Fbd+idOHoEUPXDiccy/33hOknCmN0Ie1tr6xubVd2Cnu7u0fHJaOjjsqySShbZLwRPYCrChngrY105z2UklxHHDaDcbXM797T6ViibjTk5R6MR4KFjGCtZG6oZ8LvzL1S2VkNxqoWq1BZNeQ67p1Q1DFrTcc6NhojjJYouWXPgdhQrKYCk04VqrvoFR7OZaaEU6nxUGmaIrJGA9p31CBY6q8fH7uFJ4bJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuaUL4/h/+Tjms7yHZuq+Xm1TKeAjgFZ+ACOOASNMENaIE2IGAMHsETeLYerBfr1XpbtK5Zy5kT8AvW+xcFHJZq AAAB/nicdVDLSsNAFJ34rPVVdelmsAiuwiRtsd0V3bisYB/QhjCZTNqhk0mYmQglFPwFt7p3J279Fbd+idOHoEUPXDiccy/33hOknCmN0Ie1tr6xubVd2Cnu7u0fHJaOjjsqySShbZLwRPYCrChngrY105z2UklxHHDaDcbXM797T6ViibjTk5R6MR4KFjGCtZG6oZ8LvzL1S2VkNxqoWq1BZNeQ67p1Q1DFrTcc6NhojjJYouWXPgdhQrKYCk04VqrvoFR7OZaaEU6nxUGmaIrJGA9p31CBY6q8fH7uFJ4bJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuaUL4/h/+Tjms7yHZuq+Xm1TKeAjgFZ+ACOOASNMENaIE2IGAMHsETeLYerBfr1XpbtK5Zy5kT8AvW+xcFHJZq AAAB/nicdVDLSsNAFJ34rPVVdelmsAiuwiRtsd0V3bisYB/QhjCZTNqhk0mYmQglFPwFt7p3J279Fbd+idOHoEUPXDiccy/33hOknCmN0Ie1tr6xubVd2Cnu7u0fHJaOjjsqySShbZLwRPYCrChngrY105z2UklxHHDaDcbXM797T6ViibjTk5R6MR4KFjGCtZG6oZ8LvzL1S2VkNxqoWq1BZNeQ67p1Q1DFrTcc6NhojjJYouWXPgdhQrKYCk04VqrvoFR7OZaaEU6nxUGmaIrJGA9p31CBY6q8fH7uFJ4bJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuaUL4/h/+Tjms7yHZuq+Xm1TKeAjgFZ+ACOOASNMENaIE2IGAMHsETeLYerBfr1XpbtK5Zy5kT8AvW+xcFHJZq AAAB/nicdVDLSsNAFJ34rPVVdelmsAiuwiRtsd0V3bisYB/QhjCZTNqhk0mYmQglFPwFt7p3J279Fbd+idOHoEUPXDiccy/33hOknCmN0Ie1tr6xubVd2Cnu7u0fHJaOjjsqySShbZLwRPYCrChngrY105z2UklxHHDaDcbXM797T6ViibjTk5R6MR4KFjGCtZG6oZ8LvzL1S2VkNxqoWq1BZNeQ67p1Q1DFrTcc6NhojjJYouWXPgdhQrKYCk04VqrvoFR7OZaaEU6nxUGmaIrJGA9p31CBY6q8fH7uFJ4bJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuaUL4/h/+Tjms7yHZuq+Xm1TKeAjgFZ+ACOOASNMENaIE2IGAMHsETeLYerBfr1XpbtK5Zy5kT8AvW+xcFHJZq • Recursive 'univariate partitioning' → Is any bisection acceptable? • Constant prediction in leaves → Are any non-constant leaves acceptable? AAACq3ichVE9S8NQFD2N35+tugguxVKpg/IiguJUdHGs1rZilZLE1xqbL5K0WKt/wMlN1EnBQfwZLv0DDv0J4qjg4uBNGhAtrTck77xz77nvvFzZ0lTHZawZEnp6+/oHBoeGR0bHxsORicmsY1ZshWcUUzPtXVlyuKYaPOOqrsZ3LZtLuqzxnFze8PK5Krcd1TR23JrFD3SpZKhFVZFcorKpRO3sZL4QibFF5ke0HYgBiCGIlBlpYB+HMKGgAh0cBlzCGiQ49OQhgsEi7gB14mxCqp/nOMcwaStUxalCIrZM3xLt8gFr0N7r6fhqhU7R6LVJGUWcvbBH9s4a7Im9sq+Ovep+D89LjVa5peVWIXwxnf78V6XT6uLoR9XVs4siVn2vKnm3fMa7hdLSV0+v3tNr2/H6HLtnb+T/jjXZM93AqH4oD1t8+7aLH5m8dP5jXj6ooBGKfwfWDrJLiyLhreVYcj0Y5iBmMIsETWwFSWwihQydcIxLXONGWBDSwp6w3yoVQoFmCr9C4N89C5ka Any model of

Extensions: Model trees and Multivariate trees ■ OC1/Oblique Tree (Murthy+ 1994) ■ Perceptron Tree (Utgoff , 1988) ■ Large Margin Tree (Wu+ 1999; Bennett+ 2000) ■ Margin Tree (Tibshirani & Hastie 2007) ■ Geometric Decision Tree (Manwani & Sastry, 2012) ■ HHCART (Wickramarachchi+, 2016) ■ M5/M5’ (Quinlan, 1992; Wang & Witten 1997) ■ RETIS (Karalic & Cestnik 1991) ■ SECRET (Dobra & Gehrke, 2002) ■ SMOTI (Malerba+ 2004) ■ MAUVE (Vens & Blockeel, 2006) ■ LLRT (Vogel, Asparouhov, Scheffer, 2007) ■ Cubist (Quinlan, 2011) AAAClnichVHLSsNAFD3Gd3201Y3gplgqrmQiguJCiiK6rNZqoZWQxLEdmiYxmVa0+AP+gAtxoaAifoAf4MYfcOEniEsFNy68TQOiot4wmTNn7rlzZq7hWsKXjD22Ke0dnV3dPb2Rvv6BwWgsPrThOzXP5DnTsRwvb+g+t4TNc1JIi+ddj+tVw+KbRmWxub9Z554vHHtd7rt8q6qXbLEjTF0SpcXieU0kiiW+myjKMpe6pmqxJJtkQSR+AjUESYSRcWK3KGIbDkzUUAWHDUnYgg6fvgJUMLjEbaFBnEdIBPsch4iQtkZZnDJ0Yiv0L9GqELI2rZs1/UBt0ikWDY+UCaTYA7tmL+ye3bAn9v5rrUZQo+lln2ajpeWuFj0ayb79q6rSLFH+VP3pWWIHs4FXQd7dgGnewmzp6wfHL9m5tVRjnJ2zZ/J/xh7ZHd3Arr+aF6t87eQPPwZ5oRejBqnf2/ETbExNqoRXp5PphbBVPRjFGCaoHzNIYwUZ5Kj+Hk5xiStlRJlXlpTlVqrSFmqG8SWUzAedy5ZD AAAClnichVHLSsNAFD2Nr/psq5uCm2BRXMlEBMWFFEV02VarBSshiWONTZOYTCu1+AP+gAtxoaAifoAf4MYfcOEniEsFNy68TQOiot4wmTNn7rlzZq7uWqYvGHuMSG3tHZ1d0e6e3r7+gVg8MbjmO1XP4HnDsRyvoGs+t0yb54UpLF5wPa5VdIuv6+WF5v56jXu+6dirou7yzYpWss1t09AEUWo8UVB35WKJ78lFscOFpk6q8RSbYEHIP4ESghTCyDjxWxSxBQcGqqiAw4YgbEGDT98GFDC4xG2iQZxHyAz2OQ7RQ9oqZXHK0Igt079Eq42QtWndrOkHaoNOsWh4pJQxyh7YNXth9+yGPbH3X2s1ghpNL3Wa9ZaWu2rsKLny9q+qQrPAzqfqT88C25gJvJrk3Q2Y5i2Mlr52cPyyMpsbbYyxc/ZM/s/YI7ujG9i1V+Miy3Mnf/jRyQu9GDVI+d6On2BtckIhnJ1KpefDVkUxjBGMUz+mkcYyMshT/X2c4hJXUlKakxalpVaqFAk1Q/gSUuYDohmWRQ== AAAClnichVHLSsNAFD3Gd3201Y3gplgqrmSqguJCRBFd1mof0JaQxLENTZOYTCta/AF/wIW4UFARP8APcOMPuPATxGUFNy68TQOixXrDZM6cuefOmbmqbeiuYOylS+ru6e3rHxgMDA2PjAZD4bG0a1Udjac0y7CcrKq43NBNnhK6MHjWdrhSUQ2eUcvrzf1MjTuubpm74sjmhYpSNPV9XVMEUXIonJXLkXyRH0TyosSFIs/LoSibZV5E2kHcB1H4kbBCD8hjDxY0VFEBhwlB2IACl74c4mCwiSugTpxDSPf2OU4QIG2VsjhlKMSW6V+kVc5nTVo3a7qeWqNTDBoOKSOIsWd2xxrsid2zV/b5Z626V6Pp5YhmtaXlthw8ndj5+FdVoVmg9K3q6FlgH0ueV5282x7TvIXW0teOzxo7y8lYfZpdsTfyf8le2CPdwKy9a9fbPHnewY9KXujFqEHx3+1oB+m52Tjh7YXo6prfqgFMYgoz1I9FrGILCaSo/iEucINbaUJakTakzVaq1OVrxvEjpMQXpmeWRw== AAAClnichVHLSsNAFD2Nr1pfrW4EN8VScSVTKSgupCiiy1atLbQlJHFsg2kSk2lFiz/gD7gQFwoq4gf4AW78ARf9BHGp4MaFt2lAtFhvmMyZM/fcOTNXtQ3dFYw1A1JPb1//QHAwNDQ8MjoWjozvuFbN0XhWswzLyauKyw3d5FmhC4PnbYcrVdXgOXV/tbWfq3PH1S1zWxzZvFRVyqa+p2uKIEoOR/KyES2W+UG0KCpcKHJSDsfYHPMi2gkSPojBj7QVfkARu7CgoYYqOEwIwgYUuPQVkACDTVwJDeIcQrq3z3GCEGlrlMUpQyF2n/5lWhV81qR1q6brqTU6xaDhkDKKOHtmd+yNPbF79sI+/6zV8Gq0vBzRrLa13JbHTie3Pv5VVWkWqHyrunoW2MOi51Un77bHtG6htfX147O3raXNeGOGXbFX8n/JmuyRbmDW37XrDN887+JHJS/0YtSgxO92dIKd+bkE4UwyllrxWxXEFKYxS/1YQAobSCNL9Q9xgRvcSpPSsrQmrbdTpYCvmcCPkNJfqrWWSQ== AAACmnichVHLSsNAFD3Gd31VRRB0USxVQZCJCIorqRvFjVprC1pCEqc1mBfJtKDBH/AHXLiq4EL7AX6AG3/AhZ8gLiu4ceFtGhAt1hsmc+bMPXfOzNVc0/AFYy8dUmdXd09vX39sYHBoeCQ+OnbgO2VP51ndMR0vr6k+Nw2bZ4UhTJ53Pa5amslz2ulGYz9X4Z5vOPa+OHN5wVJLtlE0dFUQpcQncoo8lzjSrCB/kVggwIWqyEo8yRZZGIlWIEcgiSh2nPgDjnAMBzrKsMBhQxA2ocKn7xAyGFziCgiI8wgZ4T7HBWKkLVMWpwyV2FP6l2h1GLE2rRs1/VCt0ykmDY+UCaTYM7tjdfbEauyVff5ZKwhrNLyc0aw1tdxVRi4nMx//qiyaBU6+VW09CxSxGno1yLsbMo1b6E195fyqnlnbSwWz7Ia9kf8qe2GPdAO78q7f7vK96zZ+NPJCL0YNkn+3oxUcLC3KhHeXk+vpqFV9mMIM5qkfK1jHJnaQpfoBqrhHTZqW0tKWtN1MlToizTh+hLT/BZu5lww= AAACmnichVHLSsNAFD3G97NVEQRdFIsPEMpUBcVVqRvFTVutLVQJSZzW0LxIpgUN/oA/4MKVggv1A/wAN/6Ai36CuFRw48LbNCBa1Bsmc+bMPXfOzFUdQ/cEY40OqbOru6e3r39gcGh4JBIdHdvz7Jqr8bxmG7ZbVBWPG7rF80IXBi86LldM1eAFtbrR3C/UuevptrUrjh1+YCoVSy/rmiKIkqMTBXl5Pravmn7xNLZIgAtFXpajcZZgQcTaQTIEcYSRsaP32MchbGiowQSHBUHYgAKPvhKSYHCIO4BPnEtID/Y5TjFA2hplccpQiK3Sv0KrUshatG7W9AK1RqcYNFxSxjDLntgNe2WP7I49s49fa/lBjaaXY5rVlpY7cuRscuf9X5VJs8DRl+pPzwJlrAVedfLuBEzzFlpLXz85f91Zz836c+yKvZD/S9ZgD3QDq/6mXWd57uIPPyp5oRejBiV/tqMd7C0lkoSzK/FUOmxVH6YwgwXqxypS2EQGearv4xK3uJOmpbS0JW23UqWOUDOObyHtfgKkXZcQ AAACmnichVHLSsNAFD2N7/qqiiDoolh8gFCmRVBciW4UN9W2tqASkjitwbxIpgUN/oA/4MJVBRfaD/AD3PgDLvoJ4lLBjQtv04Bosd4wmTNn7rlzZq7qGLonGGtEpK7unt6+/oHo4NDwyGhsbHzfsyuuxvOabdhuUVU8bugWzwtdGLzouFwxVYMX1NPN5n6hyl1Pt62cOHP4kamULb2ka4ogSo5NFuT0QvxQNf3iRXyJABeKnJZjCZZkQcTbQSoECYSRsWMPOMQxbGiowASHBUHYgAKPvgOkwOAQdwSfOJeQHuxzXCBK2gplccpQiD2lf5lWByFr0bpZ0wvUGp1i0HBJGccce2Z37I09sTp7YZ9/1vKDGk0vZzSrLS135NHLqezHvyqTZoGTb1VHzwIlrAZedfLuBEzzFlpLXz2/esuu7c358+yGvZL/GmuwR7qBVX3Xbnf53nUHPyp5oRejBqV+t6Md7KeTKcK7y4n1jbBV/ZjGLBapHytYxxYyyFN9HzXcoy7NSBvStrTTSpUioWYCP0LKfQGgC5cO AAACmnichVHLSsNAFD2N7/qqiiDoolh8gFCmUlBciW4UN9W2tqASkjitwbxIpgUN/oA/4MJVBRfaD/AD3PgDLvoJ4lLBjQtv04Bosd4wmTNn7rlzZq7qGLonGGtEpK7unt6+/oHo4NDwyGhsbHzfsyuuxvOabdhuUVU8bugWzwtdGLzouFwxVYMX1NPN5n6hyl1Pt62cOHP4kamULb2ka4ogSo5NFuT0QvxQNf3iRXyJABeKnJZjCZZkQcTbQSoECYSRsWMPOMQxbGiowASHBUHYgAKPvgOkwOAQdwSfOJeQHuxzXCBK2gplccpQiD2lf5lWByFr0bpZ0wvUGp1i0HBJGccce2Z37I09sTp7YZ9/1vKDGk0vZzSrLS135NHLqezHvyqTZoGTb1VHzwIlrAZedfLuBEzzFlpLXz2/esuu7c358+yGvZL/GmuwR7qBVX3Xbnf53nUHPyp5oRejBqV+t6Md7C8nU4R304n1jbBV/ZjGLBapHytYxxYyyFN9HzXcoy7NSBvStrTTSpUioWYCP0LKfQGor5cS AAACmnichVHLSsNAFD3G97NVEQRdFIsPEMpUFMVVqRvFTVutLVQJSZzW0LxIpgUN/oA/4MKVggv1A/wAN/6Ai36CuFRw48LbNCBa1Bsmc+bMPXfOzFUdQ/cEY40OqbOru6e3r39gcGh4JBIdHdvz7Jqr8bxmG7ZbVBWPG7rF80IXBi86LldM1eAFtbrR3C/UuevptrUrjh1+YCoVSy/rmiKIkqMTBXllPravmn7xNLZIgAtFXpGjcZZgQcTaQTIEcYSRsaP32MchbGiowQSHBUHYgAKPvhKSYHCIO4BPnEtID/Y5TjFA2hplccpQiK3Sv0KrUshatG7W9AK1RqcYNFxSxjDLntgNe2WP7I49s49fa/lBjaaXY5rVlpY7cuRscuf9X5VJs8DRl+pPzwJlrAVedfLuBEzzFlpLXz85f91Zz836c+yKvZD/S9ZgD3QDq/6mXWd57uIPPyp5oRejBiV/tqMd7C0lkoSzy/FUOmxVH6YwgwXqxypS2EQGearv4xK3uJOmpbS0JW23UqWOUDOObyHtfgKtAZcU AAACnnichVG7SgNBFD2u7/hI1EaxWQw+qjArgmIlCmIjGjUmYGTZXSfJkn25OwloEHt/wMJKwUIELf0AG3/Awk8QSwUbC282C6JivMvsnDlzz50zc3XPMgPB2FOL1NrW3tHZ1R3r6e3rjycGBrcDt+IbPGO4luvndC3glunwjDCFxXOezzVbt3hWLy/V97NV7gem62yJA4/v2lrRMQumoQmi1MRIVlUm5bxu13JHcr7I9+W8KHGhqYqaSLIUC0P+DZQIJBHFupu4Qx57cGGgAhscDgRhCxoC+naggMEjbhc14nxCZrjPcYQYaSuUxSlDI7ZM/yKtdiLWoXW9ZhCqDTrFouGTUsY4e2RX7JU9sGv2zD7+rFULa9S9HNCsN7TcU+Mnw5vv/6psmgVKX6qmngUKmAu9muTdC5n6LYyGvnp4+ro5vzFem2AX7IX8n7Mndk83cKpvxmWab5w18aOTF3oxapDysx2/wfZ0SiGcnkkuLEat6sIoxjBF/ZjFAlawjgzVP8YlbnArydKytCqtNVKllkgzhG8h5T4BYfKZHA== AAACnnichVG7SgNBFD2u72eiNorNYvBRyWwQFCtREBvxmQcksuyuk2RxX+5OAhrE3h+wsFKwEEFLP8DGH7DIJ4hlBBsLbzYLoqLeZXbOnLnnzpm5umeZgWCs1iK1trV3dHZ19/T29Q/E4oND6cAt+wZPGa7l+lldC7hlOjwlTGHxrOdzzdYtntH3lxv7mQr3A9N1dsShx3dtreiYBdPQBFFqfDSjJqfkvG5Xs8dyvsgP5LwocaGpSTWeYDMsDPknUCKQQBQbbvweeezBhYEybHA4EIQtaAjoy0EBg0fcLqrE+YTMcJ/jGD2kLVMWpwyN2H36F2mVi1iH1o2aQag26BSLhk9KGRPsiV2zOntkN+yZvf9aqxrWaHg5pFlvarmnxk5Htt/+Vdk0C5Q+VX96FihgPvRqkncvZBq3MJr6ytFZfXtha6I6yS7ZC/m/YDX2QDdwKq/G1SbfOv/Dj05e6MWoQcr3dvwE6eSMQnhzNrG4FLWqC2MYxzT1Yw6LWMUGUlT/BFe4xZ0kSyvSmrTeTJVaIs0wvoSU/QBmSJke AAACnnichVG7SgNBFD2u7/iK2ig2i8FHFSZGUKxEQWzEZ0zAhGV3ncQl+3J3EtAg9v6AhZVCChG09ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7mm4QvGnluk1rb2js6u7khPb1//QHRwaNd3Sp7OU7pjOl5GU31uGjZPCUOYPON6XLU0k6e14nJtP13mnm849o44cnnOUgu2kTd0VRClREfTSnJKzmpWJXMiZwv8UM6KAy5UJalEYyzOgpAbQSIEMYSx4UTvkcU+HOgowQKHDUHYhAqfvj0kwOASl0OFOI+QEexznCBC2hJlccpQiS3Sv0CrvZC1aV2r6QdqnU4xaXiklDHBntg1e2OP7Ia9sM8/a1WCGjUvRzRrdS13lYGzke2Pf1UWzQIH36qmngXymA+8GuTdDZjaLfS6vnx8/ra9sDVRmWRX7JX8X7Jn9kA3sMvvenWTb1008aORF3oxalDidzsawe5MPEF4cza2uBS2qgtjGMc09WMOi1jFBlJU/xRV3OJOkqUVaU1ar6dKLaFmGD9CynwBap6ZIA== AAACnnichVG7SgNBFD1Z389EbRSbxeCjkokEFCtREBvRJOYBiSy76yRZ3Je7k4AGsfcHLKwULETQ0g+w8Qcs/ASxVLCx8GazICrqXWbnzJl77pyZq7mm4QvGHiNSW3tHZ1d3T29f/8BgNDY0nPOdmqfzrO6YjlfQVJ+bhs2zwhAmL7geVy3N5Hltd6W5n69zzzcce0vsu3zbUiu2UTZ0VRClxMbySnJaLmlWo3Aolyp8Ty6JKheqklRicTbLgpB/gkQI4ghj04ndooQdONBRgwUOG4KwCRU+fUUkwOASt40GcR4hI9jnOEQvaWuUxSlDJXaX/hVaFUPWpnWzph+odTrFpOGRUsYke2CX7IXdsyv2xN5/rdUIajS97NOstbTcVaLHo5m3f1UWzQLVT9WfngXKWAi8GuTdDZjmLfSWvn5w8pJZTE82ptg5eyb/Z+yR3dEN7PqrfpHi6dM//GjkhV6MGpT43o6fIDc3myCcSsaXlsNWdWMcE5ihfsxjCWvYRJbqH+EC17iRZGlVWpc2WqlSJNSM4EtIhQ9u9Jki AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhVmRYlYBW0s8zAPiCHsrpO4ZF/sbgIx+AOCrSmsFCzED/ADbPwBi3yCWEawsfBmsyAajHeZnTNn7rlzZq5saarjMtYLCBOTU9MzwdnQ3PzCYjiytFxwzKat8LxiaqZdkiWHa6rB867qarxk2VzSZY0X5cbBYL/Y4rajmsaR27Z4RZfqhlpTFcklKqdUxWokxhLMi+goEH0Qgx9pM/KIY5zAhIImdHAYcAlrkODQV4YIBou4CjrE2YRUb5/jHCHSNimLU4ZEbIP+dVqVfdag9aCm46kVOkWjYZMyijh7Yfesz57ZA3tln3/W6ng1Bl7aNMtDLbeq4YvV3Me/Kp1mF6ffqrGeXdSw63lVybvlMYNbKEN966zbz+1l450NdsveyP8N67EnuoHRelfuMjx7PcaPTF7oxahB4u92jILCVkIknNmOpfb9VgWxhnVsUj+SSOEQaeSpfh2XuEJXCAoJYUdIDlOFgK9ZwY8QUl8k/JB+ AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmQYlYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqiDXU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKqUSScG4rltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBJxyQfw== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmPohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYLCGPjE5NTwenQzOzcfDiysFi0jaYl84JsqIZVlkSbq4rOC47iqLxsWlzUJJWXpMa+u19qcctWDP3QaZu8qol1XTlRZNEhKi/XNmuRGEswL6LDIOmDGPzIGJFHHOEYBmQ0oYFDh0NYhQibvgqSYDCJq6JDnEVI8fY5zhEibZOyOGWIxDboX6dVxWd1Wrs1bU8t0ykqDYuUUcTZC7tnffbMHtgr+/yzVser4Xpp0ywNtNyshS+W8x//qjSaHZx+q0Z6dnCCHc+rQt5Nj3FvIQ/0rbNuP7+bi3fW2C17I/83rMee6AZ6612+y/Lc9Qg/EnmhF6MGJX+3YxgUNxJJwtmtWHrPb1UQK1jFOvUjhTQOkEGB6tdxiSt0haCQELaF1CBVCPiaJfwIIf0FKTyQgA== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmEolYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqiDXU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4VwqltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBK1yQgQ== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmYohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqiDXU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4dx2LLPntyqINaxjk/qRRgYHyKJI9Ru4xBV6QlBICCkhPUwVAr5mBT9CyHwBLXyQgg== Model Trees: Constant prediction → Linear prediction Multivariate Trees: Axis-parallel splits → Multivariate linear splits

Takeaways for Part 1 “Of all the well-known learning methods, decision trees come closest to meeting the requirements for serving as an off-the-shelf procedure for data mining.” https://hastie.su.domains/Papers/ESLII.pdf Hastie, Tibshirani, Friedman, The elements of statistical learning (2009) 1. Natural handling of data of “mixed” type 2. Handling of missing values 3. Robustness to outliers in input space 4. Insensitive to monotone transformations of inputs 5. Computational scalability 6. Ability to deal with irrelevant inputs 7. Ability to extract linear combinations of features 8. Interpretability 9. Predictive Power "

Why we should learn decision tree ensembles? • What is a decision tree? (History and basic algorithm) • Why did it become so popular once in the past? (Advantages) Decision tree ensemble is one of the three key weapons in modern data science alongside linear models and deep learning. • Despite this, why did it fall out of favor? (Disadvantages) • What is ensemble learning? Why can it compensate for the shortcomings? • Practice and methods of decision tree ensembles [Part 1] [Part 2]

https://itakigawa.github.io/ A Modern Introduction to Decision Tree Ensembles Ichigaku Takigawa Institute for Liberal Arts and Sciences, Kyoto University Institute for Chemical Reaction Design and Discovery, Hokkaido University RIKEN Center for Advanced Intelligence Project Part 2

Outline We will jointly learn decision trees and ensemble learning! 1. Why we should learn decision trees and ensemble learning? 2. History of Decision Trees 3. Basics of Decision Trees 4. Ensemble Learning: Additive ensembles and decision forests 5. Random forest and Regularized gradient boosted trees 6. Practical data analysis with decision tree ensembles Part 1 Part 2

Recap: Takeaways for Part 1 “Of all the well-known learning methods, decision trees come closest to meeting the requirements for serving as an off-the-shelf procedure for data mining.” https://hastie.su.domains/Papers/ESLII.pdf Hastie, Tibshirani, Friedman, The elements of statistical learning (2009) 1. Natural handling of data of “mixed” type 2. Handling of missing values 3. Robustness to outliers in input space 4. Insensitive to monotone transformations of inputs 5. Computational scalability 6. Ability to deal with irrelevant inputs 7. Ability to extract linear combinations of features 8. Interpretability 9. Predictive Power

Why we should learn decision tree ensembles? • What is a decision tree? (History and basic algorithm) • Why did it become so popular once in the past? (Advantages) Decision tree ensemble is one of the three key weapons in modern data science alongside linear models and deep learning. • Despite this, why did it fall out of favor? (Disadvantages) • What is ensemble learning? Why can it compensate for the shortcomings? • Practice and methods of decision tree ensembles [Part 1] [Part 2]

Drawback (1): Non-smoothness The function surface of a decision tree is not smooth. (a histogram approximation.) Because of this, the prediction capability is bounded. Decision Trees Random Forest Extra Trees Nearest Neighbors Logistic Regression SVM Gaussian Process Neural Network Non-smooth models Smooth models

Drawback (1): Non-smoothness In high-dimensional real data, the assumption of smoothness isn’t that useful. Thus, this very conservative local constant approximation proves to be very effective. This drawback (non-smoothness) can be compensated by ensemble learning! For example It's very poor at capturing additive structures... (we get a unnecessarily deep tree)

Drawback (2): Instability The structure of the tree (partitions) changes significantly with just a slight change in the data. Especially for the cases of many variables. A decision tree trained with a random 90% of the data on the left.

Drawback (2): Instability Instability of decision trees High bias Low variance High bias High variance Low bias High variance Low bias Low variance Estimates Bias-variance tradeoff (a central problem in supervised learning) One cannot reduce both bias and variance simultaneously in predictions • Decision trees are a 'low bias, high variance' method, with the variance being especially large. • With just a slight difference in the data, you can get substantially different partitions, making the interpretation of decision trees somewhat unreliable. This drawback (instability) can be also compensated by ensemble learning! Truth

Drawback (3) & (4) : Fragmentation & Bias • Data fragmentation There is a bias in variable selection, such as variables with a large number of categories being more likely to be chosen for splitting • Bias in splitter selection • Opting for multi-way splits instead of binary splits can easily lead to fragmented regions • When the number of samples in each region becomes too small, it can lead to overfitting • One can set a lower limit on the number of samples in each region (min_samples_leaf) as a safeguard. The more recursive splits are performed, each region contains fewer samples… (So we cannot do any complicated statistical processing at the leaf nodes)

Model Trees: Constant prediction in leaves → Linear prediction Due to these downsides, easy extensions are impractical… Regression Trees M5 MARS Classification Trees OC1 Multivariate CART a broken line model that recursively bends straight lines Multivariate Trees: Axis-parallel splits → Multivariate linear splits Data (Regression) Data (Classification) ■ OC1/Oblique Tree (Murthy+ 1994) ■ Perceptron Tree (Utgoff , 1988) ■ Large Margin Tree (Wu+ 1999; Bennett+ 2000) ■ Margin Tree (Tibshirani & Hastie 2007) ■ Geometric Decision Tree (Manwani & Sastry, 2012) ■ HHCART (Wickramarachchi+, 2016) ■ M5/M5’ (Quinlan, 1992; Wang & Witten 1997) ■ RETIS (Karalic & Cestnik 1991) ■ SECRET (Dobra & Gehrke, 2002) ■ SMOTI (Malerba+ 2004) ■ MAUVE (Vens & Blockeel, 2006) ■ LLRT (Vogel, Asparouhov, Scheffer, 2007) ■ Cubist (Quinlan, 2011)

Outline We will jointly learn decision trees and ensemble learning! 1. Why we should learn decision trees and ensemble learning? 2. History of Decision Trees 3. Basics of Decision Trees 4. Ensemble Learning: Additive ensembles and decision forests 5. Random forest and Regularized gradient boosted trees 6. Practical data analysis with decision tree ensembles Part 1 Part 2

Ensemble learning: Two heads are better than one Predicting by multiple different models rather than a single model “Ensemble” = A group of separate things that contribute to a coordinated whole … Consensus building Single-model Prediction Multiple-model Prediction (Ensemble Learning) Model Model 1 Model 2 Model K

A brief showcase of ensemble learning examples 1. Voting/Averaging of different models For example, using the same data, we can train with Neural Net, SVM, and Random Forest, then use Voting or Averaging to produce the final prediction. 2. Voting/Averaging of different instances of the same model For example, we can train five Neural Nets with the same structure but different random initial values, then use Voting or Averaging to produce the final prediction (’Deep Ensemble') 3. Snapshot ensembling (for deep learning) After the Neural Network training has converged, the learning rate is altered to induce oscillations around a local solution, and snapshots of that process are taken for ensemble. e.g. torch.optim.swa_utils implements Stochastic Weight Averaging (SWA). 4. Stacking/Blending Predictions from base models (like decision trees, SVMs, etc.) are used as input for a meta- model that makes the final prediction.

sklearn.ensemble (Scikit-learn 1.3.0) https://scikit-learn.org/stable/modules/classes.html#module-sklearn.ensemble Methods we will cover hereafter A small note on Adaboost • Historically and theoretically important algorithm • Less hyperparameters but works well in practice • We will skip this because gradient boosted trees is a more flexible alternative to it.

Outline We will jointly learn decision trees and ensemble learning! 1. Why we should learn decision trees and ensemble learning? 2. History of Decision Trees 3. Basics of Decision Trees 4. Ensemble Learning: Additive ensembles and decision forests 5. Random forest and Regularized gradient boosted trees 6. Practical data analysis with decision tree ensembles Part 1 Part 2

Decision Tree Ensemble: Learning from multiple decision trees We need multiple decision trees from a single dataset Dataset How do we generate multiple trees? • “Randomize” (Bagging, Random Forest) Introduce some randomness into the learning of each tree to diversify and generate a set of trees. • Repeat 'random resampling from data → tree generation'. • Repeat 'using different random numbers and data each time to generate a tree'. Sequentially update the ensemble by adding a decision tree one by one to improves the prediction accuracy. • “Optimize” (Boosting)

Overcoming non-smoothness and instability DecisionTreeClassifier (max_depth=None, min_samples_leaf=1) DecisionTreeClassifier (max_depth=1) DecisionTreeClassifier (max_depth=5, min_samples_leaf=3) ExtraTreesClassifier (n_estimators=100, max_depth=None, min_samples_leaf=1) RandomForestClassifier (n_estimators=100, max_depth=None, min_samples_leaf=1) A single decision tree An ensemble of 100 decision trees XGBClassifier (n_estimators=100, max_depth=3)

Basic concept for ensembling: Additive ensemble An additive model that makes predictions by 'adding' multiple decision trees T1 AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuKfUE7lEyaaUOTzJBkhDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31ve/vMLa+sbmVnG7tLO7t39QPjxqmTjVlDVpLGLdCYlhgivWtNwK1kk0IzIUrB2Ob2d++5Fpw2PVsJOEBZIMFY84JdZJD40+7pcrftWfA60SnJMK5Kj3y9+9QUxTyZSlghjTxX5ig4xoy6lg01IvNSwhdEyGrOuoIpKZIJu/OkVnThmgKNZulEVz9fdFRqQxExm6TUnsyCx7M/FfL5RLyTa6DjKuktQyRRfBUSqQjdGsBzTgmlErJo4Qqrn7HdER0YRa11bJlYKXK1glrYsq9qv4/rJSu8nrKcIJnMI5YLiCGtxBHZpAYQjP8AKv3pP35r17H4vVgpffHMMfeJ8/E+CUIg== AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuKfUE7lEyaaUOTzJBkhDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31ve/vMLa+sbmVnG7tLO7t39QPjxqmTjVlDVpLGLdCYlhgivWtNwK1kk0IzIUrB2Ob2d++5Fpw2PVsJOEBZIMFY84JdZJD40+7pcrftWfA60SnJMK5Kj3y9+9QUxTyZSlghjTxX5ig4xoy6lg01IvNSwhdEyGrOuoIpKZIJu/OkVnThmgKNZulEVz9fdFRqQxExm6TUnsyCx7M/FfL5RLyTa6DjKuktQyRRfBUSqQjdGsBzTgmlErJo4Qqrn7HdER0YRa11bJlYKXK1glrYsq9qv4/rJSu8nrKcIJnMI5YLiCGtxBHZpAYQjP8AKv3pP35r17H4vVgpffHMMfeJ8/E+CUIg== AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuKfUE7lEyaaUOTzJBkhDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31ve/vMLa+sbmVnG7tLO7t39QPjxqmTjVlDVpLGLdCYlhgivWtNwK1kk0IzIUrB2Ob2d++5Fpw2PVsJOEBZIMFY84JdZJD40+7pcrftWfA60SnJMK5Kj3y9+9QUxTyZSlghjTxX5ig4xoy6lg01IvNSwhdEyGrOuoIpKZIJu/OkVnThmgKNZulEVz9fdFRqQxExm6TUnsyCx7M/FfL5RLyTa6DjKuktQyRRfBUSqQjdGsBzTgmlErJo4Qqrn7HdER0YRa11bJlYKXK1glrYsq9qv4/rJSu8nrKcIJnMI5YLiCGtxBHZpAYQjP8AKv3pP35r17H4vVgpffHMMfeJ8/E+CUIg== AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuKfUE7lEyaaUOTzJBkhDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31ve/vMLa+sbmVnG7tLO7t39QPjxqmTjVlDVpLGLdCYlhgivWtNwK1kk0IzIUrB2Ob2d++5Fpw2PVsJOEBZIMFY84JdZJD40+7pcrftWfA60SnJMK5Kj3y9+9QUxTyZSlghjTxX5ig4xoy6lg01IvNSwhdEyGrOuoIpKZIJu/OkVnThmgKNZulEVz9fdFRqQxExm6TUnsyCx7M/FfL5RLyTa6DjKuktQyRRfBUSqQjdGsBzTgmlErJo4Qqrn7HdER0YRa11bJlYKXK1glrYsq9qv4/rJSu8nrKcIJnMI5YLiCGtxBHZpAYQjP8AKv3pP35r17H4vVgpffHMMfeJ8/E+CUIg== T2 AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047JiX9AOJZNm2tAkMyQZoYz9BLe6dydu/Rm3folpOwttPXDhcM69nMsJE8GN9bwvtLa+sbm1Xdgp7u7tHxyWjo5bJk41ZU0ai1h3QmKY4Io1LbeCdRLNiAwFa4fj25nffmTa8Fg17CRhgSRDxSNOiXXSQ6Nf7ZfKXsWbA68SPydlyFHvl757g5imkilLBTGm63uJDTKiLaeCTYu91LCE0DEZsq6jikhmgmz+6hSfO2WAo1i7URbP1d8XGZHGTGToNiWxI7PszcR/vVAuJdvoOsi4SlLLFF0ER6nANsazHvCAa0atmDhCqObud0xHRBNqXVtFV4q/XMEqaVUrvlfx7y/LtZu8ngKcwhlcgA9XUIM7qEMTKAzhGV7gFT2hN/SOPharayi/OYE/QJ8/FXSUIw== AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047JiX9AOJZNm2tAkMyQZoYz9BLe6dydu/Rm3folpOwttPXDhcM69nMsJE8GN9bwvtLa+sbm1Xdgp7u7tHxyWjo5bJk41ZU0ai1h3QmKY4Io1LbeCdRLNiAwFa4fj25nffmTa8Fg17CRhgSRDxSNOiXXSQ6Nf7ZfKXsWbA68SPydlyFHvl757g5imkilLBTGm63uJDTKiLaeCTYu91LCE0DEZsq6jikhmgmz+6hSfO2WAo1i7URbP1d8XGZHGTGToNiWxI7PszcR/vVAuJdvoOsi4SlLLFF0ER6nANsazHvCAa0atmDhCqObud0xHRBNqXVtFV4q/XMEqaVUrvlfx7y/LtZu8ngKcwhlcgA9XUIM7qEMTKAzhGV7gFT2hN/SOPharayi/OYE/QJ8/FXSUIw== AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047JiX9AOJZNm2tAkMyQZoYz9BLe6dydu/Rm3folpOwttPXDhcM69nMsJE8GN9bwvtLa+sbm1Xdgp7u7tHxyWjo5bJk41ZU0ai1h3QmKY4Io1LbeCdRLNiAwFa4fj25nffmTa8Fg17CRhgSRDxSNOiXXSQ6Nf7ZfKXsWbA68SPydlyFHvl757g5imkilLBTGm63uJDTKiLaeCTYu91LCE0DEZsq6jikhmgmz+6hSfO2WAo1i7URbP1d8XGZHGTGToNiWxI7PszcR/vVAuJdvoOsi4SlLLFF0ER6nANsazHvCAa0atmDhCqObud0xHRBNqXVtFV4q/XMEqaVUrvlfx7y/LtZu8ngKcwhlcgA9XUIM7qEMTKAzhGV7gFT2hN/SOPharayi/OYE/QJ8/FXSUIw== AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047JiX9AOJZNm2tAkMyQZoYz9BLe6dydu/Rm3folpOwttPXDhcM69nMsJE8GN9bwvtLa+sbm1Xdgp7u7tHxyWjo5bJk41ZU0ai1h3QmKY4Io1LbeCdRLNiAwFa4fj25nffmTa8Fg17CRhgSRDxSNOiXXSQ6Nf7ZfKXsWbA68SPydlyFHvl757g5imkilLBTGm63uJDTKiLaeCTYu91LCE0DEZsq6jikhmgmz+6hSfO2WAo1i7URbP1d8XGZHGTGToNiWxI7PszcR/vVAuJdvoOsi4SlLLFF0ER6nANsazHvCAa0atmDhCqObud0xHRBNqXVtFV4q/XMEqaVUrvlfx7y/LtZu8ngKcwhlcgA9XUIM7qEMTKAzhGV7gFT2hN/SOPharayi/OYE/QJ8/FXSUIw== Preliminary: What does 'adding decision trees (classification or regression trees)' mean? AAACJnicbVDLSsNAFJ3UV62vqks3g0WoCCGRom4KRTduhAp9QRPDZDpth84kYWYilpC/8CP8Bre6difiTv/E6UOwrQcuHM65l3vv8SNGpbKsTyOztLyyupZdz21sbm3v5Hf3GjKMBSZ1HLJQtHwkCaMBqSuqGGlFgiDuM9L0B1cjv3lPhKRhUFPDiLgc9QLapRgpLXl5swwdxKI+8ix4Ah0Zcy+hZTu9u/nVKax5tOj4PHlIj718wTKtMeAisaekAKaoevlvpxPimJNAYYakbNtWpNwECUUxI2nOiSWJEB6gHmlrGiBOpJuM/0rhkVY6sBsKXYGCY/XvRIK4lEPu606OVF/OeyPxP68dq+6Fm9AgihUJ8GRRN2ZQhXAUEuxQQbBiQ00QFlTfCnEfCYSVjnJmi89TnYk9n8AiaZya9plZui0VKpfTdLLgAByCIrDBOaiAa1AFdYDBI3gGL+DVeDLejHfjY9KaMaYz+2AGxtcP5lSkyw== = ↵0 + M X i=1 ↵iTi(x) AAACZXicbVFBSxtBGJ3d2Gq1tqlKLx46NJZakLBbxPYSCO3Fi2AhUSGbLt9OvpjBmd115ts2Ydlf6cljT/0NvXUSIxr1wcDjve/xfbxJciUtBcG159eWnj1fXnmxuvZy/dXr+puNE5sVRmBXZCozZwlYVDLFLklSeJYbBJ0oPE0uvk/9019orMzSDk1y7Gs4T+VQCiAnxXXd4pEtdFzKVlD9POKdj7HcjRJdjqtPezy6LGAQEY6p/D1Cg3zH+cGtz10WVD6CONhbCN4Zknfu5J0qrjeCZjADf0zCOWmwOY7j+p9okIlCY0pCgbW9MMipX4IhKRRWq1FhMQdxAefYczQFjbZfzmqp+AenDPgwM+6lxGfq/UQJ2tqJTtykBhrZh95UfMrrFTT82i9lmheEqbhZNCwUp4xPO+YDaVCQmjgCwkh3KxcjMCDI/cTClkRPOwkfNvCYnHxuhgfN/R/7jfa3eTsrbJu9Z7ssZF9Ymx2yY9Zlgl2xf57v1by//rq/5b+9GfW9eWaTLcB/9x8im7Zk = M X i=0 T0 i (x), where T0 0 (x) = ↵0, T0 i (x) = ↵iTi(x)

Adding decision trees → For classification trees, add using class probabilities Additive ensemble of classification trees = + + + + + RandomForestClassifier (n_estimators=6, max_leaf_nodes=4) 6 × DecisionTreeClassifier(max_leaf_nodes=4) Set the number of regions to 4 After ensemble, the number of regions increases significantly more than 4 due to combinations!

Adding decision trees → For regression trees, add them directly. Additive ensemble of regression trees RandomForestRegressor (n_estimators=6, max_leaf_nodes=8) 6 × DecisionTreeRegressor(max_leaf_nodes=8) = + + + + + Set the number of regions to 8 After ensemble, the number of regions increases significantly more than 8 due to combinations!

A high-level glance: “Randomize” type RandomForestRegressor (n_estimators=10, max_leaf_nodes=12) 10 x DecisionTreeRegressor(max_leaf_nodes=12) Take the average Independently generate N decision trees Independently generate N decision trees ExtraTreesRegressor (n_estimators=10, max_leaf_nodes=32) 10 x ExtraTreeRegressor(max_leaf_nodes=32) Take the average The 'Randomize' type (such as Bagging and Random Forest) leads to 'smoothing'.

A high-level glance:“Optimize” type The 'Optimize' type (like Boosting) aims for 'optimization' Learning rate (Step size)

Wrap up for now: “Randomize” vs “Optimize” ‣ RandomForestRegressor max_depth=None, max_leaf_nodes=None ‣ ExtraTreesRegressor max_depth=None, max_leaf_nodes=None ‣ GradientBoostingRegressor max_depth=3, max_leaf_nodes=None ‣ HistGradientBoostingRegressor max_depth=None, max_leaf_nodes=31 ‣ XGBRegressor max_depth=3 ‣ LGBMRegressor num_leaves=31, max_depth=-1 • Use a large base tree (e.g. Fully-grown tree) • Each decision tree is trained to overfit. • Use a small base tree (Size-bounded tree) • Each decision tree is trained to underfit. “Weak" learner Scikit default values (Examples of Regressor) • Random Projection Ensemble (2017) • Random Composite Forest (2016) • Random Rotation Ensemble (2016) • Oblique Random Forests (2011) • Rotation Forests (2006) Other examples • Globally Induced Forest (ICML2017) • Regularized Greedy Forest (TPAMI 2014) • CatBoost (NeurIPS2018) • “Randomize” (Bagging, Random Forest) • “Optimize” (Boosting) Other examples

Integrating “Randomize” and “Optimize” strategies Modern implementations of the “Optimize” type can also use the “Randomize” type (Stochastic Gradient Boosting) • Friedman’s Stochastic Gradient Boosting (Comput Stat Data Anal, 2002) • Breiman’s Iterated Bagging (Mach Learn, 2001) • Ensemble on Random Patches (Louppe & Geurts, ECML PKDD 2012) The basic idea is to introduce 'Randomize' into the 'Optimize' process.

Decision Tree Ensemble: Learning from multiple decision trees We need multiple decision trees from a single dataset • “Randomize” (Bagging, Random Forest) Dataset How do we generate multiple trees? Introduce some randomness into the learning of each tree to diversify and generate a set of trees. • Repeat 'random resampling from data → tree generation'. • Repeat 'using different random numbers and data each time to generate a tree'. Sequentially update the ensemble by adding a decision tree one by one to improves the prediction accuracy. • “Optimize” (Boosting)

Preparation: Bootstrap and variance reduction Bootstrap = Resampling from the empirical distribution AAACt3ichVHLTttAFD2YPniUJi0bJDajRlRBasM1ReWxQrBhySuAhGlkuxMY4YyNPYkIET/AD1Cpq1ZigfoB3XXTBfwACz4BsaRSN130xrFUtQh6LXvOnHvP9Zm5XhSoxBBddlndDx4+etzT29f/ZOBpLv/s+VoS1mNflv0wCOMNz01koLQsG2UCuRHF0q15gVz3dufb+fWGjBMV6lXTjORWzd3Wqqp81zBVyY8X7df2mB59p4VjQmGPSeG4URSH+4JKb95OCeeVKHZyjtJV0xwVlXyBSpSGuA3sDBSQxWKY/woH7xHCRx01SGgYxgFcJPxswgYhYm4LLeZiRirNSxyij7V1rpJc4TK7y99t3m1mrOZ9u2eSqn3+S8BvzEqBEbqgU7qhc/pCV/Trzl6ttEfbS5NXr6OVUSV3NLTy87+qGq8GO39U93o2qGIq9arYe5Qy7VP4HX3j4PhmZWZ5pPWSPtM1+/9El/SdT6AbP/yTJbn88R4/HnvhG+MB2f+O4zZYGy/ZjJcmCrNz2ah6MIwXKPI8JjGLBSyizP0/4BvOcG5NWxWrau10Sq2uTDOIv8La+w2awaAA AAACqXichVHLSsNQED3GV62vqhvBjVgqglAnIiiuim5cWrW1+ECS69UG0yQkaaEWf8CF2wquFFyIn+GmP+DCTxCXFdy4cJIGREt1QnLPPTNn7rkZ3TENzyd66VK6e3r7+mMD8cGh4ZHRxNh43rPLrpA5YZu2W9A1T5qGJXO+4Zuy4LhSK+mm3NXP1oP8bkW6nmFbO37VkYcl7dQyTgyh+QGlLqh0lEhSmsKYbgdqBJKIYtNONHCAY9gQKKMECQs+YxMaPH72oYLgMHeIGnMuIyPMS1wgztoyV0mu0Jg94+8p7/Yj1uJ90NML1YJPMfl1WTmNFD3TAzWpQY/0Sp8de9XCHoGXKq96Syudo9HLye2Pf1UlXn0Uv1V/evZxgpXQq8HenZAJbiFa+sp5vbm9upWqzdIdvbH/W3qhJ76BVXkX91m5dfOHH529dP5jQT6q4BGqvwfWDvKLaZVxdimZWYuGGcMUZjDHE1tGBhvYRI5PKOIKdVwr80pWKSh7rVKlK9JM4Eco4gv9upe5 AAACI3icbZDLSgMxFIYz9VbrbdSlm2ARBKHMSFE3haIblxXsBTrjkEkzbdpkZkgy0jLMQ/gQPoNbXbsTNy5c+Caml4Vt/SHw5z/ncJLPjxmVyrK+jNzK6tr6Rn6zsLW9s7tn7h80ZJQITOo4YpFo+UgSRkNSV1Qx0ooFQdxnpOkPbsb15iMRkkbhvRrFxOWoG9KAYqR05Jlnjo9EOsw8CivQCQTCqZ2ltpU5MuFe2q/Y2cP4Code3zOLVsmaCC4be2aKYKaaZ/44nQgnnIQKMyRl27Zi5aZIKIoZyQpOIkmM8AB1SVvbEHEi3XTyqQye6KQDg0joEyo4Sf9OpIhLOeK+7uRI9eRibRz+V2snKrhyUxrGiSIhni4KEgZVBMeEYIcKghUbaYOwoPqtEPeQJqM0x7ktPs80E3uRwLJpnJfsi1L5rlysXs/o5MEROAanwAaXoApuQQ3UAQZP4AW8gjfj2Xg3PozPaWvOmM0cgjkZ3790HKSw ¯ xi = 1 10 10 X j=1 xj In the case of a normal distribution Resampling and taking the average leads to a reduction in variance • Sample N items from the N data points, allowing for duplicates (sampling with replacement) • Some are duplicated and some are not selected, resulting in a random subset • The probability that a sample is never selected: AAACD3icbVC7TsMwFHXKq5RXgIGBxaJCYqiqpOI1VrAwFom2SG0UOa7TWrWdyHZQq6gfwTewwsyGWPkERv4Ep81AW45k6eice3SvTxAzqrTjfFuFldW19Y3iZmlre2d3z94/aKkokZg0ccQi+RggRRgVpKmpZuQxlgTxgJF2MLzN/PYTkYpG4kGPY+Jx1Bc0pBhpI/n20ch3K3Dk1yqw24u0ynjqOhPfLjtVZwq4TNyclEGOhm//mDxOOBEaM6RUx3Vi7aVIaooZmZS6iSIxwkPUJx1DBeJEeen0AxN4apQeDCNpntBwqv5NpIgrNeaBmeRID9Sil4n/eZ1Eh9deSkWcaCLwbFGYMKgjmLUBe1QSrNnYEIQlNbdCPEASYW06m9sS8KwTd7GBZdKqVd3L6sX9ebl+k7dTBMfgBJwBF1yBOrgDDdAEGEzAC3gFb9az9W59WJ+z0YKVZw7BHKyvXypLm2w= x1, x2, . . . , x10 AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PQi8cI5gHJEmYns8mQmdllZlYMS/AbvOrZm3j1Wzz6J84mezCJBQ1FVTfdXUHMmTau++0UVlbX1jeKm6Wt7Z3dvfL+QVNHiSK0QSIeqXaANeVM0oZhhtN2rCgWAaetYHSb+a1HqjSL5IMZx9QXeCBZyAg2Vmp1A6zSp0mvXHGr7hRomXg5qUCOeq/80+1HJBFUGsKx1h3PjY2fYmUY4XRS6iaaxpiM8IB2LJVYUO2n03Mn6MQqfRRGypY0aKr+nUix0HosAtspsBnqRS8T//M6iQmv/ZTJODFUktmiMOHIRCj7HfWZosTwsSWYKGZvRWSIFSbGJjS3JRBZJt5iAsukeVb1LqsX9+eV2k2eThGO4BhOwYMrqMEd1KEBBEbwAq/w5jw7786H8zlrLTj5zCHMwfn6BXKplmE= ¯ x AAACC3icbVDLSsNAFJ3UV62vaJduBovgqiTia1l047KCfUATymQ6aYfOTMLMRAghn+A3uNW1O3HrR7j0T5y0WdjWAxcO59zLuZwgZlRpx/m2KmvrG5tb1e3azu7e/oF9eNRVUSIx6eCIRbIfIEUYFaSjqWakH0uCeMBIL5jeFX7viUhFI/Go05j4HI0FDSlG2khDu+5pyomCXigRztw8c518aDecpjMDXCVuSRqgRHto/3ijCCecCI0ZUmrgOrH2MyQ1xYzkNS9RJEZ4isZkYKhAJtHPZs/n8NQoIxhG0ozQcKb+vcgQVyrlgdnkSE/UsleI/3mDRIc3fkZFnGgi8DwoTBjUESyagCMqCdYsNQRhSc2vEE+QqUGbvhZSAl504i43sEq65033qnn5cNFo3ZbtVMExOAFnwAXXoAXuQRt0AAYpeAGv4M16tt6tD+tzvlqxyps6WID19QtS/psh ⇥ 1 10 reduced to one tenth

Bagging (Bootstrap Aggregating) Method 1: Bagging (Bootstrap Aggregating) Decision trees have a high prediction variance = If the samples change slightly, the tree structure (or partitions) also changes significantly. Bagging is particularly effective for such a “have high variance and low bias” methods. tree = DecisionTreeClassifier() BaggingClassifier (tree, n_estimators=5) BaggingClassifier (tree, n_estimators=10) BaggingClassifier (tree, n_estimators=100) BaggingClassifier (tree, n_estimators=500) • Generate multiple bootstrap samples and average the models trained on each one. (When using a random subset without duplicates, it's called 'Pasting'). • A simple and powerful general method to reduce the prediction variance.

Problems with decision tree bagging The importance of diversity: The effect of bagging is highest when each weak learner is independent. On the other hand, when decision trees are simply bagged, individual trees tend to be highly correlated with each other. Basic Strategy of “Randomize”-type Ensembles AAACpnichVE9S8NQFD2NX7V+tOoiuBRLpVN5FUFxKnVxkn7YD6ilJvFVQ9MkJGmhFv+AroqDk4KD+DNc+gcc+hPEsYKLgzdpQLS03pC88869577zciVDVSybsZ5PmJicmp7xzwbm5hcWg6Gl5YKlN02Z52Vd1c2SJFpcVTSetxVb5SXD5GJDUnlRqu85+WKLm5aia4d22+CVhniqKTVFFm2iMqlqKMLizI3wMEh4IAIv0nqoiyOcQIeMJhrg0GATViHCoqeMBBgM4iroEGcSUtw8xwUCpG1SFacKkdg6fU9pV/ZYjfZOT8tVy3SKSq9JyjCi7JU9sT7rsmf2xr5G9uq4PRwvbVqlgZYb1eDlau7zX1WDVhtnP6qxnm3UsON6Vci74TLOLeSBvnV+28/tZqOdDfbA3sn/PeuxF7qB1vqQHzM8ezfGj0ReRv8xJ+9V0AgTfwc2DAqb8QThzFYkmfKG6cca1hGjiW0jiX2kkacTOK5wjRshJhwIeaE4KBV8nmYFv0I4/gZs35cc AAACrXichVHLTsJAFL3UF+ID1I2JGyLBuMIpMdG4IrpxycMCCa+0dcAJfaUdSJD4Ay7duMCNJi6Mn+GGH3DBJxiXmLhx4W1pYpSAt2nnzLn33J6Zq1gaczghg4AwMzs3vxBcDC0tr6yGI2vrecds2SqVVFMz7aIiO1RjBpU44xotWjaVdUWjBaV54uYLbWo7zDTOeMeiFV1uGKzOVJkjVSw7rKHL1WQtEiMJ4kV0HIg+iIEfaTPShzKcgwkqtEAHCgZwxBrI4OBTAhEIWMhVoIucjYh5eQpXEEJtC6soVsjINvHbwF3JZw3cuz0dT63iXzR8bVRGIU5eyRMZkj55Jm/ka2KvrtfD9dLBVRlpqVULX2/mPv9V6bhyuPhRTfXMoQ6HnleG3i2PcU+hjvTty9th7igb7+6QB/KO/u/JgLzgCYz2h/qYodneFD8Kepl8Y27er8ARin8HNg7yyYSIOLMfSx37wwzCFmzDLk7sAFJwCmmQvEndQA/uhD1BEspCdVQqBHzNBvwKofENqlyaHQ== AAACqXichVHLSsNAFD2Nr1ofrboR3BRLRRDKVATFVdGNyz7sA9tSkjhtQ9MkJGmhFn/AhdsKrhRciJ/hpj/gop8gLiu4ceFNGhAt1huSOXPuPXfO5EqGqlg2YwOfMDU9Mzvnnw8sLC4tB0MrqzlLb5kyz8q6qpsFSbS4qmg8ayu2yguGycWmpPK81Dh28vk2Ny1F107tjsHLTbGmKVVFFm2HKpl1vRKKsBhzIzwO4h6IwIukHuqjhHPokNFCExwabMIqRFj0FBEHg0FcGV3iTEKKm+e4RIC0LariVCES26BvjXZFj9Vo7/S0XLVMp6j0mqQMI8pe2CMbsj57Yq/s889eXbeH46VDqzTScqMSvFrPfPyratJqo/6tmujZRhUHrleFvBsu49xCHunbF71h5jAd7W6xe/ZG/u/YgD3TDbT2u/yQ4unbCX4k8vL3H3PyXgWNMP57YOMgtxuLE07tRRJH3jD92MAmtmli+0jgBElk6YQ6rtHDjbAjpISCcDYqFXyeZg0/QpC/AAHtmJ0= 1. We need to make the trees somehow dissimilar as possible. 2. Besides, each tree needs to have a small training error. (We can reduce the training error by increasing the depth of a tree) The effect of bagging: The variance of the average of random variables, each with a variance of (independent) (correlated at )

Method 2: Random Subspace and Random Patches • Limit the set of available variables to a random subset for each decision tree training. • Also known as feature bagging or attribute bagging. • Ensemble method applicable beyond just decision trees. Random Subspace Method (Ho, 1998) Random Patches Method (Louppe & Geurts, 2012) Another idea: Instead of taking random subsets of data for each decision tree learning, one can take random subsets of features (variables) → Row sampling (data instances) vs. Column sampling (features/variables) • Perform bagging and feature bagging simultaneously. • Random patch = random subsets of both samples and features. • Ensemble method applicable beyond just decision trees.

Method 3: Random Forest • At each best split search, select a random subset of input variables. • This reduces tree correlation and enhances the effects of bagging. Random Forest (Breiman, 2001) 2. Search for the best split using only the selected variables 3. Execute the best split … and recursively apply to its children L & R. At every call of best split search, do the following: 2 4 1 3 1. Select a random subset of variables.

• Even if we randomly constraint the variable set during each split search and we sometimes miss the relevant variables, deeper trees can achieve low training errors anyway. • Inherently effective with multivariate methods (univariate results similar to bagging). • Random variable selection uses sampling without replacement (Fisher–Yates Shuffle). Random Forest Essentials •sqrt(n_features) •log2(n_features) •n_features ‣ Number of trees (n_estimators) ‣ Number of features to select (max_features) Classification tree common settings (Breiman's original) Plain decision tree bagging (without variable subset selection) Note: In scikit-learn, RandomForestClassifier defaults to 'sqrt' for max_features, while RandomForestRegressor defaults to 'n_features,' so please be aware of this distinction Two key hyperparameters bagging = sampling with replacement

Random Forest is similar to Nearest Neighbors KNeighborsClassifier KNeighborsRegressor RandomForestClassifier RandomForestRegressor Lin and Jeon, Random Forests and Adaptive Nearest Neighbors, Journal of the American Statistical Association, 101(474): 578-590, 2006.

Method 4: ExtraTrees 2 4 1 3 • A Random Forest variant • replaces the best split search on selected variables with a random split ExtraTrees (Geurts, Ernst, Wehenkel, 2006) 1. Select a random subset of variables. 2. Taking a random split point for each selected variable (uniformly distributed within the data range) 3. Execute the best split … and recursively apply to its children L & R. At every call of best split search, do the following:

• Essentially shares properties with Random Forest, being effective primarily with multivariate data, with the only difference being the substitution of complete random splits for the exhaustive search for the best split. • Since the exhaustive search for the best split is the most time-consuming part of decision tree learning, even with large or high-dimensional data, it's extremely fast. • Decision trees can reduce training errors even with this extremely randomized setting, as long as the trees are made deep, and since the split points are random, the trees are dissimilar. • Predictive accuracy is equivalent to or even better than Random Forest, especially in regression. • Note that the default setting in Scikit-learn is bootstrap=False, which means it doesn't perform bootstrapping (bagging), making it more prone to overfitting, so be cautious. ExtraTrees Essentials While Random Forest without bagging doesn't make much sense, Extra Trees without bagging is an extremely promising and interesting method based on empirical evidence

• Random forests usually assume bagging as a major premise, and since they are trained on a random subset of the training data, they are almost impossible to overfit in principle. • Conversely, if you want the training error to be zero anyway in special situations, they are impossible to overfit. Noteworthy Property 1 • Extra Trees without bagging (Scikit's default setting) generates many completely different decision trees with training error 0 because the split points are random • Very interestingly, when taking this ensemble of many overfitting trees, there are many cases where the training error is (nearly) zero and the test error is also small (the well-known "benign overfitting" phenomenon). → This is why bagging is off by default in Scikit • Extra Trees with bagging on should always be tried, but interestingly, experience has shown that the test error is often smaller with "bagging off”. The "benign overfitting (or harmless overfitting)“ phenomenon

Random Forest is the same as decision tree bagging for univariate (1-dimensional) data because there is only one variable to choose from, but this is not the case with Extra Trees! Noteworthy Property 2 ExtraTreesRegressor(n_estimators=10) RandomForestRegressor(n_estimators=10) Since the "Random Split Points" make the number of split point candidates less than finite, the ensemble has the characteristics of a natural continuous interpolation between sample points, and is recommended over Random Forest, especially for "regression"! The “natural conservative interpolation” phenomenon

Noteworthy Property 2 KernelRidge(kernel='rbf', alpha=0.05, gamma=0.1) KernelRidge(kernel='rbf', alpha=1e-4, gamma=0.1) KernelRidge(kernel='rbf', alpha=1e-4, gamma=2.0) ExtraTreesRegressor(n_estimators=50) DecisionTreeRegressor() conservative conservative ʁ The “natural convervative interpolation” phenomenon groundless groundless groundless

Noteworthy Property 2 Text text Geurts, Ernst, Wehenkel, Extremely randomized trees. Mach Learn 63, 3–42 (2006). https://doi.org/10.1007/s10994-006-6226-1

• Random Forest is the same as decision tree bagging for univariate (1D) • ExtraTrees is more non-trivial and interesting in higher dimensions! (If it were just a random partition, the approximation would be too inefficient in higher dimensions.) Bagging vs ExtraTrees without bagging DecisionTreeRegressor RandomForestRegressor
 (n_estimators=50) ExtraTreesRegressor
 (n_estimators=50) depth n_leaves depth n_leaves KernelRidge

Input Space paritions of tree ensembles Refinement of partitions and randomness Random forests #tree=10 Extra trees #tree=10 Diversify the trees by adding some randomness Random subsamples, Random subspaces, Random thresholds, Random rotations, Random projections, ...

Decision Tree Ensemble: Learning from multiple decision trees We need multiple decision trees from a single dataset • “Randomize” (Bagging, Random Forest) Dataset How do we generate multiple trees? Introduce some randomness into the learning of each tree to diversify and generate a set of trees. • Repeat 'random resampling from data → tree generation'. • Repeat 'using different random numbers and data each time to generate a tree'. Sequentially update the ensemble by adding a decision tree one by one to improves the prediction accuracy. • “Optimize” (Boosting)

A high-level glance: “Randomize” vs “Optimize” The 'Optimize' type (like Boosting) aims for 'optimization' Learning rate (Step size)

Preparation: supervised learning as optimization Change the parameter so that the loss becomes smaller AAACp3ichVG7SgNBFD2u73fURrBZDIqChLtBUQRBtLEzPhIDRsLuOtHFfbE7CWiwF3/AwkrBQgRb7W38AQs/QSwVbCy82SyIinqX2Tlz5p47Z+Yavm2FkuixQWlsam5pbWvv6Ozq7ulN9PXnQq8cmCJrerYX5A09FLbliqy0pC3yfiB0x7DFhrG3WNvfqIggtDx3Xe77YsvRd1yrZJm6ZKqYGC4YTrUgd4XUD9U5dawOi9qEGqP0eDGRpBRFof4EWgySiCPjJW5QwDY8mCjDgYALydiGjpC/TWgg+MxtocpcwMiK9gUO0cHaMmcJztCZ3eP/Dq82Y9blda1mGKlNPsXmEbBSxQg90CW90D1d0RO9/1qrGtWoednn2ahrhV/sPR5ce/tX5fAssfup+tOzRAkzkVeLvfsRU7uFWddXDk5e1mZXR6qjdE7P7P+MHumOb+BWXs2LFbF6+ocfg73wi3GDtO/t+Aly6ZQ2laKVyeT8QtyqNgxhGGPcj2nMYwkZZLn+Ea5xg1tlXFlWckq+nqo0xJoBfAlF/wDm+Zyq ✓ = (✓1, ✓2) AAACi3ichVHLTsJAFL3UF6II6sbEDZFgXJFbH9EQF0Rj4pKHPBIgpK0DNJS2aQcSJP6ASzcucKOJC+MH+AFu/AEXfIJxiYkbF15KE6NEvM10zpy5586ZubKpqTZH7HmEicmp6RnvrG9u3r8QCC4uZW2jaSksoxiaYeVlyWaaqrMMV7nG8qbFpIassZxcPxzs51rMslVDP+Ftk5UaUlVXK6oicaLyRV5jXCqL5WAYo+hEaBSILgiDGwkj+AhFOAUDFGhCAxjowAlrIIFNXwFEQDCJK0GHOIuQ6uwzOAcfaZuUxShDIrZO/yqtCi6r03pQ03bUCp2i0bBIGYIIvuA99vEZH/AVP/+s1XFqDLy0aZaHWmaWAxcr6Y9/VQ2aOdS+VWM9c6jAnuNVJe+mwwxuoQz1rbOrfjqWinTW8RbfyP8N9vCJbqC33pW7JEt1x/iRyQu9GDVI/N2OUZDdjIo7UUxuh+MHbqu8sAprsEH92IU4HEMCMk4fLqEL14Jf2BJiwv4wVfC4mmX4EcLRF0xPksM= ✓1 AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaUUgLKYKW/uQPqMjMdNPB+WPmKpj0Ai3btLBNQYvoAXqANr1ACx8hWhq0adFxHIiS7Ax37ne/e75zv3uPZKqKzRF7HmFicmp6xjvrm5v3LwSCi0t522haMsvJhmpYRUm0maroLMcVrrKiaTFRk1RWkBoHg/1Ci1m2YujHvG2yiibWdOVUkUVOVLHM64yL1Xg1GMYoOhEaBTEXhMGNlBF8hDKcgAEyNEEDBjpwwiqIYNNXghggmMRVoEOcRUhx9hmcg4+0TcpilCES26B/jVYll9VpPahpO2qZTlFpWKQMQQRf8B77+IwP+Iqff9bqODUGXto0S0MtM6uBi5Xsx78qjWYO9W/VWM8cTmHX8aqQd9NhBreQh/rW2VU/m8hEOut4i2/k/wZ7+EQ30Fvv8l2aZbpj/EjkhV6MGhT73Y5RkI9HY9tRTG+Fk/tuq7ywCmuwQf3YgSQcQQpyTh8uoQvXgl/YFBLC3jBV8LiaZfgRwuEXTm+SxA== ✓2 Change so that the output values are close to the desired values for all samples AAACAXicdVDLSsNAFJ3UV62vqks3g0VwFRIpVXdFNy4r2AckoUymk3boTBJmboQSuvIb3Oranbj1S1z6J07aCj4PXDiccy/33hOmgmtwnDertLS8srpWXq9sbG5t71R39zo6yRRlbZqIRPVCopngMWsDB8F6qWJEhoJ1w/Fl4XdvmdI8iW9gkrJAkmHMI04JGMnzQ5n7MGJApv1qzbHdc9dtOPg3cW1nhhpaoNWvvvuDhGaSxUAF0dpznRSCnCjgVLBpxc80SwkdkyHzDI2JZDrIZydP8ZFRBjhKlKkY8Ez9OpETqfVEhqZTEhjpn14h/uV5GURnQc7jNAMW0/miKBMYElz8jwdcMQpiYgihiptbMR0RRSiYlL5tCWWRyefj+H/SObHdhl2/rteaF4t0yugAHaJj5KJT1ERXqIXaiKIE3aMH9GjdWU/Ws/Uyby1Zi5l99A3W6wfh55hS ✓

• Initialize with random values AAACi3ichVHLTsJAFL3UF6II6sbEDZFgXJFbH9EQF0Rj4pKHPBIgpK0DNJS2aQcSJP6ASzcucKOJC+MH+AFu/AEXfIJxiYkbF15KE6NEvM10zpy5586ZubKpqTZH7HmEicmp6RnvrG9u3r8QCC4uZW2jaSksoxiaYeVlyWaaqrMMV7nG8qbFpIassZxcPxzs51rMslVDP+Ftk5UaUlVXK6oicaLyRV5jXCqL5WAYo+hEaBSILgiDGwkj+AhFOAUDFGhCAxjowAlrIIFNXwFEQDCJK0GHOIuQ6uwzOAcfaZuUxShDIrZO/yqtCi6r03pQ03bUCp2i0bBIGYIIvuA99vEZH/AVP/+s1XFqDLy0aZaHWmaWAxcr6Y9/VQ2aOdS+VWM9c6jAnuNVJe+mwwxuoQz1rbOrfjqWinTW8RbfyP8N9vCJbqC33pW7JEt1x/iRyQu9GDVI/N2OUZDdjIo7UUxuh+MHbqu8sAprsEH92IU4HEMCMk4fLqEL14Jf2BJiwv4wVfC4mmX4EcLRF0xPksM= ✓1 AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaUUgLKYKW/uQPqMjMdNPB+WPmKpj0Ai3btLBNQYvoAXqANr1ACx8hWhq0adFxHIiS7Ax37ne/e75zv3uPZKqKzRF7HmFicmp6xjvrm5v3LwSCi0t522haMsvJhmpYRUm0maroLMcVrrKiaTFRk1RWkBoHg/1Ci1m2YujHvG2yiibWdOVUkUVOVLHM64yL1Xg1GMYoOhEaBTEXhMGNlBF8hDKcgAEyNEEDBjpwwiqIYNNXghggmMRVoEOcRUhx9hmcg4+0TcpilCES26B/jVYll9VpPahpO2qZTlFpWKQMQQRf8B77+IwP+Iqff9bqODUGXto0S0MtM6uBi5Xsx78qjWYO9W/VWM8cTmHX8aqQd9NhBreQh/rW2VU/m8hEOut4i2/k/wZ7+EQ30Fvv8l2aZbpj/EjkhV6MGhT73Y5RkI9HY9tRTG+Fk/tuq7ywCmuwQf3YgSQcQQpyTh8uoQvXgl/YFBLC3jBV8LiaZfgRwuEXTm+SxA== ✓2 The initial value of Parameter optimization by iterating “Move a little bit”

Parameter optimization by iterating “Move a little bit” • Initialize with random values Iterate • Move a little bit towards the direction that reduces the value of (Learning rate) AAACh3ichVHLSsNQED3Gd3206kZwUyyKqzoRX7jysXHZh62CFknibRuaJiFJC1r8ARduFVwpuBA/wA9w4w+48BPEZQU3LpykAdFinZDMuefOmZx7R7UN3fWIXrqk7p7evv6BwcjQ8MhoNDY2nnetmqOJnGYZlrOnKq4wdFPkPN0zxJ7tCKWqGmJXrWz5+7t14bi6Ze54x7YoVJWSqRd1TfF8ipIkH8YSnIKItwM5BAmEkbJiDzjAESxoqKEKARMeYwMKXH72IYNgM1dAgzmHkR7sC5wiwtoaVwmuUJit8LfEq/2QNXnt93QDtcZ/Mfh1WBnHDD3THTXpie7plT7/7NUIevhejjmrLa2wD6Nnk9mPf1VVzh7K36qOnj0UsRp41dm7HTD+KbSWvn5y0cyuZWYas3RDb+z/ml7okU9g1t+127TIXHXwo7IXvjEekPx7HO0gv5CUl5KUXkysb4ajGsAUpjHH81jBOraRQo77l3GOC1xKg9K8tCyttkqlrlAzgR8hbXwB3TiQWA== 0.01 Step size AAACi3ichVHLTsJAFL3UF6II6sbEDZFgXJFbH9EQF0Rj4pKHPBIgpK0DNJS2aQcSJP6ASzcucKOJC+MH+AFu/AEXfIJxiYkbF15KE6NEvM10zpy5586ZubKpqTZH7HmEicmp6RnvrG9u3r8QCC4uZW2jaSksoxiaYeVlyWaaqrMMV7nG8qbFpIassZxcPxzs51rMslVDP+Ftk5UaUlVXK6oicaLyRV5jXCqL5WAYo+hEaBSILgiDGwkj+AhFOAUDFGhCAxjowAlrIIFNXwFEQDCJK0GHOIuQ6uwzOAcfaZuUxShDIrZO/yqtCi6r03pQ03bUCp2i0bBIGYIIvuA99vEZH/AVP/+s1XFqDLy0aZaHWmaWAxcr6Y9/VQ2aOdS+VWM9c6jAnuNVJe+mwwxuoQz1rbOrfjqWinTW8RbfyP8N9vCJbqC33pW7JEt1x/iRyQu9GDVI/N2OUZDdjIo7UUxuh+MHbqu8sAprsEH92IU4HEMCMk4fLqEL14Jf2BJiwv4wVfC4mmX4EcLRF0xPksM= ✓1 AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaUUgLKYKW/uQPqMjMdNPB+WPmKpj0Ai3btLBNQYvoAXqANr1ACx8hWhq0adFxHIiS7Ax37ne/e75zv3uPZKqKzRF7HmFicmp6xjvrm5v3LwSCi0t522haMsvJhmpYRUm0maroLMcVrrKiaTFRk1RWkBoHg/1Ci1m2YujHvG2yiibWdOVUkUVOVLHM64yL1Xg1GMYoOhEaBTEXhMGNlBF8hDKcgAEyNEEDBjpwwiqIYNNXghggmMRVoEOcRUhx9hmcg4+0TcpilCES26B/jVYll9VpPahpO2qZTlFpWKQMQQRf8B77+IwP+Iqff9bqODUGXto0S0MtM6uBi5Xsx78qjWYO9W/VWM8cTmHX8aqQd9NhBreQh/rW2VU/m8hEOut4i2/k/wZ7+EQ30Fvv8l2aZbpj/EjkhV6MGhT73Y5RkI9HY9tRTG+Fk/tuq7ywCmuwQf3YgSQcQQpyTh8uoQvXgl/YFBLC3jBV8LiaZfgRwuEXTm+SxA== ✓2 The initial value of

How to Move? → In the Direction of the ‘Gradient' By repeatedly moving a little bit in the opposite direction of the gradient vector, you approach the nearest local minimum! How should we choose this direction? AAADrHiclVE9bxNBEJ3L8RGOjzjQINFEWIlCY/aiIBBVBBSU+XISyWdZu+e1vcre7Wp3bRFO/gNI1BRUIFEgSgoaqGj4AxT5CYgySDQUzJ1PENsiKHO629k389693WFaCusIOfRm/DNnz52fvRBcvHT5ylxl/uqOVX0T83qspDJ7jFouRcrrTjjJ97ThNGGS77L9h3l9d8CNFSrddgeaNxPaTUVHxNQh1Jr3liLGuyLNWEKdEU+GQfSIS0cj1+OOtsIoGgdWcmDQVs5OVnQQ8bR9TEcbpZ0KpvQ1NU5QGSHQM0kmlbXD5Ygl2UhoeOv2n5ZjJk7NGnd6avrEcVqVKqmRIhamk7BMqlDGuqp8gAjaoCCGPiTAIQWHuQQKFp8GhEBAI9aEDDGDmSjqHIYQILePXRw7KKL7+O3irlGiKe5zTVuwY/yLxNcgcwEWyVfylhyRL+Qd+UZ+/VMrKzRyLwe4shGX69bcs+tbP//LSnB10PvLOtGzgw7cK7wK9K4LJD9FPOIPnr442rq/uZgtkdfkO/p/RQ7JZzxBOvgRv9ngmy9P8MPQC94YDiicHMd0srNSC+/UyMZqde1BOapZuAE3YRnncRfW4DGsQx1i77n33vvoffJr/rbf8Juj1hmv5FyDsfA7vwF/sgLy 2 6 6 6 4 ✓1 ✓2 . . . ✓p 3 7 7 7 5 / 2 6 6 6 4 @loss(✓)/@✓1 @loss(✓)/@✓2 . . . @loss(✓)/@✓p 3 7 7 7 5 The gradient vector

Gradient AAAClHichVHLSsNAFL2Nr1ofjQoiuAkWRUHKTVEUUSgWwZW01aqgJSRxWkPzIklLa/EH/AEXXSl2IX6AH+DGH3DRTxCXCm5ceJsGRMV6w2TOnLnnzpm5iq1rrofYCnE9vX39A+HByNDwyGiUHxvfd62yo7KcaumWc6jILtM1k+U8zdPZoe0w2VB0dqCUUu39gwpzXM0y97yazfKGXDS1gqbKHlESH60JG0JhviqJi0JVSixIfAzj6IfwG4gBiEEQaYu/h2M4AQtUKIMBDEzwCOsgg0vfEYiAYBOXhzpxDiHN32dwDhHSlimLUYZMbIn+RVodBaxJ63ZN11erdIpOwyGlALP4hLf4io94h8/48Wetul+j7aVGs9LRMluKXkztvv+rMmj24PRL1dWzBwVY9b1q5N32mfYt1I6+cnb5uruWna3P4TW+kP8rbOED3cCsvKnNDMs2uvhRyAu9GDVI/NmO32A/EReX45hZiiU3g1aFYRpmYJ76sQJJ2IY05PyeNeAGmtwkt86luK1OKhcKNBPwLbidTyDslEI= y = f(x1, x2) AAAChnichVG7SgNBFD2ur/hM1EawEUPEKtwEQ8QqaGOZh9FADGF3HeOSfbG7CWrwBwRbU1gpWIgf4AfY+AMWfoJYKthYeLNZEA3Gu8zOmTP33DkzV7F1zfWIngekwaHhkdHQ2PjE5NR0ODIzu+NaDUcVRdXSLaekyK7QNVMUPc3TRcl2hGwouthV6pud/d2mcFzNMre9Y1tUDLlmageaKntMFY6qyWokSnHyY7EXJAIQRRBZK3KPPezDgooGDAiY8BjrkOHyV0YCBJu5ClrMOYw0f1/gFOOsbXCW4AyZ2Tr/a7wqB6zJ605N11erfIrOw2HlImL0RLf0Ro90Ry/0+Wetll+j4+WYZ6WrFXY1fDZf+PhXZfDs4fBb1dezhwOs+V419m77TOcWalffPGm/FdbzsdYyXdMr+7+iZ3rgG5jNd/UmJ/KXffwo7IVfjBuU+N2OXrCTjCdSccqtRjMbQatCWMASVrgfaWSwhSyKXL+Gc1ygLYWkuJSS0t1UaSDQzOFHSJkvVTaQmA== x2 AAAChnichVHLSsNAFD2Nr/quuhHciKXiqtyIUnFVdONSW/uAWkoSpzU0TUKSFmvxBwS3duFKwYX4AX6AG3/AhZ8gLhXcuPA2DYgW6w2TOXPmnjtn5qq2obse0XNIGhgcGh4Jj46NT0xOTUdmZrOuVXc0kdEsw3LyquIKQzdFxtM9Q+RtRyg11RA5tbrd2c81hOPqlrnvNW1RrCkVUy/rmuIxlT4uyaVIlOLkx2IvkAMQRRC7VuQeBziEBQ111CBgwmNsQIHLXwEyCDZzRbSYcxjp/r7AKcZYW+cswRkKs1X+V3hVCFiT152arq/W+BSDh8PKRcToiW7pjR7pjl7o889aLb9Gx0uTZ7WrFXZp+mw+/fGvqsazh6NvVV/PHsrY8L3q7N32mc4ttK6+cdJ+S2+mYq1luqZX9n9Fz/TANzAb79rNnkhd9vGjshd+MW6Q/LsdvSC7GpfX47S3Fk1uBa0KYwFLWOF+JJDEDnaR4foVnOMCbSksxaV1KdFNlUKBZg4/Qkp+AVMWkJc= x1 AAACCHicdVDLSgMxFM34rPU16tJNsAh1M8yUqW13RTcuK9gHtKVk0kwbmmSGJFMsQ3/Ab3Cra3fi1r9w6Z+YPgQreuDC4Zx7ufeeIGZUadf9sNbWNza3tjM72d29/YND++i4oaJEYlLHEYtkK0CKMCpIXVPNSCuWBPGAkWYwup75zTGRikbiTk9i0uVoIGhIMdJG6tl2R6CAIRjmOwFP76cXPTvnOu4c0HUKlXKx7BtSKlZ8Q7yllQNL1Hr2Z6cf4YQToTFDSrU9N9bdFElNMSPTbCdRJEZ4hAakbahAnKhuOr98Cs+N0odhJE0JDefqz4kUcaUmPDCdHOmh+u3NxL+8dqLDcjelIk40EXixKEwY1BGcxQD7VBKs2cQQhCU1t0I8RBJhbcJa2RLwqcnk+3H4P2kUHO/S8W/9XPVqmU4GnIIzkAceKIEquAE1UAcYjMEjeALP1oP1Yr1ab4vWNWs5cwJWYL1/AS7mmf8= rf(x) AAACCXicdVDLTgIxFO3gC/EFunTTSExwIRkGBNwR3bjERB4JENIpHWhoO5O2o5IJX+A3uNW1O+PWr3Dpn9gBTMToSW5ycs69ufceN2BUadv+sBIrq2vrG8nN1Nb2zu5eOrPfVH4oMWlgn/my7SJFGBWkoalmpB1IgrjLSMsdX8Z+65ZIRX1xoycB6XE0FNSjGGkj9dOZ065ALkPQy3VdHt1PT/rprJ0/r5adMwfaeduuOMVyTJxKySnCglFiZMEC9X76szvwcciJ0JghpToFO9C9CElNMSPTVDdUJEB4jIakY6hAnKheNDt9Co+NMoCeL00JDWfqz4kIcaUm3DWdHOmR+u3F4l9eJ9RetRdREYSaCDxf5IUMah/GOcABlQRrNjEEYUnNrRCPkERYm7SWtrh8ajL5fhz+T5pOvlDOl65L2drFIp0kOARHIAcKoAJq4ArUQQNgcAcewRN4th6sF+vVepu3JqzFzAFYgvX+BZf+mjE= rf(x) Gradient The gradient vector consists of partial derivatives with respect to each variable. The change in the function value when each variable is altered a little bit. = The direction of the normal vector to the isosurface. AAAC+XiclVJNb9MwGHbCx0b5WAdHLtYqpE6UKum6Mg5I09hhxyHRbVITRY7rdNbsJLLfoFWWfwJXOHNDXPk1O/JP5qRFsJXLXsnSo+f9eF4/dloKriEIrjz/3v0HD9fWH7UeP3n6bKO9+fxEF5WibEwLUaizlGgmeM7GwEGws1IxIlPBTtOLD3X+9DNTmhf5J5iXLJZklvOMUwKOSjY9FGWKUBOVRAEnAmfdKJXm0m7bv9xlEtrWexwJLhNjokZ1omZpbIL+u71gZzfsBf1BuDMaDR0ImrDRIRNALI6gwIFtLWSyrpv1+o4jem6BQQ9H0wJ0jafb+A1uJq1m7F33s0m784fDqyBcgg5axnHS/u30aCVZDlQQrSdhUEJsaq+oYO6mlWYloRdkxiYO5kQyHZtmKYtfOWaKs0K5kwNu2H87DJFaz2XqKiWBc307V5P/y00qyPZiw/OyApbThVBWCezcr98dT7liFMTcAUIVd7tiek7cm4D7HTdUUll7Et52YBWcDPrhqD/8OOzsHyzdWUcv0RbqohC9RfvoCB2jMaLezPviffW++cb/7v/wfy5KfW/Z8wLdCP/XNSpG4vM= @f(x) @x1 = lim !0 f(x1 + , x2, . . . , xd) f(x1, x2, . . . , xd) AAAChnichVG7SgNBFD2ur/hM1EawEUPEKtwEQ8QqaGOZh9FADGF3HeOSfbG7CWrwBwRbU1gpWIgf4AfY+AMWfoJYKthYeLNZEA3Gu8zOmTP33DkzV7F1zfWIngekwaHhkdHQ2PjE5NR0ODIzu+NaDUcVRdXSLaekyK7QNVMUPc3TRcl2hGwouthV6pud/d2mcFzNMre9Y1tUDLlmageaKntMFY6qyWokSnHyY7EXJAIQRRBZK3KPPezDgooGDAiY8BjrkOHyV0YCBJu5ClrMOYw0f1/gFOOsbXCW4AyZ2Tr/a7wqB6zJ605N11erfIrOw2HlImL0RLf0Ro90Ry/0+Wetll+j4+WYZ6WrFXY1fDZf+PhXZfDs4fBb1dezhwOs+V419m77TOcWalffPGm/FdbzsdYyXdMr+7+iZ3rgG5jNd/UmJ/KXffwo7IVfjBuU+N2OXrCTjCdSccqtRjMbQatCWMASVrgfaWSwhSyKXL+Gc1ygLYWkuJSS0t1UaSDQzOFHSJkvVTaQmA== x2 AAAChnichVHLSsNAFD2Nr/quuhHciKXiqtyIUnFVdONSW/uAWkoSpzU0TUKSFmvxBwS3duFKwYX4AX6AG3/AhZ8gLhXcuPA2DYgW6w2TOXPmnjtn5qq2obse0XNIGhgcGh4Jj46NT0xOTUdmZrOuVXc0kdEsw3LyquIKQzdFxtM9Q+RtRyg11RA5tbrd2c81hOPqlrnvNW1RrCkVUy/rmuIxlT4uyaVIlOLkx2IvkAMQRRC7VuQeBziEBQ111CBgwmNsQIHLXwEyCDZzRbSYcxjp/r7AKcZYW+cswRkKs1X+V3hVCFiT152arq/W+BSDh8PKRcToiW7pjR7pjl7o889aLb9Gx0uTZ7WrFXZp+mw+/fGvqsazh6NvVV/PHsrY8L3q7N32mc4ttK6+cdJ+S2+mYq1luqZX9n9Fz/TANzAb79rNnkhd9vGjshd+MW6Q/LsdvSC7GpfX47S3Fk1uBa0KYwFLWOF+JJDEDnaR4foVnOMCbSksxaV1KdFNlUKBZg4/Qkp+AVMWkJc= x1 AAACpXichVFNLyxBFD2a52M8DDYSm8mb+FhN7gghVsLGRuJrhsRIp7rVUNHT3emumWAyWwt/wMLqvcRC2OIH2PgDFn6CvCWJjYU7PZ0IgtuprlOn7rl1qq7lOyrURPdNRnPLr9a29o5E5++u7p5kb18+9MqBLXO253jBuiVC6ShX5rTSjlz3AylKliPXrN25+v5aRQah8txVve/LzZLYdlVR2UIzZSZThWIg7GrBF4FWwkkVa294zxyrJcxkmjIUReozyMYgjTgWveQ1CtiCBxtllCDhQjN2IBDyt4EsCD5zm6gyFzBS0b5EDQnWljlLcoZgdpf/27zaiFmX1/WaYaS2+RSHR8DKFIbojs7okW7pnB7o5cta1ahG3cs+z1ZDK32z52hg5flHVYlnjZ031beeNYqYirwq9u5HTP0WdkNfOTh+XJleHqoO0z/6z/7/0j3d8A3cypN9uiSXT77xY7EXfjFuUPZjOz6D/FgmO5GhpfH0zGzcqnYM4g9GuR+TmME8FpHj+oe4wCWujBFjwVg18o1UoynW9ONdGOYrpNKdFQ== @f @x2 AAACpXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsoxKQVJSZXxxQkFpVkJuYopNUi2BXxhrVc8QLKBnoGYKCAyTCEMpQZoCAgX2A7QwxDCkM+QzJDKUMuQypDHkMJkJ3DkMhQDITRDIYMBgwFQLFYhmqgWBGQlQmWT2WoZeAC6i0FqkoFqkgEimYDyXQgLxoqmgfkg8wsButOBtqSA8RFQJ0KDKoGVw1WGnw2OGGw2uClwR+cZlWDzQC5pRJIJ0H0phbE83dJBH8nqCsXSJcwZCB04XVzCUMagwXYrZlAtxeARUC+SIboL6ua/jnYKki1Ws1gkcFroPsXGtw0OAz0QV7Zl+SlgalBs/G4JwnoFmCIASPIED06MBlhRnqGpnoGgSbKDk7QqOJgkGZQYtAAxoc5gwODB0MAQyjQ/DaGNQxbGbYxqTP5MoUwhUGUMjFC9QgzoACmeACisJ0U @f @x1 AAAC2HicjVFNa9wwEJXdtPno1yY95iKyFFIoi704zeYQCC2UHFPIJqErs0iyvBGRZCPJyy7C0FvpNf8ut/afRHa2aT56yIDg8eaNZuYNKQU3NoqugvDZ0vMXyyuray9fvX7ztrO+cWKKSlM2pIUo9BnBhgmu2NByK9hZqRmWRLBTcvGlyZ9OmTa8UMd2XrJU4oniOafYemrc+e1Q+8lIT0jqol7Uxseo198b7AwSD3Z39pJBUiOFicAw30ZEuln9oYb7EBE24coRia3ms3oN5RpTh0qsLcfijvYfNxvHNUJPU/Zb5TQrrHlqSdaWMJXdDjXudP9uBR+DeAG6YBFH484flBW0kkxZKrAxozgqbeqaFlQwv2RlWInpBZ6wkYcKS2ZS15pYw/eeyWBeaP+UhS17t8JhacxcEq/0852bh7mG/F9uVNl8kDquysoyRW8a5ZWAtoDNWWHGNaNWzD3AVHM/K6Tn2Dtm/fHvdSGy8SR+6MBjcNLvxZ96ybeke/B54c4K2ARbYBvEYBccgENwBIaABl8DEVTBNPwe/gh/hr9upGGwqHkH7kV4eQ2Md+O1 rf(x) = 2 6 6 6 6 4 @f(x) @x1 @f(x) @x2 . . . @f(x) @xd 3 7 7 7 7 5

Gradient Descent When minimizing, you should move in the opposite direction of the gradient LR (learning rate) = Step size: How far to move in that direction (typically set as a small value initially to avoid overshooting) Gradient descent

Tuning the learning rate (step size) What's a good value for the learning rate (LR)? LR medium LR small LR large It certainly heads towards the nearest local minimum, but requires many iterations It can advance significantly with few iterations, but it's too coarse LR

A high-level glance: “Randomize” vs “Optimize” The 'Optimize' type (like Boosting) aims for 'optimization' Learning rate (Step size) We usually set a small value, say 0.1 for now. AAACAHicbVC7SgNBFJ31GeMramkzGASrZVd8NULQxjKCeWCyhNnJ3WTIzOwyMyuEJY3fYKu1ndj6J5b+iZNkC5N44MLhnHu5954w4Uwbz/t2lpZXVtfWCxvFza3tnd3S3n5dx6miUKMxj1UzJBo4k1AzzHBoJgqICDk0wsHt2G88gdIslg9mmEAgSE+yiFFirPTYBkPwNfZcv1Mqe643AV4kfk7KKEe1U/ppd2OaCpCGcqJ1y/cSE2REGUY5jIrtVENC6ID0oGWpJAJ0kE0uHuFjq3RxFCtb0uCJ+nciI0LroQhtpyCmr+e9sfif10pNdBVkTCapAUmni6KUYxPj8fu4yxRQw4eWEKqYvRXTPlGEGhvSzJZQjGwm/nwCi6R+6voX7vn9Wblyk6dTQIfoCJ0gH12iCrpDVVRDFEn0gl7Rm/PsvDsfzue0dcnJZw7QDJyvX1IdliA= ⌘ = 0.1

The basic idea of Gradient Boosting Start with a constant prediction and iteratively add a decision tree to the current ensemble to minimize the loss function. AAACAXicbVDLSsNAFJ3UV62vqks3g0Wom5KIr2VREJcVrC2koUymk3boPMLMRCwhK7/Bra7diVu/xKV/4rTNwrYeuHA4517uvSeMGdXGdb+dwtLyyupacb20sbm1vVPe3XvQMlGYNLFkUrVDpAmjgjQNNYy0Y0UQDxlphcPrsd96JEpTKe7NKCYBR31BI4qRsZJ/03WrnZCnT9lxt1xxa+4EcJF4OamAHI1u+afTkzjhRBjMkNa+58YmSJEyFDOSlTqJJjHCQ9QnvqUCcaKDdHJyBo+s0oORVLaEgRP170SKuNYjHtpOjsxAz3tj8T/PT0x0GaRUxIkhAk8XRQmDRsLx/7BHFcGGjSxBWFF7K8QDpBA2NqWZLSHPbCbefAKL5OGk5p3Xzu5OK/WrPJ0iOACHoAo8cAHq4BY0QBNgIMELeAVvzrPz7nw4n9PWgpPP7IMZOF+/PgSXSQ== F0(x) AAACAXicbVDLSsNAFJ3UV62vqks3g0Wom5KIr2VREJcVrC2koUymk3boPMLMRCwhK7/Bra7diVu/xKV/4rTNwrYeuHA4517uvSeMGdXGdb+dwtLyyupacb20sbm1vVPe3XvQMlGYNLFkUrVDpAmjgjQNNYy0Y0UQDxlphcPrsd96JEpTKe7NKCYBR31BI4qRsZJ/0/WqnZCnT9lxt1xxa+4EcJF4OamAHI1u+afTkzjhRBjMkNa+58YmSJEyFDOSlTqJJjHCQ9QnvqUCcaKDdHJyBo+s0oORVLaEgRP170SKuNYjHtpOjsxAz3tj8T/PT0x0GaRUxIkhAk8XRQmDRsLx/7BHFcGGjSxBWFF7K8QDpBA2NqWZLSHPbCbefAKL5OGk5p3Xzu5OK/WrPJ0iOACHoAo8cAHq4BY0QBNgIMELeAVvzrPz7nw4n9PWgpPP7IMZOF+/P5+XSg== F1(x) AAACAXicbVC7SgNBFJ2NrxhfUUubwSDEJuwGX2VQEMsI5gGbJcxOJsmQeSwzs2JYtvIbbLW2E1u/xNI/cZJsYRIPXDiccy/33hNGjGrjut9ObmV1bX0jv1nY2t7Z3SvuHzS1jBUmDSyZVO0QacKoIA1DDSPtSBHEQ0Za4ehm4rceidJUigczjkjA0UDQPsXIWMm/7VbLnZAnT+lpt1hyK+4UcJl4GSmBDPVu8afTkzjmRBjMkNa+50YmSJAyFDOSFjqxJhHCIzQgvqUCcaKDZHpyCk+s0oN9qWwJA6fq34kEca3HPLSdHJmhXvQm4n+eH5v+VZBQEcWGCDxb1I8ZNBJO/oc9qgg2bGwJworaWyEeIoWwsSnNbQl5ajPxFhNYJs1qxbuonN+flWrXWTp5cASOQRl44BLUwB2ogwbAQIIX8ArenGfn3flwPmetOSebOQRzcL5+AUE6l0s= F2(x) AAACAXicbVDJSgNBEO2JW4xb1KOXxiDES5hxPwYF8RjBLDAZQk+nJ2nSy9DdI4ZhTn6DVz17E69+iUf/xM5yMIkPCh7vVVFVL4wZ1cZ1v53c0vLK6lp+vbCxubW9U9zda2iZKEzqWDKpWiHShFFB6oYaRlqxIoiHjDTDwc3Ibz4SpakUD2YYk4CjnqARxchYyb/tnJbbIU+fsuNOseRW3DHgIvGmpASmqHWKP+2uxAknwmCGtPY9NzZBipShmJGs0E40iREeoB7xLRWIEx2k45MzeGSVLoyksiUMHKt/J1LEtR7y0HZyZPp63huJ/3l+YqKrIKUiTgwReLIoShg0Eo7+h12qCDZsaAnCitpbIe4jhbCxKc1sCXlmM/HmE1gkjZOKd1E5vz8rVa+n6eTBATgEZeCBS1AFd6AG6gADCV7AK3hznp1358P5nLTmnOnMPpiB8/ULQtWXTA== F3(x) AAACAXicbVC7TgJBFJ3FF+ILtbSZSEywIbvGV0m0scQEkGTZkNlhFibMYzMzaySbrfwGW63tjK1fYumfOMAWAp7kJifn3Jt77wljRrVx3W+nsLK6tr5R3Cxtbe/s7pX3D9paJgqTFpZMqk6INGFUkJahhpFOrAjiISMP4eh24j88EqWpFE0zjknA0UDQiGJkrOQ3e261G/L0KTvtlStuzZ0CLhMvJxWQo9Er/3T7EiecCIMZ0tr33NgEKVKGYkayUjfRJEZ4hAbEt1QgTnSQTk/O4IlV+jCSypYwcKr+nUgR13rMQ9vJkRnqRW8i/uf5iYmug5SKODFE4NmiKGHQSDj5H/apItiwsSUIK2pvhXiIFMLGpjS3JeSZzcRbTGCZtM9q3mXt4v68Ur/J0ymCI3AMqsADV6AO7kADtAAGEryAV/DmPDvvzofzOWstOPnMIZiD8/ULVJqXVw== T0(x) AAACAXicbVC7TgJBFJ3FF+ILtbSZSEywIbvGV0m0scQEkGTZkNlhFibMYzMzaySbrfwGW63tjK1fYumfOMAWAp7kJifn3Jt77wljRrVx3W+nsLK6tr5R3Cxtbe/s7pX3D9paJgqTFpZMqk6INGFUkJahhpFOrAjiISMP4eh24j88EqWpFE0zjknA0UDQiGJkrOQ3e161G/L0KTvtlStuzZ0CLhMvJxWQo9Er/3T7EiecCIMZ0tr33NgEKVKGYkayUjfRJEZ4hAbEt1QgTnSQTk/O4IlV+jCSypYwcKr+nUgR13rMQ9vJkRnqRW8i/uf5iYmug5SKODFE4NmiKGHQSDj5H/apItiwsSUIK2pvhXiIFMLGpjS3JeSZzcRbTGCZtM9q3mXt4v68Ur/J0ymCI3AMqsADV6AO7kADtAAGEryAV/DmPDvvzofzOWstOPnMIZiD8/ULVjWXWA== T1(x) AAACAXicbVC7SgNBFJ2NrxhfUUubwSDEJuwGX2XQxjJCXpAsYXYymwyZxzIzK4ZlK7/BVms7sfVLLP0TJ8kWJvHAhcM593LvPUHEqDau++3k1tY3Nrfy24Wd3b39g+LhUUvLWGHSxJJJ1QmQJowK0jTUMNKJFEE8YKQdjO+mfvuRKE2laJhJRHyOhoKGFCNjpW6jXy33Ap48pef9YsmtuDPAVeJlpAQy1PvFn95A4pgTYTBDWnc9NzJ+gpShmJG00Is1iRAeoyHpWioQJ9pPZien8MwqAxhKZUsYOFP/TiSIaz3hge3kyIz0sjcV//O6sQlv/ISKKDZE4PmiMGbQSDj9Hw6oItiwiSUIK2pvhXiEFMLGprSwJeCpzcRbTmCVtKoV76py+XBRqt1m6eTBCTgFZeCBa1AD96AOmgADCV7AK3hznp1358P5nLfmnGzmGCzA+foFV9CXWQ== T2(x) AAACAXicbVC7SgNBFJ2NrxhfUUubwSDEJuz6LoM2lhHygs0SZieTZMg8lplZMSxb+Q22WtuJrV9i6Z84SbYwiQcuHM65l3vvCSNGtXHdbye3srq2vpHfLGxt7+zuFfcPmlrGCpMGlkyqdog0YVSQhqGGkXakCOIhI61wdDfxW49EaSpF3YwjEnA0ELRPMTJW8uvd83In5MlTetotltyKOwVcJl5GSiBDrVv86fQkjjkRBjOkte+5kQkSpAzFjKSFTqxJhPAIDYhvqUCc6CCZnpzCE6v0YF8qW8LAqfp3IkFc6zEPbSdHZqgXvYn4n+fHpn8TJFREsSECzxb1YwaNhJP/YY8qgg0bW4KwovZWiIdIIWxsSnNbQp7aTLzFBJZJ86ziXVUuHy5K1dssnTw4AsegDDxwDargHtRAA2AgwQt4BW/Os/PufDifs9ack80cgjk4X79Za5da T3(x) AAACDnicbVDLSsNAFJ3UV62vquDGzWARKkJJxNey6MZlhb6gCWUynbZDZ5IwcyOW2H/wG9zq2p249Rdc+idO2yxs64ELh3Pu5VyOHwmuwba/rczS8srqWnY9t7G5tb2T392r6zBWlNVoKELV9IlmggesBhwEa0aKEekL1vAHt2O/8cCU5mFQhWHEPEl6Ae9ySsBI7fzBqcuAuLQTAq627aLry+RxdNLOF+ySPQFeJE5KCihFpZ3/cTshjSULgAqidcuxI/ASooBTwUY5N9YsInRAeqxlaEAk014y+X+Ej43Swd1QmQkAT9S/FwmRWg+lbzYlgb6e98bif14rhu61l/AgioEFdBrUjQWGEI/LwB2uGAUxNIRQxc2vmPaJIhRMZTMpvhyZTpz5BhZJ/azkXJYu7s8L5Zu0nSw6REeoiBx0hcroDlVQDVH0hF7QK3qznq1368P6nK5mrPRmH83A+voFyhib3Q== +⌘ · T0(x) AAACDnicbVDLSsNAFJ3UV62vquDGzWARKkJJxNey6MZlhb6gCWUynbZDZ5IwcyOW2H/wG9zq2p249Rdc+idO2yxs64ELh3Pu5VyOHwmuwba/rczS8srqWnY9t7G5tb2T392r6zBWlNVoKELV9IlmggesBhwEa0aKEekL1vAHt2O/8cCU5mFQhWHEPEl6Ae9ySsBI7fzBqcuAuLQTAq62naLry+RxdNLOF+ySPQFeJE5KCihFpZ3/cTshjSULgAqidcuxI/ASooBTwUY5N9YsInRAeqxlaEAk014y+X+Ej43Swd1QmQkAT9S/FwmRWg+lbzYlgb6e98bif14rhu61l/AgioEFdBrUjQWGEI/LwB2uGAUxNIRQxc2vmPaJIhRMZTMpvhyZTpz5BhZJ/azkXJYu7s8L5Zu0nSw6REeoiBx0hcroDlVQDVH0hF7QK3qznq1368P6nK5mrPRmH83A+voFy7Ob3g== +⌘ · T1(x) AAACDnicbVDLSsNAFJ3UV62vquDGzWARKkJJiq9l0Y3LCn1BE8JkOm2HziRh5kYssf/gN7jVtTtx6y+49E+cPha2euDC4Zx7OZcTxIJrsO0vK7O0vLK6ll3PbWxube/kd/caOkoUZXUaiUi1AqKZ4CGrAwfBWrFiRAaCNYPBzdhv3jOleRTWYBgzT5JeyLucEjCSnz84dRkQl3YiwDW/XHQDmT6MTvx8wS7ZE+C/xJmRApqh6ue/3U5EE8lCoIJo3XbsGLyUKOBUsFHOTTSLCR2QHmsbGhLJtJdO/h/hY6N0cDdSZkLAE/X3RUqk1kMZmE1JoK8XvbH4n9dOoHvlpTyME2AhnQZ1E4EhwuMycIcrRkEMDSFUcfMrpn2iCAVT2VxKIEemE2exgb+kUS45F6Xzu7NC5XrWThYdoiNURA66RBV0i6qojih6RM/oBb1aT9ab9W59TFcz1uxmH83B+vwBzU6b3w== +⌘ · T2(x)

The basic idea of Gradient Boosting Update Gradient values of the loss func. at training points AAACOHicbZDLSgMxFIYz9VbrbdSlm2ARK2iZEW8boejGRRcV7AU6pWTSTBuauZBkhGGY1/EhfAa3Cu50JW59AjPtiLb1QODn+8/JSX47YFRIw3jVcnPzC4tL+eXCyura+oa+udUQfsgxqWOf+bxlI0EY9UhdUslIK+AEuTYjTXt4nfrNe8IF9b07GQWk46K+Rx2KkVSoq1eq+6Xo0BogGUfJAbyER5bDEY6tAHFJEYNVZcMfP/nlGUq6etEoG6OCs8LMRBFkVevq71bPx6FLPIkZEqJtGoHsxOm1mJGkYIWCBAgPUZ+0lfSQS0QnHv00gXuK9KDjc3U8CUf070SMXCEi11adLpIDMe2l8D+vHUrnohNTLwgl8fB4kRMyKH2YxgZ7lBMsWaQEwpyqt0I8QCopqcKd2GK7aSbmdAKzonFcNs/Kp7cnxcpVlk4e7IBdUAImOAcVcANqoA4weABP4Bm8aI/am/ahfY5bc1o2sw0mSvv6BtzZrRg= L0(y, ˆ y) = @L(y, ˆ y) @ˆ y AAACN3icbVDLShxBFK3W+IyPMS6zKTIEFGHoFh/ZRRSGLCcwD2F6aKprbmsxVV1N1W3j0PTn+BF+g1tduArZSLb5g9SMgzrqgQuHc+7l3nviTAqLvn/vzcx+mJtfWFxa/riyurZe2fjUtjo3HFpcS21OY2ZBihRaKFDCaWaAqVhCJx6cjPzOBRgrdNrEYQY9xc5SkQjO0ElR5Xs9KnAnKLfCWBWX5TYNJSTIjNG/aD3CJ3mHhoCMhryvkTafjahS9Wv+GPQtCSakSiZoRJU/YV/zXEGKXDJru4GfYa9gBgWXUC6HuYWM8QE7g66jKVNge8X40ZJ+dUqfJtq4SpGO1ZcTBVPWDlXsOhXDc/vaG4nved0ck2+9QqRZjpDyx0VJLilqOkqN9oUBjnLoCONGuFspP2eGcXTZTm2JVekyCV4n8Ja0d2vBQW3/51716HiSziL5TL6QLRKQQ3JEfpAGaRFOrsgNuSV33rX323vw/j62zniTmU0yBe/ff+pRq+s= Ft+1(x) Ft(x) + ⌘ · Tt(x) AAACA3icbVDLSsNAFL2pr1pfVZduBotQNyURX8uiIC4r2Ae0oUymk3boTBJnJmIJWfoNbnXtTtz6IS79E6dtFrb1wIXDOfdyLseLOFPatr+t3NLyyupafr2wsbm1vVPc3WuoMJaE1knIQ9nysKKcBbSumea0FUmKhcdp0xtej/3mI5WKhcG9HkXUFbgfMJ8RrI3k3nQTnZY7nkie0uNusWRX7AnQInEyUoIMtW7xp9MLSSxooAnHSrUdO9JugqVmhNO00IkVjTAZ4j5tGxpgQZWbTJ5O0ZFResgPpZlAo4n69yLBQqmR8MymwHqg5r2x+J/XjrV/6SYsiGJNAzIN8mOOdIjGDaAek5RoPjIEE8nMr4gMsMREm55mUjyRmk6c+QYWSeOk4pxXzu5OS9WrrJ08HMAhlMGBC6jCLdSgDgQe4AVe4c16tt6tD+tzupqzspt9mIH19QuGOpiZ Ft(x) AAACB3icbVDLSsNAFJ3UV62PRl26GSxCRSiJ+FoWBXFZwdpCG8JkOmmHzkzCzEQsIR/gN7jVtTtx62e49E+ctlnY1gMXDufcy7mcIGZUacf5tgpLyyura8X10sbm1nbZ3tl9UFEiMWniiEWyHSBFGBWkqalmpB1LgnjASCsYXo/91iORikbiXo9i4nHUFzSkGGkj+Xb5xk/1sZtVuwFPn7Ij3644NWcCuEjcnFRAjoZv/3R7EU44ERozpFTHdWLtpUhqihnJSt1EkRjhIeqTjqECcaK8dPJ4Bg+N0oNhJM0IDSfq34sUcaVGPDCbHOmBmvfG4n9eJ9HhpZdSESeaCDwNChMGdQTHLcAelQRrNjIEYUnNrxAPkERYm65mUgKemU7c+QYWycNJzT2vnd2dVupXeTtFsA8OQBW44ALUwS1ogCbAIAEv4BW8Wc/Wu/VhfU5XC1Z+swdmYH39AuFKmTo= Ft+1(x) AAACEHicbVDLSgNBEJyNrxhfUS+Cl8EgRISwK76OQS8eI+QF2SXMTibJkJndZaZXDEv8CL/Bq569iVf/wKN/4iTZg0ksaCiquunu8iPBNdj2t5VZWl5ZXcuu5zY2t7Z38rt7dR3GirIaDUWomj7RTPCA1YCDYM1IMSJ9wRr+4HbsNx6Y0jwMqjCMmCdJL+BdTgkYqZ0/OMUuA4Jd2gkBV9tQdH2ZPI5O2vmCXbInwIvESUkBpai08z9uJ6SxZAFQQbRuOXYEXkIUcCrYKOfGmkWEDkiPtQwNiGTaSyYfjPCxUTq4GypTAeCJ+nciIVLrofRNpyTQ1/PeWPzPa8XQvfYSHkQxsIBOF3VjgSHE4zhwhytGQQwNIVRxcyumfaIIBRPazBZfjkwmznwCi6R+VnIuSxf354XyTZpOFh2iI1REDrpCZXSHKqiGKHpCL+gVvVnP1rv1YX1OWzNWOrOPZmB9/QLtWZx1 +⌘ · Tt(x) A regression tree AAACAXicbVC7TgJBFJ3FF+ILtbSZSEywIbvGV0m0scQEkGTZkNlhFibMYzMzaySbrfwGW63tjK1fYumfOMAWAp7kJifn3Jt77wljRrVx3W+nsLK6tr5R3Cxtbe/s7pX3D9paJgqTFpZMqk6INGFUkJahhpFOrAjiISMP4eh24j88EqWpFE0zjknA0UDQiGJkrOQ3e6baDXn6lJ32yhW35k4Bl4mXkwrI0eiVf7p9iRNOhMEMae17bmyCFClDMSNZqZtoEiM8QgPiWyoQJzpIpydn8MQqfRhJZUsYOFX/TqSIaz3moe3kyAz1ojcR//P8xETXQUpFnBgi8GxRlDBoJJz8D/tUEWzY2BKEFbW3QjxECmFjU5rbEvLMZuItJrBM2mc177J2cX9eqd/k6RTBETgGVeCBK1AHd6ABWgADCV7AK3hznp1358P5nLUWnHzmEMzB+foFwcaXmw== Tt(x) AAACJXicbVDLSgMxFM3UV62vqks3wSK0oGVGfC2LblxWsA/olJJkMm1oMjMkGbEM8xV+hN/gVtfuRHAl/omZtoitHggczjmXe3NwxJnStv1h5RYWl5ZX8quFtfWNza3i9k5ThbEktEFCHso2RopyFtCGZprTdiQpEpjTFh5eZX7rjkrFwuBWjyLaFagfMJ8RpI3UKx65mPV52cUiuU977BC6AumBFElfIi/90StZTFZ6xZJdtceAf4kzJSUwRb1X/HK9kMSCBppwpFTHsSPdTZDUjHCaFtxY0QiRIerTjqEBElR1k/G3UnhgFA/6oTQv0HCs/p5IkFBqJLBJZkereS8T//M6sfYvugkLoljTgEwW+TGHOoRZR9BjkhLNR4YgIpm5FZIBkoho0+TMFixS04kz38Bf0jyuOmfV05uTUu1y2k4e7IF9UAYOOAc1cA3qoAEIeABP4Bm8WI/Wq/VmvU+iOWs6swtmYH1+AzBLpjY= xi, grad(xi) AAACDnicbVDLSsNAFJ3UV62vqODGTbCILUhJxNdGKAriwkUF+4A2hMl00g6dScLMRAwx/+A3uNW1O3HrL7j0T5y2WdjWAxcO59zLuRw3pERI0/zWcnPzC4tL+eXCyura+oa+udUQQcQRrqOABrzlQoEp8XFdEklxK+QYMpfipju4GvrNB8wFCfx7GYfYZrDnE48gKJXk6DsXtwel2CGH144sdVyWPKYOKZcdvWhWzBGMWWJlpAgy1Bz9p9MNUMSwLxGFQrQtM5R2ArkkiOK00IkEDiEawB5uK+pDhoWdjP5PjX2ldA0v4Gp8aYzUvxcJZELEzFWbDMq+mPaG4n9eO5LeuZ0QP4wk9tE4yIuoIQNjWIbRJRwjSWNFIOJE/WqgPuQQSVXZRIrLUtWJNd3ALGkcVazTysndcbF6mbWTB7tgD5SABc5AFdyAGqgDBJ7AC3gFb9qz9q59aJ/j1ZyW3WyDCWhfvwCem2I= = L0(yi, Ft(xi))

The basic idea of Gradient Boosting Update Gradient values of the loss func. at training points AAACOHicbZDLSgMxFIYz9VbrbdSlm2ARK2iZEW8boejGRRcV7AU6pWTSTBuauZBkhGGY1/EhfAa3Cu50JW59AjPtiLb1QODn+8/JSX47YFRIw3jVcnPzC4tL+eXCyura+oa+udUQfsgxqWOf+bxlI0EY9UhdUslIK+AEuTYjTXt4nfrNe8IF9b07GQWk46K+Rx2KkVSoq1eq+6Xo0BogGUfJAbyER5bDEY6tAHFJEYNVZcMfP/nlGUq6etEoG6OCs8LMRBFkVevq71bPx6FLPIkZEqJtGoHsxOm1mJGkYIWCBAgPUZ+0lfSQS0QnHv00gXuK9KDjc3U8CUf070SMXCEi11adLpIDMe2l8D+vHUrnohNTLwgl8fB4kRMyKH2YxgZ7lBMsWaQEwpyqt0I8QCopqcKd2GK7aSbmdAKzonFcNs/Kp7cnxcpVlk4e7IBdUAImOAcVcANqoA4weABP4Bm8aI/am/ahfY5bc1o2sw0mSvv6BtzZrRg= L0(y, ˆ y) = @L(y, ˆ y) @ˆ y AAACN3icbVDLShxBFK3W+IyPMS6zKTIEFGHoFh/ZRRSGLCcwD2F6aKprbmsxVV1N1W3j0PTn+BF+g1tduArZSLb5g9SMgzrqgQuHc+7l3nviTAqLvn/vzcx+mJtfWFxa/riyurZe2fjUtjo3HFpcS21OY2ZBihRaKFDCaWaAqVhCJx6cjPzOBRgrdNrEYQY9xc5SkQjO0ElR5Xs9KnAnKLfCWBWX5TYNJSTIjNG/aD3CJ3mHhoCMhryvkTafjahS9Wv+GPQtCSakSiZoRJU/YV/zXEGKXDJru4GfYa9gBgWXUC6HuYWM8QE7g66jKVNge8X40ZJ+dUqfJtq4SpGO1ZcTBVPWDlXsOhXDc/vaG4nved0ck2+9QqRZjpDyx0VJLilqOkqN9oUBjnLoCONGuFspP2eGcXTZTm2JVekyCV4n8Ja0d2vBQW3/51716HiSziL5TL6QLRKQQ3JEfpAGaRFOrsgNuSV33rX323vw/j62zniTmU0yBe/ff+pRq+s= Ft+1(x) Ft(x) + ⌘ · Tt(x) AAACA3icbVDLSsNAFL2pr1pfVZduBotQNyURX8uiIC4r2Ae0oUymk3boTBJnJmIJWfoNbnXtTtz6IS79E6dtFrb1wIXDOfdyLseLOFPatr+t3NLyyupafr2wsbm1vVPc3WuoMJaE1knIQ9nysKKcBbSumea0FUmKhcdp0xtej/3mI5WKhcG9HkXUFbgfMJ8RrI3k3nQTnZY7nkie0uNusWRX7AnQInEyUoIMtW7xp9MLSSxooAnHSrUdO9JugqVmhNO00IkVjTAZ4j5tGxpgQZWbTJ5O0ZFResgPpZlAo4n69yLBQqmR8MymwHqg5r2x+J/XjrV/6SYsiGJNAzIN8mOOdIjGDaAek5RoPjIEE8nMr4gMsMREm55mUjyRmk6c+QYWSeOk4pxXzu5OS9WrrJ08HMAhlMGBC6jCLdSgDgQe4AVe4c16tt6tD+tzupqzspt9mIH19QuGOpiZ Ft(x) AAACB3icbVDLSsNAFJ3UV62PRl26GSxCRSiJ+FoWBXFZwdpCG8JkOmmHzkzCzEQsIR/gN7jVtTtx62e49E+ctlnY1gMXDufcy7mcIGZUacf5tgpLyyura8X10sbm1nbZ3tl9UFEiMWniiEWyHSBFGBWkqalmpB1LgnjASCsYXo/91iORikbiXo9i4nHUFzSkGGkj+Xb5xk/1sZtVuwFPn7Ij3644NWcCuEjcnFRAjoZv/3R7EU44ERozpFTHdWLtpUhqihnJSt1EkRjhIeqTjqECcaK8dPJ4Bg+N0oNhJM0IDSfq34sUcaVGPDCbHOmBmvfG4n9eJ9HhpZdSESeaCDwNChMGdQTHLcAelQRrNjIEYUnNrxAPkERYm65mUgKemU7c+QYWycNJzT2vnd2dVupXeTtFsA8OQBW44ALUwS1ogCbAIAEv4BW8Wc/Wu/VhfU5XC1Z+swdmYH39AuFKmTo= Ft+1(x) AAACEHicbVDLSgNBEJyNrxhfUS+Cl8EgRISwK76OQS8eI+QF2SXMTibJkJndZaZXDEv8CL/Bq569iVf/wKN/4iTZg0ksaCiquunu8iPBNdj2t5VZWl5ZXcuu5zY2t7Z38rt7dR3GirIaDUWomj7RTPCA1YCDYM1IMSJ9wRr+4HbsNx6Y0jwMqjCMmCdJL+BdTgkYqZ0/OMUuA4Jd2gkBV9tQdH2ZPI5O2vmCXbInwIvESUkBpai08z9uJ6SxZAFQQbRuOXYEXkIUcCrYKOfGmkWEDkiPtQwNiGTaSyYfjPCxUTq4GypTAeCJ+nciIVLrofRNpyTQ1/PeWPzPa8XQvfYSHkQxsIBOF3VjgSHE4zhwhytGQQwNIVRxcyumfaIIBRPazBZfjkwmznwCi6R+VnIuSxf354XyTZpOFh2iI1REDrpCZXSHKqiGKHpCL+gVvVnP1rv1YX1OWzNWOrOPZmB9/QLtWZx1 +⌘ · Tt(x) A regression tree AAACAXicbVC7TgJBFJ3FF+ILtbSZSEywIbvGV0m0scQEkGTZkNlhFibMYzMzaySbrfwGW63tjK1fYumfOMAWAp7kJifn3Jt77wljRrVx3W+nsLK6tr5R3Cxtbe/s7pX3D9paJgqTFpZMqk6INGFUkJahhpFOrAjiISMP4eh24j88EqWpFE0zjknA0UDQiGJkrOQ3e6baDXn6lJ32yhW35k4Bl4mXkwrI0eiVf7p9iRNOhMEMae17bmyCFClDMSNZqZtoEiM8QgPiWyoQJzpIpydn8MQqfRhJZUsYOFX/TqSIaz3moe3kyAz1ojcR//P8xETXQUpFnBgi8GxRlDBoJJz8D/tUEWzY2BKEFbW3QjxECmFjU5rbEvLMZuItJrBM2mc177J2cX9eqd/k6RTBETgGVeCBK1AHd6ABWgADCV7AK3hznp1358P5nLUWnHzmEMzB+foFwcaXmw== Tt(x) AAACJXicbVDLSgMxFM3UV62vqks3wSK0oGVGfC2LblxWsA/olJJkMm1oMjMkGbEM8xV+hN/gVtfuRHAl/omZtoitHggczjmXe3NwxJnStv1h5RYWl5ZX8quFtfWNza3i9k5ThbEktEFCHso2RopyFtCGZprTdiQpEpjTFh5eZX7rjkrFwuBWjyLaFagfMJ8RpI3UKx65mPV52cUiuU977BC6AumBFElfIi/90StZTFZ6xZJdtceAf4kzJSUwRb1X/HK9kMSCBppwpFTHsSPdTZDUjHCaFtxY0QiRIerTjqEBElR1k/G3UnhgFA/6oTQv0HCs/p5IkFBqJLBJZkereS8T//M6sfYvugkLoljTgEwW+TGHOoRZR9BjkhLNR4YgIpm5FZIBkoho0+TMFixS04kz38Bf0jyuOmfV05uTUu1y2k4e7IF9UAYOOAc1cA3qoAEIeABP4Bm8WI/Wq/VmvU+iOWs6swtmYH1+AzBLpjY= xi, grad(xi) AAACDnicbVDLSsNAFJ3UV62vqODGTbCILUhJxNdGKAriwkUF+4A2hMl00g6dScLMRAwx/+A3uNW1O3HrL7j0T5y2WdjWAxcO59zLuRw3pERI0/zWcnPzC4tL+eXCyura+oa+udUQQcQRrqOABrzlQoEp8XFdEklxK+QYMpfipju4GvrNB8wFCfx7GYfYZrDnE48gKJXk6DsXtwel2CGH144sdVyWPKYOKZcdvWhWzBGMWWJlpAgy1Bz9p9MNUMSwLxGFQrQtM5R2ArkkiOK00IkEDiEawB5uK+pDhoWdjP5PjX2ldA0v4Gp8aYzUvxcJZELEzFWbDMq+mPaG4n9eO5LeuZ0QP4wk9tE4yIuoIQNjWIbRJRwjSWNFIOJE/WqgPuQQSVXZRIrLUtWJNd3ALGkcVazTysndcbF6mbWTB7tgD5SABc5AFdyAGqgDBJ7AC3gFb9qz9q59aJ/j1ZyW3WyDCWhfvwCem2I= = L0(yi, Ft(xi)) AAACH3icbZDLSsNAFIYn9VbrLerSzWARK2hJireNUHTjwkUFe4Emlsl00g6dXJiZCCHkEXwIn8Gtrt2J2y59E6dtBNt6YODn/8/hnPmckFEhDWOo5RYWl5ZX8quFtfWNzS19e6chgohjUscBC3jLQYIw6pO6pJKRVsgJ8hxGms7gZpQ3nwgXNPAfZBwS20M9n7oUI6msjn54V4qPrT6SSZwewStouRzhxEyTSlqKT36Dx0pHLxplY1xwXpiZKIKsah392+oGOPKILzFDQrRNI5R2grikmJG0YEWChAgPUI+0lfSRR4SdjD+UwgPldKEbcPV8Ccfu34kEeULEnqM6PST7YjYbmf9l7Ui6l3ZC/TCSxMeTRW7EoAzgiA7sUk6wZLESCHOqboW4jxQSqRhObXG8VDExZwnMi0albJ6Xz+5Pi9XrjE4e7IF9UAImuABVcAtqoA4weAav4A28ay/ah/apfU1ac1o2swumShv+AEqronY= L(y, ˆ y) = 1 2 (y ˆ y)2 Example) Least squares AAACNXicbVDJSgNBEO1x34169NIYBD0YZsTtIoiCePCgYFTIhKGn05k06WXorlHDMF/jR/gNXvXkwYMgXv0FOzHg+qDg8V4VVfXiVHALvv/kDQwODY+Mjo1PTE5Nz8yW5ubPrc4MZVWqhTaXMbFMcMWqwEGwy9QwImPBLuL2Qde/uGLGcq3OoJOyuiSJ4k1OCTgpKu2Gx1olhictIMboaxxKAi0j88SQRrESxjK/KSK+indxJ+J4DR9G8KVGpbJf8XvAf0nQJ2XUx0lUegkbmmaSKaCCWFsL/BTqOTHAqWDFRJhZlhLaJgmrOaqIZLae994s8LJTGripjSsFuKd+n8iJtLYjY9fZfcL+9rrif14tg+ZOPecqzYAp+rmomQkMGnczww1uGAXRcYRQw92tmLaIIRRcsj+2xLJwmQS/E/hLztcrwVZl83SjvLffT2cMLaIltIICtI320BE6QVVE0S26Rw/o0bvznr1X7+2zdcDrzyygH/DePwAvYqwp =) grad(xi) = yi Ft(xi)

Method 5: Gradient Boosted Trees Additive tree ensemble with iterations to add regression trees that better approximate the gradient sample values • Gradient Tree Boosting • GBM (Gradient Boosting Machines) • GBDT (Gradient Boosted Decision Trees) • GBRT (Gradient Boosted Regression Trees) • MART (Multiple Additive Regression Trees) • TreeNet • AnyBoost • Regression trees are used even for classification, but arbitrary loss functions can be learned • Can be viewed as approximate gradient descent in function space (Mason+ NIPS'99) • Various names for historical reasons Forest Forest Tree AAACTHicbZDPSxwxFMcza2vVVru1x16CS2EFXWakv46ilx56UNhVYbMMb7KZ3WAyE5I3xWE6/5h/hHdv9ljP3qTQ7DpQfz0IfPl838tLvolR0mEYXgathRcvF18tLa+8frO69rb9bv3I5YXlYsBzlduTBJxQMhMDlKjEibECdKLEcXK6P/OPfwrrZJ71sTRipGGSyVRyQI/idr8fY5clujqrY7lJGRhj8zPKlEixR7dZaoFXzIBFCYr+6JZbbApYlfVm/R83qKbMyskUf8XtTtgL50WfiqgRHdLUQdz+zcY5L7TIkCtwbhiFBkfV7HquRL3CCicM8FOYiKGXGWjhRtX89zX96MmYprn1J0M6p/cnKtDOlTrxnRpw6h57M/icNyww/TaqZGYKFBm/W5QWimJOZ1HSsbSCoyq9AG6lfyvlU/CBoQ/8wZZE1z6T6HECT8XRTi/60vt8+Kmzu9eks0Q+kA3SJRH5SnbJd3JABoSTc3JF/pDr4CK4CW6Dv3etraCZeU8eVGvxH19AtU0= Tt(xi) ⇡ @L(y, ˆ y) @ˆ y AAAB+3icbVBNS8NAEJ3Ur1q/qh69LBbBU0nEr4tQ9OKxgrGFNpTNdtMu3d2E3Y0QSn6DVz17E6/+GI/+E7dtDrb1wcDjvRlm5oUJZ9q47rdTWlldW98ob1a2tnd296r7B086ThWhPol5rNoh1pQzSX3DDKftRFEsQk5b4ehu4reeqdIslo8mS2gg8ECyiBFsrORnN1mP9ao1t+5OgZaJV5AaFGj2qj/dfkxSQaUhHGvd8dzEBGOsDCOc5pVuqmmCyQgPaMdSiQXVwXh6bI5OrNJHUaxsSYOm6t+JMRZaZyK0nQKboV70JuJ/Xic10XUwZjJJDZVktihKOTIxmnyO+kxRYnhmCSaK2VsRGWKFibH5zG0JRW4z8RYTWCZPZ3Xvsn7xcF5r3BbplOEIjuEUPLiCBtxDE3wgwOAFXuHNyZ1358P5nLWWnGLmEObgfP0CgWeVQw== y = yi AAACDXicbVDLSsNAFJ34rPVVFVduBotQNyURXxuhKIjLCvYBbQmT6aQdOpOEmRuxhHyD3+BW1+7Erd/g0j9x2mZhWw9cOJxzL+dyvEhwDbb9bS0sLi2vrObW8usbm1vbhZ3dug5jRVmNhiJUTY9oJnjAasBBsGakGJGeYA1vcDPyG49MaR4GDzCMWEeSXsB9TgkYyS3st/sEkmF6detCqe3J5Cl1+bFbKNpleww8T5yMFFGGqlv4aXdDGksWABVE65ZjR9BJiAJOBUvz7ViziNAB6bGWoQGRTHeS8fspPjJKF/uhMhMAHqt/LxIitR5Kz2xKAn09643E/7xWDP5lJ+FBFAML6CTIjwWGEI+6wF2uGAUxNIRQxc2vmPaJIhRMY1MpnkxNJ85sA/OkflJ2zstn96fFynXWTg4doENUQg66QBV0h6qohihK0At6RW/Ws/VufVifk9UFK7vZQ1Owvn4BOzecMQ== ˆ y = Ft(xi)

10.11 Right-Sized Trees for Boosting 361 Algorithm 10.3 Gradient Tree Boosting Algorithm. 1. Initialize f0(x) = arg minγ N i=1 L(yi, γ). 2. For m = 1 to M: (a) For i = 1, 2, . . . , N compute rim = − ∂L(yi, f(xi)) ∂f(xi) f=fm−1 . (b) Fit a regression tree to the targets rim giving terminal regions Rjm, j = 1, 2, . . . , Jm. (c) For j = 1, 2, . . . , Jm compute γjm = arg min γ xi ∈Rjm L (yi, fm−1(xi) + γ) . (d) Update fm(x) = fm−1(x) + Jm j=1 γjmI(x ∈ Rjm). 3. Output ˆ f(x) = fM (x). The algorithm for classification is similar. Lines 2(a)–(d) are repeated K times at each iteration m, once for each class using (10.38). The result at line 3 is K different (coupled) tree expansions fkM (x), k = 1, 2, . . . , K. These produce probabilities via (10.21) or do classification as in (10.20). Friedman’s gradient boosting https://hastie.su.domains/Papers/ESLII.pdf Hastie, Tibshirani, Friedman, The elements of statistical learning (2009) AAACL3icdVDLSgMxFM3UV62vqks3wSK0C8u0VOtGKOrChYta7AM6dcikmTY0kxmSjFCG+RQ/wm9wq2txI7r0L0xfYH0cuHA4517uvccJGJXKNF+NxMLi0vJKcjW1tr6xuZXe3mlIPxSY1LHPfNFykCSMclJXVDHSCgRBnsNI0xmcj/zmHRGS+vxGDQPS8VCPU5dipLRkp8vWBWEKZWUOnkLLFQhH3L6C3K7FEY9h1nKQiIaxlg7hjNdyt0U7nTHzxSNTA/4mhbw5RgZMUbXTH1bXx6FHuMIMSdkumIHqREgoihmJU1YoSYDwAPVIW1OOPCI70fjBGB5opQtdX+jiCo7V7xMR8qQceo7u9JDqy5/eSPzLa4fKPelElAehIhxPFrkhg8qHo7RglwqCFRtqgrCg+laI+0inpHSmc1scL9aZzB6H/5NGMV84zpeuS5nK2TSdJNgD+yALCqAMKuASVEEdYHAPHsETeDYejBfjzXiftCaM6cwumIPx+QUy6aiI (s) = nLnR n (¯ yL ¯ yR)2 AAACNXicdVDLSgMxFM34rO9Rl26CRWgXlmmpj40g6sKFiypWhc5YMmnahiaZIcmIwzBf40f4DW515cKFIG79BTN9gM8DF07OvZebc/yQUaUd59kaG5+YnJrOzczOzS8sLtnLKxcqiCQmdRywQF75SBFGBalrqhm5CiVB3Gfk0u8dZv3LGyIVDcS5jkPicdQRtE0x0kZq2nsuR7oreRLK4DZOC+4RYRoVVLEI96Bonpg6gwXXRzKJU/PchCN+VryuNO28U6psOQbwNymXnD7yYIha0351WwGOOBEaM6RUo+yE2kuQ1BQzks66kSIhwj3UIQ1DBeJEeUnfZgo3jNKC7UCaEhr21a8bCeJKxdw3k5kp9bOXiX/1GpFu73oJFWGkicCDQ+2IQR3ALDPYopJgzWJDEJbU/BXiLpIIa5Pstys+T00mI+Pwf3JRKZW3S9XTan7/YJhODqyBdVAAZbAD9sExqIE6wOAOPIBH8GTdWy/Wm/U+GB2zhjur4Busj0/4Bare proxy( (s)) = nLnR(¯ yL ¯ yR)2 1. A special splitting criteria “Friedman’s MSE” is used instead of standard MSE. 2. Instead of for the entire tree , each leaf constant is optimized separately. AAAB+nicbVA9TwJBEJ3DL8Qv1NJmIzGxInfGr5JoY4mJIAlcyN4yBxt27y67eyYE+Qu2WtsZW/+Mpf/EPbhCwJdM8vLeTGbmBYng2rjut1NYWV1b3yhulra2d3b3yvsHTR2nimGDxSJWrYBqFDzChuFGYCtRSGUg8DEY3mb+4xMqzePowYwS9CXtRzzkjJpM6qCh3XLFrbpTkGXi5aQCOerd8k+nF7NUYmSYoFq3PTcx/pgqw5nASamTakwoG9I+ti2NqETtj6e3TsiJVXokjJWtyJCp+ndiTKXWIxnYTknNQC96mfif105NeO2PeZSkBiM2WxSmgpiYZI+THlfIjBhZQpni9lbCBlRRZmw8c1sCObGZeIsJLJPmWdW7rF7cn1dqN3k6RTiCYzgFD66gBndQhwYwGMALvMKb8+y8Ox/O56y14OQzhzAH5+sXwhKU2A== ⌘ AAAB+nicbVA9TwJBEJ3DL8Qv1NJmIzGxInfGr5JoY4mJIAlcyN4yBxt27y67eyYE+Qu2WtsZW/+Mpf/EPbhCwJdM8vLeTGbmBYng2rjut1NYWV1b3yhulra2d3b3yvsHTR2nimGDxSJWrYBqFDzChuFGYCtRSGUg8DEY3mb+4xMqzePowYwS9CXtRzzkjJpM6qCh3XLFrbpTkGXi5aQCOerd8k+nF7NUYmSYoFq3PTcx/pgqw5nASamTakwoG9I+ti2NqETtj6e3TsiJVXokjJWtyJCp+ndiTKXWIxnYTknNQC96mfif105NeO2PeZSkBiM2WxSmgpiYZI+THlfIjBhZQpni9lbCBlRRZmw8c1sCObGZeIsJLJPmWdW7rF7cn1dqN3k6RTiCYzgFD66gBndQhwYwGMALvMKb8+y8Ox/O56y14OQzhzAH5+sXwhKU2A== ⌘ AAAB/nicbVA9SwNBEJ2LXzF+RS1tDoNgFe7ErzJoYxnBxEByhLnNXrJkd+/Y3RPCEfA32GptJ7b+FUv/iZvkCpP4YODx3gwz88KEM20879sprKyurW8UN0tb2zu7e+X9g6aOU0Vog8Q8Vq0QNeVM0oZhhtNWoiiKkNPHcHg78R+fqNIslg9mlNBAYF+yiBE0Vmp1+igEdv1uueJVvSncZeLnpAI56t3yT6cXk1RQaQhHrdu+l5ggQ2UY4XRc6qSaJkiG2KdtSyUKqoNseu/YPbFKz41iZUsad6r+nchQaD0Soe0UaAZ60ZuI/3nt1ETXQcZkkhoqyWxRlHLXxO7kebfHFCWGjyxBopi91SUDVEiMjWhuSyjGNhN/MYFl0jyr+pfVi/vzSu0mT6cIR3AMp+DDFdTgDurQAAIcXuAV3pxn5935cD5nrQUnnzmEOThfv33nllk= 1 AAAB/nicbVDLSgNBEOyNrxhfUY9eBoPgKewGX8egF48RzAOSJfROJsmQmd1lZlYIS8Bv8Kpnb+LVX/HonzhJ9mASCxqKqm66u4JYcG1c99vJra1vbG7ltws7u3v7B8XDo4aOEkVZnUYiUq0ANRM8ZHXDjWCtWDGUgWDNYHQ39ZtPTGkehY9mHDNf4iDkfU7RWKnVGaCU2K10iyW37M5AVomXkRJkqHWLP51eRBPJQkMFat323Nj4KSrDqWCTQifRLEY6wgFrWxqiZNpPZ/dOyJlVeqQfKVuhITP170SKUuuxDGynRDPUy95U/M9rJ6Z/46c8jBPDQjpf1E8EMRGZPk96XDFqxNgSpIrbWwkdokJqbEQLWwI5sZl4ywmskkal7F2VLx8uStXbLJ08nMApnIMH11CFe6hBHSgIeIFXeHOenXfnw/mct+acbOYYFuB8/QJ/epZa 2 AAAB/nicbVDLSgNBEOyNrxhfUY9eFoPgKez6Pga9eIxgHpAsoXcymwyZmV1mZoUQAn6DVz17E6/+ikf/xEmyB5NY0FBUddPdFSacaeN5305uZXVtfSO/Wdja3tndK+4f1HWcKkJrJOaxaoaoKWeS1gwznDYTRVGEnDbCwd3EbzxRpVksH80woYHAnmQRI2is1Gz3UAjsnHeKJa/sTeEuEz8jJchQ7RR/2t2YpIJKQzhq3fK9xAQjVIYRTseFdqppgmSAPdqyVKKgOhhN7x27J1bpulGsbEnjTtW/EyMUWg9FaDsFmr5e9Cbif14rNdFNMGIySQ2VZLYoSrlrYnfyvNtlihLDh5YgUcze6pI+KiTGRjS3JRRjm4m/mMAyqZ+V/avy5cNFqXKbpZOHIziGU/DhGipwD1WoAQEOL/AKb86z8+58OJ+z1pyTzRzCHJyvX4ENlls= 3 AAAB/nicbVDLSgNBEOyNrxhfUY9eBoPgKexKfByDXjxGMA9IltA7mSRDZnaXmVkhLAG/wauevYlXf8Wjf+Ik2YNJLGgoqrrp7gpiwbVx3W8nt7a+sbmV3y7s7O7tHxQPjxo6ShRldRqJSLUC1EzwkNUNN4K1YsVQBoI1g9Hd1G8+MaV5FD6accx8iYOQ9zlFY6VWZ4BSYrfSLZbcsjsDWSVeRkqQodYt/nR6EU0kCw0VqHXbc2Pjp6gMp4JNCp1EsxjpCAesbWmIkmk/nd07IWdW6ZF+pGyFhszUvxMpSq3HMrCdEs1QL3tT8T+vnZj+jZ/yME4MC+l8UT8RxERk+jzpccWoEWNLkCpubyV0iAqpsREtbAnkxGbiLSewShoXZe+qfPlQKVVvs3TycAKncA4eXEMV7qEGdaAg4AVe4c15dt6dD+dz3ppzspljWIDz9QuCoJZc 4

Customize the loss function as you like F0 AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sRh/eYWV1bX1jeJmaWt7Z3evvH/QNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0cK9cwVU8A1omfk4qkKPeK393+zFNJVOWCmJMx8eJDTKiLaeCTUrd1LCE0BEZsI6jikhmgmz26gSdOKWPoli7URbN1N8XGZHGjGXoNiWxQ7PoTcV/vVAuJNvoMsi4SlLLFJ0HR6lANkbTHlCfa0atGDtCqObud0SHRBNqXVslV4q/WMEyaZ5VfVz1784rtau8niIcwTGcgg8XUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/wJlBM= AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sRh/eYWV1bX1jeJmaWt7Z3evvH/QNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0cK9cwVU8A1omfk4qkKPeK393+zFNJVOWCmJMx8eJDTKiLaeCTUrd1LCE0BEZsI6jikhmgmz26gSdOKWPoli7URbN1N8XGZHGjGXoNiWxQ7PoTcV/vVAuJNvoMsi4SlLLFJ0HR6lANkbTHlCfa0atGDtCqObud0SHRBNqXVslV4q/WMEyaZ5VfVz1784rtau8niIcwTGcgg8XUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/wJlBM= AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sRh/eYWV1bX1jeJmaWt7Z3evvH/QNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0cK9cwVU8A1omfk4qkKPeK393+zFNJVOWCmJMx8eJDTKiLaeCTUrd1LCE0BEZsI6jikhmgmz26gSdOKWPoli7URbN1N8XGZHGjGXoNiWxQ7PoTcV/vVAuJNvoMsi4SlLLFJ0HR6lANkbTHlCfa0atGDtCqObud0SHRBNqXVslV4q/WMEyaZ5VfVz1784rtau8niIcwTGcgg8XUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/wJlBM= AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sRh/eYWV1bX1jeJmaWt7Z3evvH/QNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0cK9cwVU8A1omfk4qkKPeK393+zFNJVOWCmJMx8eJDTKiLaeCTUrd1LCE0BEZsI6jikhmgmz26gSdOKWPoli7URbN1N8XGZHGjGXoNiWxQ7PoTcV/vVAuJNvoMsi4SlLLFJ0HR6lANkbTHlCfa0atGDtCqObud0SHRBNqXVslV4q/WMEyaZ5VfVz1784rtau8niIcwTGcgg8XUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/wJlBM= F1 AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sb7/5RVWVtfWN4qbpa3tnd298v5B08SppqxBYxHrdkgME1yxhuVWsHaiGZGhYK1wdD31W49MGx6rBztOWCDJQPGIU2KddH/Tw71yxa/6M6BlgnNSgRz1Xvm7249pKpmyVBBjOthPbJARbTkVbFLqpoYlhI7IgHUcVUQyE2SzVyfoxCl9FMXajbJopv6+yIg0ZixDtymJHZpFbyr+64VyIdlGl0HGVZJapug8OEoFsjGa9oD6XDNqxdgRQjV3vyM6JJpQ69oquVLwYgXLpHlWxX4V351Xald5PUU4gmM4BQwXUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/2dlBQ= AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sb7/5RVWVtfWN4qbpa3tnd298v5B08SppqxBYxHrdkgME1yxhuVWsHaiGZGhYK1wdD31W49MGx6rBztOWCDJQPGIU2KddH/Tw71yxa/6M6BlgnNSgRz1Xvm7249pKpmyVBBjOthPbJARbTkVbFLqpoYlhI7IgHUcVUQyE2SzVyfoxCl9FMXajbJopv6+yIg0ZixDtymJHZpFbyr+64VyIdlGl0HGVZJapug8OEoFsjGa9oD6XDNqxdgRQjV3vyM6JJpQ69oquVLwYgXLpHlWxX4V351Xald5PUU4gmM4BQwXUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/2dlBQ= AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sb7/5RVWVtfWN4qbpa3tnd298v5B08SppqxBYxHrdkgME1yxhuVWsHaiGZGhYK1wdD31W49MGx6rBztOWCDJQPGIU2KddH/Tw71yxa/6M6BlgnNSgRz1Xvm7249pKpmyVBBjOthPbJARbTkVbFLqpoYlhI7IgHUcVUQyE2SzVyfoxCl9FMXajbJopv6+yIg0ZixDtymJHZpFbyr+64VyIdlGl0HGVZJapug8OEoFsjGa9oD6XDNqxdgRQjV3vyM6JJpQ69oquVLwYgXLpHlWxX4V351Xald5PUU4gmM4BQwXUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/2dlBQ= AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sb7/5RVWVtfWN4qbpa3tnd298v5B08SppqxBYxHrdkgME1yxhuVWsHaiGZGhYK1wdD31W49MGx6rBztOWCDJQPGIU2KddH/Tw71yxa/6M6BlgnNSgRz1Xvm7249pKpmyVBBjOthPbJARbTkVbFLqpoYlhI7IgHUcVUQyE2SzVyfoxCl9FMXajbJopv6+yIg0ZixDtymJHZpFbyr+64VyIdlGl0HGVZJapug8OEoFsjGa9oD6XDNqxdgRQjV3vyM6JJpQ69oquVLwYgXLpHlWxX4V351Xald5PUU4gmM4BQwXUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/2dlBQ= F2 AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFkUxGVF+4B2KJk004YmmSHJCGXsJ7jVvTtx68+49UtM21lo64ELh3Pu5VxOmAhurOd9oZXVtfWNzcJWcXtnd2+/dHDYNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0qr1S2at4M+Bl4uekDDnqvdJ3tx/TVDJlqSDGdHwvsUFGtOVUsEmxmxqWEDoiA9ZxVBHJTJDNXp3gU6f0cRRrN8rimfr7IiPSmLEM3aYkdmgWvan4rxfKhWQbXQYZV0lqmaLz4CgV2MZ42gPuc82oFWNHCNXc/Y7pkGhCrWur6ErxFytYJs1qxfcq/t15uXaV11OAYziBM/DhAmpwC3VoAIUBPMMLvKIn9Ibe0cd8dQXlN0fwB+jzB/8xlBU= AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFkUxGVF+4B2KJk004YmmSHJCGXsJ7jVvTtx68+49UtM21lo64ELh3Pu5VxOmAhurOd9oZXVtfWNzcJWcXtnd2+/dHDYNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0qr1S2at4M+Bl4uekDDnqvdJ3tx/TVDJlqSDGdHwvsUFGtOVUsEmxmxqWEDoiA9ZxVBHJTJDNXp3gU6f0cRRrN8rimfr7IiPSmLEM3aYkdmgWvan4rxfKhWQbXQYZV0lqmaLz4CgV2MZ42gPuc82oFWNHCNXc/Y7pkGhCrWur6ErxFytYJs1qxfcq/t15uXaV11OAYziBM/DhAmpwC3VoAIUBPMMLvKIn9Ibe0cd8dQXlN0fwB+jzB/8xlBU= AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFkUxGVF+4B2KJk004YmmSHJCGXsJ7jVvTtx68+49UtM21lo64ELh3Pu5VxOmAhurOd9oZXVtfWNzcJWcXtnd2+/dHDYNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0qr1S2at4M+Bl4uekDDnqvdJ3tx/TVDJlqSDGdHwvsUFGtOVUsEmxmxqWEDoiA9ZxVBHJTJDNXp3gU6f0cRRrN8rimfr7IiPSmLEM3aYkdmgWvan4rxfKhWQbXQYZV0lqmaLz4CgV2MZ42gPuc82oFWNHCNXc/Y7pkGhCrWur6ErxFytYJs1qxfcq/t15uXaV11OAYziBM/DhAmpwC3VoAIUBPMMLvKIn9Ibe0cd8dQXlN0fwB+jzB/8xlBU= AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFkUxGVF+4B2KJk004YmmSHJCGXsJ7jVvTtx68+49UtM21lo64ELh3Pu5VxOmAhurOd9oZXVtfWNzcJWcXtnd2+/dHDYNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0qr1S2at4M+Bl4uekDDnqvdJ3tx/TVDJlqSDGdHwvsUFGtOVUsEmxmxqWEDoiA9ZxVBHJTJDNXp3gU6f0cRRrN8rimfr7IiPSmLEM3aYkdmgWvan4rxfKhWQbXQYZV0lqmaLz4CgV2MZ42gPuc82oFWNHCNXc/Y7pkGhCrWur6ErxFytYJs1qxfcq/t15uXaV11OAYziBM/DhAmpwC3VoAIUBPMMLvKIn9Ibe0cd8dQXlN0fwB+jzB/8xlBU= F3 AAAB+nicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegIB4jmgckS5idzCZD5rHMzAphzSd41bs38erPePVLnCR70GhBQ1HVTTUVJZwZ6/ufXmFpeWV1rbhe2tjc2t4p7+41jUo1oQ2iuNLtCBvKmaQNyyyn7URTLCJOW9Hoauq3Hqg2TMl7O05oKPBAspgRbJ10d9077ZUrftWfAf0lQU4qkKPeK391+4qkgkpLODamE/iJDTOsLSOcTkrd1NAEkxEe0I6jEgtqwmz26gQdOaWPYqXdSItm6s+LDAtjxiJymwLboVn0puK/XiQWkm18EWZMJqmlksyD45Qjq9C0B9RnmhLLx45gopn7HZEh1phY11bJlRIsVvCXNE+qgV8Nbs8qtcu8niIcwCEcQwDnUIMbqEMDCAzgCZ7hxXv0Xr03732+WvDym334Be/jGwDUlBY= AAAB+nicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegIB4jmgckS5idzCZD5rHMzAphzSd41bs38erPePVLnCR70GhBQ1HVTTUVJZwZ6/ufXmFpeWV1rbhe2tjc2t4p7+41jUo1oQ2iuNLtCBvKmaQNyyyn7URTLCJOW9Hoauq3Hqg2TMl7O05oKPBAspgRbJ10d9077ZUrftWfAf0lQU4qkKPeK391+4qkgkpLODamE/iJDTOsLSOcTkrd1NAEkxEe0I6jEgtqwmz26gQdOaWPYqXdSItm6s+LDAtjxiJymwLboVn0puK/XiQWkm18EWZMJqmlksyD45Qjq9C0B9RnmhLLx45gopn7HZEh1phY11bJlRIsVvCXNE+qgV8Nbs8qtcu8niIcwCEcQwDnUIMbqEMDCAzgCZ7hxXv0Xr03732+WvDym334Be/jGwDUlBY= AAAB+nicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegIB4jmgckS5idzCZD5rHMzAphzSd41bs38erPePVLnCR70GhBQ1HVTTUVJZwZ6/ufXmFpeWV1rbhe2tjc2t4p7+41jUo1oQ2iuNLtCBvKmaQNyyyn7URTLCJOW9Hoauq3Hqg2TMl7O05oKPBAspgRbJ10d9077ZUrftWfAf0lQU4qkKPeK391+4qkgkpLODamE/iJDTOsLSOcTkrd1NAEkxEe0I6jEgtqwmz26gQdOaWPYqXdSItm6s+LDAtjxiJymwLboVn0puK/XiQWkm18EWZMJqmlksyD45Qjq9C0B9RnmhLLx45gopn7HZEh1phY11bJlRIsVvCXNE+qgV8Nbs8qtcu8niIcwCEcQwDnUIMbqEMDCAzgCZ7hxXv0Xr03732+WvDym334Be/jGwDUlBY= AAAB+nicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegIB4jmgckS5idzCZD5rHMzAphzSd41bs38erPePVLnCR70GhBQ1HVTTUVJZwZ6/ufXmFpeWV1rbhe2tjc2t4p7+41jUo1oQ2iuNLtCBvKmaQNyyyn7URTLCJOW9Hoauq3Hqg2TMl7O05oKPBAspgRbJ10d9077ZUrftWfAf0lQU4qkKPeK391+4qkgkpLODamE/iJDTOsLSOcTkrd1NAEkxEe0I6jEgtqwmz26gQdOaWPYqXdSItm6s+LDAtjxiJymwLboVn0puK/XiQWkm18EWZMJqmlksyD45Qjq9C0B9RnmhLLx45gopn7HZEh1phY11bJlRIsVvCXNE+qgV8Nbs8qtcu8niIcwCEcQwDnUIMbqEMDCAzgCZ7hxXv0Xr03732+WvDym334Be/jGwDUlBY= F10 AAAB/XicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7APaoWTSTBubZIYkI5Sh+Atude9O3Potbv0S03YW2nrgwuGcezmXEyaCG4vxl1dYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0ybXis7u04YYEkA8UjTol1UvOml/l40itXcBXPgJaJn5MK5Kj3yt/dfkxTyZSlghjT8XFig4xoy6lgk1I3NSwhdEQGrOOoIpKZIJt9O0EnTumjKNZulEUz9fdFRqQxYxm6TUns0Cx6U/FfL5QLyTa6DDKuktQyRefBUSqQjdG0CtTnmlErxo4Qqrn7HdEh0YRaV1jJleIvVrBMmmdVH1f9u/NK7SqvpwhHcAyn4MMF1OAW6tAACg/wDC/w6j15b9679zFfLXj5zSH8gff5A0PKlVo= AAAB/XicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7APaoWTSTBubZIYkI5Sh+Atude9O3Potbv0S03YW2nrgwuGcezmXEyaCG4vxl1dYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0ybXis7u04YYEkA8UjTol1UvOml/l40itXcBXPgJaJn5MK5Kj3yt/dfkxTyZSlghjT8XFig4xoy6lgk1I3NSwhdEQGrOOoIpKZIJt9O0EnTumjKNZulEUz9fdFRqQxYxm6TUns0Cx6U/FfL5QLyTa6DDKuktQyRefBUSqQjdG0CtTnmlErxo4Qqrn7HdEh0YRaV1jJleIvVrBMmmdVH1f9u/NK7SqvpwhHcAyn4MMF1OAW6tAACg/wDC/w6j15b9679zFfLXj5zSH8gff5A0PKlVo= AAAB/XicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7APaoWTSTBubZIYkI5Sh+Atude9O3Potbv0S03YW2nrgwuGcezmXEyaCG4vxl1dYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0ybXis7u04YYEkA8UjTol1UvOml/l40itXcBXPgJaJn5MK5Kj3yt/dfkxTyZSlghjT8XFig4xoy6lgk1I3NSwhdEQGrOOoIpKZIJt9O0EnTumjKNZulEUz9fdFRqQxYxm6TUns0Cx6U/FfL5QLyTa6DDKuktQyRefBUSqQjdG0CtTnmlErxo4Qqrn7HdEh0YRaV1jJleIvVrBMmmdVH1f9u/NK7SqvpwhHcAyn4MMF1OAW6tAACg/wDC/w6j15b9679zFfLXj5zSH8gff5A0PKlVo= AAAB/XicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7APaoWTSTBubZIYkI5Sh+Atude9O3Potbv0S03YW2nrgwuGcezmXEyaCG4vxl1dYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0ybXis7u04YYEkA8UjTol1UvOml/l40itXcBXPgJaJn5MK5Kj3yt/dfkxTyZSlghjT8XFig4xoy6lgk1I3NSwhdEQGrOOoIpKZIJt9O0EnTumjKNZulEUz9fdFRqQxYxm6TUns0Cx6U/FfL5QLyTa6DDKuktQyRefBUSqQjdG0CtTnmlErxo4Qqrn7HdEh0YRaV1jJleIvVrBMmmdVH1f9u/NK7SqvpwhHcAyn4MMF1OAW6tAACg/wDC/w6j15b9679zFfLXj5zSH8gff5A0PKlVo= http://arogozhnikov.github.io/2016/06/24/gradient_boosting_explained.html (softmax) Loss Gradient Regression (MSE) Regression (MAE) K-class classi caiton (cross entropy) MSE case AAACBHicbVDLSsNAFJ3UV62vqks3g0UQhJKIr2XRjcsKfUETy2QyaYfOJGHmRiihW7/Bra7diVv/w6V/4rTNwrYeuHA4517O5fiJ4Bps+9sqrKyurW8UN0tb2zu7e+X9g5aOU0VZk8YiVh2faCZ4xJrAQbBOohiRvmBtf3g38dtPTGkeRw0YJcyTpB/xkFMCRno8cxkQlwYx4EbP7pUrdtWeAi8TJycVlKPeK/+4QUxTySKggmjddewEvIwo4FSwcclNNUsIHZI+6xoaEcm0l02/HuMTowQ4jJWZCPBU/XuREan1SPpmUxIY6EVvIv7ndVMIb7yMR0kKLKKzoDAVGGI8qQAHXDEKYmQIoYqbXzEdEEUomKLmUnw5Np04iw0sk9Z51bmqXj5cVGq3eTtFdISO0Sly0DWqoXtUR01EkUIv6BW9Wc/Wu/Vhfc5WC1Z+c4jmYH39Al0vmHA= +⌘ · T0 AAACBHicbVDLSsNAFJ3UV62vqks3g0UQhJKIr2XRjcsKfUETy2QyaYfOJGHmRiihW7/Bra7diVv/w6V/4rTNwrYeuHA4517O5fiJ4Bps+9sqrKyurW8UN0tb2zu7e+X9g5aOU0VZk8YiVh2faCZ4xJrAQbBOohiRvmBtf3g38dtPTGkeRw0YJcyTpB/xkFMCRno8cxkQlwYx4EbP6ZUrdtWeAi8TJycVlKPeK/+4QUxTySKggmjddewEvIwo4FSwcclNNUsIHZI+6xoaEcm0l02/HuMTowQ4jJWZCPBU/XuREan1SPpmUxIY6EVvIv7ndVMIb7yMR0kKLKKzoDAVGGI8qQAHXDEKYmQIoYqbXzEdEEUomKLmUnw5Np04iw0sk9Z51bmqXj5cVGq3eTtFdISO0Sly0DWqoXtUR01EkUIv6BW9Wc/Wu/Vhfc5WC1Z+c4jmYH39Al7CmHE= +⌘ · T1 AAACBHicbVDLSsNAFJ34rPVVdelmsAiCUJLia1l047JCX9DEMplM2qGTSZi5EUro1m9wq2t34tb/cOmfOG2zsK0HLhzOuZdzOX4iuAbb/rZWVtfWNzYLW8Xtnd29/dLBYUvHqaKsSWMRq45PNBNcsiZwEKyTKEYiX7C2P7yb+O0npjSPZQNGCfMi0pc85JSAkR7PXQbEpUEMuNGr9kplu2JPgZeJk5MyylHvlX7cIKZpxCRQQbTuOnYCXkYUcCrYuOimmiWEDkmfdQ2VJGLay6Zfj/GpUQIcxsqMBDxV/15kJNJ6FPlmMyIw0IveRPzP66YQ3ngZl0kKTNJZUJgKDDGeVIADrhgFMTKEUMXNr5gOiCIUTFFzKX40Np04iw0sk1a14lxVLh8uyrXbvJ0COkYn6Aw56BrV0D2qoyaiSKEX9IrerGfr3fqwPmerK1Z+c4TmYH39AmBVmHI= +⌘ · T2 AAACBHicbVDLSsNAFJ34rPVVdelmsAiCUBLfy6IblxX6giaWyWTSDp1kwsyNUEK3foNbXbsTt/6HS//EaZuFbT1w4XDOvZzL8RPBNdj2t7W0vLK6tl7YKG5ube/slvb2m1qmirIGlUKqtk80EzxmDeAgWDtRjES+YC1/cDf2W09MaS7jOgwT5kWkF/OQUwJGejx1GRCXBhJwvXveLZXtij0BXiROTsooR61b+nEDSdOIxUAF0brj2Al4GVHAqWCjoptqlhA6ID3WMTQmEdNeNvl6hI+NEuBQKjMx4In69yIjkdbDyDebEYG+nvfG4n9eJ4Xwxst4nKTAYjoNClOBQeJxBTjgilEQQ0MIVdz8immfKELBFDWT4kcj04kz38AiaZ5VnKvK5cNFuXqbt1NAh+gInSAHXaMqukc11EAUKfSCXtGb9Wy9Wx/W53R1ycpvDtAMrK9fYeiYcw== +⌘ · T3 AAACBHicbVDLSsNAFJ34rPVVdelmsAiCUBLxuSu6cVmhL2himUwm7dBJJszcCCV06ze41bU7cet/uPRPnLZZ2NYDFw7n3Mu5HD8RXINtf1tLyyura+uFjeLm1vbObmlvv6llqihrUCmkavtEM8Fj1gAOgrUTxUjkC9byB3djv/XElOYyrsMwYV5EejEPOSVgpMdTlwFxaSAB17s33VLZrtgT4EXi5KSMctS6pR83kDSNWAxUEK07jp2AlxEFnAo2KrqpZgmhA9JjHUNjEjHtZZOvR/jYKAEOpTITA56ofy8yEmk9jHyzGRHo63lvLP7ndVIIr72Mx0kKLKbToDAVGCQeV4ADrhgFMTSEUMXNr5j2iSIUTFEzKX40Mp048w0skuZZxbmsXDycl6u3eTsFdIiO0Aly0BWqontUQw1EkUIv6BW9Wc/Wu/VhfU5Xl6z85gDNwPr6BWtamHk= +⌘ · T9

What impurity to use in regression of gradient values? A regression tree Partitions AAACDHicbVDLTgIxFO3gC/E1PnZuGomJC0Jm8LkkunGJRJAEJpNO6UBD25m0HRMk/ILf4FbX7oxb/8Glf2IHZiHgSdqcnHNP7s0JYkaVdpxvK7e0vLK6ll8vbGxube/Yu3tNFSUSkwaOWCRbAVKEUUEammpGWrEkiAeMPASDm9R/eCRS0Ujc62FMPI56goYUI20k3z6o+24J1v1K+p2WYKcbaeXbRafsTAAXiZuRIshQ8+0fk8MJJ0JjhpRqu06svRGSmmJGxoVOokiM8AD1SNtQgThR3mhy/RgeG6ULw0iaJzScqH8TI8SVGvLATHKk+2reS8X/vHaiwytvREWcaCLwdFGYMKgjmFYBu1QSrNnQEIQlNbdC3EcSYW0Km9kS8LHpxJ1vYJE0K2X3onx+d1asXmft5MEhOAInwAWXoApuQQ00AAZP4AW8gjfr2Xq3PqzP6WjOyjL7YAbW1y8bs5m2 R1, R2, R3, . . . Constants AAACDHicbVC7TsMwFHXKq5RXeGwsFhUSQ1Ul5TlWsDAWidJKbRQ5rtNatZ3IdpBK1F/gG1hhZkOs/AMjf4LTZqAtR7J1dM49ulcniBlV2nG+rcLS8srqWnG9tLG5tb1j7+49qCiRmDRxxCLZDpAijArS1FQz0o4lQTxgpBUMbzK/9UikopG416OYeBz1BQ0pRtpIvn2AfbcCsV/LvtMK7PYirXy77FSdCeAicXNSBjkavv1jcjjhRGjMkFId14m1lyKpKWZkXOomisQID1GfdAwViBPlpZPrx/DYKD0YRtI8oeFE/ZtIEVdqxAMzyZEeqHkvE//zOokOr7yUijjRRODpojBhUEcwqwL2qCRYs5EhCEtqboV4gCTC2hQ2syXgY9OJO9/AInmoVd2L6vndWbl+nbdTBIfgCJwAF1yCOrgFDdAEGDyBF/AK3qxn6936sD6nowUrz+yDGVhfv27Gmek= c1, c2, c3, . . . What we want to minimize: AAACJHicbZDLSgMxFIYz9VbrrerSTbAILZYyI942QlEQFy4q9AZtHTJp2oYmmSHJiGWYl/AhfAa3unYnLtwIvonpBbTVHwIf/zmHc/J7AaNK2/aHlZibX1hcSi6nVlbX1jfSm1tV5YcSkwr2mS/rHlKEUUEqmmpG6oEkiHuM1Lz+xbBeuyNSUV+U9SAgLY66gnYoRtpYbjrfVCF3I3rmxLcCXmcHLs3Dy2zT49F97NLcfvmHc246YxfskeBfcCaQAROV3PRXs+3jkBOhMUNKNRw70K0ISU0xI3GqGSoSINxHXdIwKBAnqhWNfhXDPeO0YceX5gkNR+7viQhxpQbcM50c6Z6arQ3N/2qNUHdOWxEVQaiJwONFnZBB7cNhRLBNJcGaDQwgLKm5FeIekghrE+TUFo/HJhNnNoG/UD0oOMeFo5vDTPF8kk4S7IBdkAUOOAFFcAVKoAIweABP4Bm8WI/Wq/VmvY9bE9ZkZhtMyfr8Bqm5pCU= n X i=1 L(yi, F(xi) + T(xi)) AAACXHicdZFPS8MwGMaz+t85rQpevASHMA+OdvjvKHrxqOJ0sM6SZmkXl6Q1ScVR+gX9Bl4Ev4FXPZluO7ipLwR+PO/z8iZPgoRRpR3ntWTNzM7NLywuLZdXKqtr9vrGrYpTiUkTxyyWrQApwqggTU01I61EEsQDRu6C/nnRv3siUtFY3OhBQjocRYKGFCNtJN/uehF5VAwJDfe9UCKcuXnWyOGIvYBGrOaplPuGefac+xR6VMBr/yGHkU8Lg9y7b+TZf6aeT3Pfrjp1Z1jwN7hjqIJxXfr2u9eNccqJ0Jghpdquk+hOhqSmmJF82UsVSRDuo4i0DQrEiepkwzRyuGuULgxjaY551lD9OZEhrtSAB8bJke6p6V4h/tVrpzo86WRUJKkmAo8WhSmDOoZFtLBLJcGaDQwgLKm5K8Q9ZHLU5gMmtgS8yMSdTuA33Dbq7lH98Oqgeno2TmcRbIMdUAMuOAan4AJcgibA4AV8gE/wVXqzZq2yVRlZrdJ4ZhNMlLX1DSf6uP4= > 1 2 P xi 2Rj gi 2 P xi 2Rj hi The minimum when we tune AAAB+XicbVC7TsNAEFyHVwivACXNiQiJKrIRrzKChjII8pASKzpfzsmRu7N1d0aKrHwCLdR0iJavoeRPuDguSMJIK41mdrW7E8ScaeO6305hZXVtfaO4Wdra3tndK+8fNHWUKEIbJOKRagdYU84kbRhmOG3HimIRcNoKRrdTv/VMlWaRfDTjmPoCDyQLGcHGSg+k99QrV9yqmwEtEy8nFchR75V/uv2IJIJKQzjWuuO5sfFTrAwjnE5K3UTTGJMRHtCOpRILqv00O3WCTqzSR2GkbEmDMvXvRIqF1mMR2E6BzVAvelPxP6+TmPDaT5mME0MlmS0KE45MhKZ/oz5TlBg+tgQTxeytiAyxwsTYdOa2BGJiM/EWE1gmzbOqd1m9uD+v1G7ydIpwBMdwCh5cQQ3uoA4NIDCAF3iFNyd13p0P53PWWnDymUOYg/P1C/UBlGQ= cj Use this for the impurity for AAAB+XicdVDLTgJBEOz1ifhCPXqZSEw8bRYCAjeiF4/44JHAhswOszAyO7uZmTUhGz7Bq569Ga9+jUf/xAHWRIxW0kmlqjvdXV7EmdKO82GtrK6tb2xmtrLbO7t7+7mDw5YKY0lok4Q8lB0PK8qZoE3NNKedSFIceJy2vfHlzG8/UKlYKO70JKJugIeC+YxgbaTbm/59P5d3bGcO5NjFWrVcLRlSKddKhhRSKw8pGv3cZ28QkjigQhOOleoWnEi7CZaaEU6n2V6saITJGA9p11CBA6rcZH7qFJ0aZYD8UJoSGs3VnxMJDpSaBJ7pDLAeqd/eTPzL68bar7oJE1GsqSCLRX7MkQ7R7G80YJISzSeGYCKZuRWREZaYaJPO0hYvmJpMvh9H/5NW0S6c2+XrUr5+kaaTgWM4gTMoQAXqcAUNaAKBITzCEzxbifVivVpvi9YVK505giVY719Gz5Sc Rj AAACMnicbZDLSgMxFIYzXmu9VV26CRahRSkz4g1BKAriwkUVe4FOGTJpqmmTzJBkxDLMu/gQPoNb3equuPUhTNsRvB0IfPz/OZyT3w8ZVdq2X62JyanpmdnMXHZ+YXFpObeyWlNBJDGp4oAFsuEjRRgVpKqpZqQRSoK4z0jd750O/fodkYoG4lr3Q9Li6EbQDsVIG8nLHR1DV0Xci7tJCq7P4/vEo9ClAl55Rr8o9D26Dc8KX1ZxC3vdopfL2yV7VPAvOCnkQVoVLzdw2wGOOBEaM6RU07FD3YqR1BQzkmTdSJEQ4R66IU2DAnGiWvHojwncNEobdgJpntBwpH6fiBFXqs9908mRvlW/vaH4n9eMdOewFVMRRpoIPF7UiRjUARwGBttUEqxZ3wDCkppbIb5FEmFtYv2xxeeJycT5ncBfqO2UnP3S3uVuvnySppMB62ADFIADDkAZnIMKqAIMHsATeAYv1qP1Zg2s93HrhJXOrIEfZX18AkBPqhI= = X j X xi 2Rj L(yi, F(xi) + cj) Separate cases by partition Constant Quadratic Taylor approximation AAACEXicbVDLSsNAFJ3UV62vqCtxM1jEFqQk4msjFAVx4aKCfUBbwmQ6bYfOJGFmIoYQ/Ai/wa2u3Ylbv8Clf+K0zcK2HrhwOOde7r3HDRiVyrK+jczc/MLiUnY5t7K6tr5hbm7VpB8KTKrYZ75ouEgSRj1SVVQx0ggEQdxlpO4OroZ+/YEISX3vXkUBaXPU82iXYqS05Jg7PYfCC3h7UIgcenhdaLk8fkwcWiw6Zt4qWSPAWWKnJA9SVBzzp9XxcciJpzBDUjZtK1DtGAlFMSNJrhVKEiA8QD3S1NRDnMh2PHohgfta6cCuL3R5Co7UvxMx4lJG3NWdHKm+nPaG4n9eM1Td83ZMvSBUxMPjRd2QQeXDYR6wQwXBikWaICyovhXiPhIIK53axBaXJzoTezqBWVI7KtmnpZO743z5Mk0nC3bBHigAG5yBMrgBFVAFGDyBF/AK3oxn4934MD7HrRkjndkGEzC+fgF36Zwc gi = L0(yi, F(xi)) AAACEnicbVDLSgMxFM34rPVVdaebYJG2IGVGfG2EoiAuXFSwD2iHIZNm2tAkMyQZsQwFP8JvcKtrd+LWH3Dpn5g+Frb1wIXDOfdy7z1+xKjStv1tzc0vLC4tp1bSq2vrG5uZre2qCmOJSQWHLJR1HynCqCAVTTUj9UgSxH1Gan73auDXHohUNBT3uhcRl6O2oAHFSBvJy+x2PAov4G0ul+959PA63/R58tj3aKHgZbJ20R4CzhJnTLJgjLKX+Wm2QhxzIjRmSKmGY0faTZDUFDPSTzdjRSKEu6hNGoYKxIlyk+EPfXhglBYMQmlKaDhU/04kiCvV477p5Eh31LQ3EP/zGrEOzt2EiijWRODRoiBmUIdwEAhsUUmwZj1DEJbU3ApxB0mEtYltYovP+yYTZzqBWVI9KjqnxZO742zpcpxOCuyBfZAHDjgDJXADyqACMHgCL+AVvFnP1rv1YX2OWues8cwOmID19Qvg8pxO hi = L00(yi, F(xi)) AAACZXicbZFNa9tAEIZXcj8cp2ndJvTSQ5eagk2LkUw/cjQplB56cEucGCxXrNYre+3dldgdFQuhX5lTjjn1N/TWta1C7GRg4eGdd5jh3SgV3IDnXTtu7cHDR4/rB43DJ0dPnzWfv7gwSaYpG9JEJHoUEcMEV2wIHAQbpZoRGQl2GS2/rPuXv5k2PFHnkKdsIslM8ZhTAlYKmzIgaaqTFQ5MJsNiUVYQRLJYlSHHAVf4Z7jWBYuh3fjezkP+Hn9t/3d0OvgdnlknDReWglgTWvhl0SvxfKv+6uFA89kcOmGz5XW9TeG74FfQQlUNwuZNME1oJpkCKogxY99LYVIQDZwKVjaCzLCU0CWZsbFFRSQzk2ITS4nfWmWK40TbpwBv1NsTBZHG5DKyTklgbvZ7a/G+3jiD+HRScJVmwBTdLoozgSHB64zxlGtGQeQWCNXc3orpnNhcwP7EzpZIljYTfz+Bu3DR6/qfuh9/fGj1z6p06ugVeoPayEefUR99QwM0RBRdob+O69ScP+6Re+K+3Fpdp5o5Rjvlvv4H7iy3PA== ⇡ X j X xi 2Rj ✓ L(yi, F(xi)) + gicj + 1 2 hic2 j ◆ AAACMHicbZDLTgIxFIY7eEO8jbp000gMEBIyQ7ywMSG6cYmJXBIYSad0oKFzSdsxksm8ig/hM7jVta4MW5/CDsxCwJM0/fKf8+e0vx0wKqRhfGmZtfWNza3sdm5nd2//QD88agk/5Jg0sc983rGRIIx6pCmpZKQTcIJcm5G2Pb5N+u0nwgX1vQc5CYjloqFHHYqRVFJfrznF5/KoBK+hghIsQ6eQ3CNFPYcjHJlxVI2dwlx9rCb6wJci19fzRsWYFVwFM4U8SKvR16fKiEOXeBIzJETXNAJpRYhLihmJc71QkADhMRqSrkIPuURY0eyHMTxTygA6PlfHk3Cm/nVEyBVi4tpq0kVyJJZ7ifhfrxtKp2ZF1AtCSTw8X+SEDEofJnHBAeUESzZRgDCn6q0Qj5AKRqpQF7bYbqwyMZcTWIVWtWJeVi7uz/P1mzSdLDgBp6AITHAF6uAONEATYPAC3sA7+NBetU/tW5vORzNa6jkGC6X9/AJlyaUI f(x + h) = f(x) + f0(x)h + 1 2 f00(x)h2 + . . . Taylor approximation

Impurity improvement (search criterion for best split) AAAB93icbVDLSgNBEOyNrxhfUY9eFoPgKeyKr2PQi8cEzAOSJcxOepMhM7PLzKywhHyBVz17E69+jkf/xEmyBxMtaCiquunuChPOtPG8L6ewtr6xuVXcLu3s7u0flA+PWjpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2O72d++wmVZrF8NFmCgSBDySJGibFSQ/fLFa/qzeH+JX5OKpCj3i9/9wYxTQVKQznRuut7iQkmRBlGOU5LvVRjQuiYDLFrqSQCdTCZHzp1z6wycKNY2ZLGnau/JyZEaJ2J0HYKYkZ61ZuJ/3nd1ES3wYTJJDUo6WJRlHLXxO7sa3fAFFLDM0sIVcze6tIRUYQam83SllBMbSb+agJ/Seui6l9XrxqXldpdnk4RTuAUzsGHG6jBA9ShCRQQnuEFXp3MeXPenY9Fa8HJZ45hCc7nD4aqk5c= s AAADGXicjVJNaxRBEO0ZPxLXr40eBWkMwi6SZWbBJBchJCIePERxN4Gdsenp7Zlt0t0zdNdIlmHO/oQc/A1e9exNvHryqL/Enp09mA9ICpp+vKp6VbzupJDCQhD89vxr12/cXFm91bl95+69+921B2Obl4bxEctlbg4TarkUmo9AgOSHheFUJZIfJEd7Tf7gIzdW5Po9zAseK5ppkQpGwVFkzcPRSy6B9mwfv8AbUWooq8K6GtY4kjyFSaelokRkshfZUhGHVXVcE4EjofG7GmdENGnT/zCsq4tLZkTUeKOV7F0qSd5cQbQpamQ7zy7Xu8qSZLlmJzIim0F/ecekux4MgkXg8yBcgvWdVyeP/+6NT/ZJ9080zVmpuAYmqbWTMCggrqgBwSR3+qXlBWVHNOMTBzVV3MbV4ilr/NQxU5zmxh0NeMH+31FRZe1cJa5SUZjZs7mGvCg3KSHdjiuhixK4Zu2gtJQYctz8CzwVhjOQcwcoM8LtitmMOl/B/Z5TUxJVO0/Csw6cB+PhINwcPH/rzNlFbayiR+gJ6qEQbaEd9BrtoxFi3ifvi/fV++Z/9r/7P/yfbanvLXseolPh//oHxxUA8w== (s) = 1 2 " P xi 2R gi 2 P xi 2R hi P xi 2RL gi 2 P xi 2RL hi + P xi 2RR gi 2 P xi 2RR hi !# The impurity improvement for split s is given by We just need to calculate AAAB/nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PQi8cI5gHJEmYns8mQmdllZlYMy4Lf4FXP3sSrv+LRP3GS7MEkFjQUVd10dwUxZ9q47rdTWFldW98obpa2tnd298r7B00dJYrQBol4pNoB1pQzSRuGGU7bsaJYBJy2gtHtxG89UqVZJB/MOKa+wAPJQkawsVK7G4j0KeuxXrniVt0p0DLxclKBHPVe+afbj0giqDSEY607nhsbP8XKMMJpVuommsaYjPCAdiyVWFDtp9N7M3RilT4KI2VLGjRV/06kWGg9FoHtFNgM9aI3Ef/zOokJr/2UyTgxVJLZojDhyERo8jzqM0WJ4WNLMFHM3orIECtMjI1obksgMpuJt5jAMmmeVb3L6sX9eaV2k6dThCM4hlPw4ApqcAd1aAABDi/wCm/Os/PufDifs9aCk88cwhycr181QZbN xi AAACEXicbVDLSsNAFJ3UV62vqCtxM1jEFqQk4msjFAVx4aKCfUBbwmQ6bYfOJGFmIoYQ/Ai/wa2u3Ylbv8Clf+K0zcK2HrhwOOde7r3HDRiVyrK+jczc/MLiUnY5t7K6tr5hbm7VpB8KTKrYZ75ouEgSRj1SVVQx0ggEQdxlpO4OroZ+/YEISX3vXkUBaXPU82iXYqS05Jg7PYfCC3h7UIgcenhdaLk8fkwcWiw6Zt4qWSPAWWKnJA9SVBzzp9XxcciJpzBDUjZtK1DtGAlFMSNJrhVKEiA8QD3S1NRDnMh2PHohgfta6cCuL3R5Co7UvxMx4lJG3NWdHKm+nPaG4n9eM1Td83ZMvSBUxMPjRd2QQeXDYR6wQwXBikWaICyovhXiPhIIK53axBaXJzoTezqBWVI7KtmnpZO743z5Mk0nC3bBHigAG5yBMrgBFVAFGDyBF/AK3oxn4934MD7HrRkjndkGEzC+fgF36Zwc gi = L0(yi, F(xi)) AAACEnicbVDLSgMxFM34rPVVdaebYJG2IGVGfG2EoiAuXFSwD2iHIZNm2tAkMyQZsQwFP8JvcKtrd+LWH3Dpn5g+Frb1wIXDOfdy7z1+xKjStv1tzc0vLC4tp1bSq2vrG5uZre2qCmOJSQWHLJR1HynCqCAVTTUj9UgSxH1Gan73auDXHohUNBT3uhcRl6O2oAHFSBvJy+x2PAov4G0ul+959PA63/R58tj3aKHgZbJ20R4CzhJnTLJgjLKX+Wm2QhxzIjRmSKmGY0faTZDUFDPSTzdjRSKEu6hNGoYKxIlyk+EPfXhglBYMQmlKaDhU/04kiCvV477p5Eh31LQ3EP/zGrEOzt2EiijWRODRoiBmUIdwEAhsUUmwZj1DEJbU3ApxB0mEtYltYovP+yYTZzqBWVI9KjqnxZO742zpcpxOCuyBfZAHDjgDJXADyqACMHgCL+AVvFnP1rv1YX2OWues8cwOmID19Qvg8pxO hi = L00(yi, F(xi)) for each sample Calculation of this score is straightforward. Regression tree learning explicitly defined as minimizing the loss function: AAACJHicbZDLSgMxFIYz9VbrrerSTbAILZYyI942QlEQFy4q9AZtHTJp2oYmmSHJiGWYl/AhfAa3unYnLtwIvonpBbTVHwIf/zmHc/J7AaNK2/aHlZibX1hcSi6nVlbX1jfSm1tV5YcSkwr2mS/rHlKEUUEqmmpG6oEkiHuM1Lz+xbBeuyNSUV+U9SAgLY66gnYoRtpYbjrfVCF3I3rmxLcCXmcHLs3Dy2zT49F97NLcfvmHc246YxfskeBfcCaQAROV3PRXs+3jkBOhMUNKNRw70K0ISU0xI3GqGSoSINxHXdIwKBAnqhWNfhXDPeO0YceX5gkNR+7viQhxpQbcM50c6Z6arQ3N/2qNUHdOWxEVQaiJwONFnZBB7cNhRLBNJcGaDQwgLKm5FeIekghrE+TUFo/HJhNnNoG/UD0oOMeFo5vDTPF8kk4S7IBdkAUOOAFFcAVKoAIweABP4Bm8WI/Wq/VmvY9bE9ZkZhtMyfr8Bqm5pCU= n X i=1 L(yi, F(xi) + T(xi)) AAACMnicbZDLSgMxFIYzXmu9VV26CRahRSkz4g1BKAriwkUVe4FOGTJpqmmTzJBkxDLMu/gQPoNb3equuPUhTNsRvB0IfPz/OZyT3w8ZVdq2X62JyanpmdnMXHZ+YXFpObeyWlNBJDGp4oAFsuEjRRgVpKqpZqQRSoK4z0jd750O/fodkYoG4lr3Q9Li6EbQDsVIG8nLHR1DV0Xci7tJCq7P4/vEo9ClAl55Rr8o9D26Dc8KX1ZxC3vdopfL2yV7VPAvOCnkQVoVLzdw2wGOOBEaM6RU07FD3YqR1BQzkmTdSJEQ4R66IU2DAnGiWvHojwncNEobdgJpntBwpH6fiBFXqs9908mRvlW/vaH4n9eMdOewFVMRRpoIPF7UiRjUARwGBttUEqxZ3wDCkppbIb5FEmFtYv2xxeeJycT5ncBfqO2UnP3S3uVuvnySppMB62ADFIADDkAZnIMKqAIMHsATeAYv1qP1Zg2s93HrhJXOrIEfZX18AkBPqhI= = X j X xi 2Rj L(yi, F(xi) + cj) Impurity before split Impurity after split

Method 6: XGBoost XGBoost (Chen & Guestrin, KDD ) Very fast due to huge engineering e orts (parallelization, sparsity, cache, ) Flexible and largely customizable with many detailed options The splitting criterion for the base regression-tree learning directly comes from ( ) Combined use of Randomize type: Bagging-like operation (row- and column- sampling) can be used together Regression tree learning explicitly defined as minimizing the loss function with regularization Regularized gradient boosted trees . . . . . ( ) AAADDXicbZJNb9MwGMed8DbGyzrgxsWiQuq0qXLSNUtuE0iIA4eB6DapKZHjuK07O4liB1G5+Qx8Bq5w5oa47jPsyDfBTcrL1j2S5b+ex8/Pfz9ynHMmFUIXln3j5q3bdzbubt67/+DhVmv70bHMyoLQAcl4VpzGWFLOUjpQTHF6mhcUi5jTk/js5bJ+8pEWkmXpezXP6UjgScrGjGBlUtG29SSUpYj0rIKNCGOhP1URgyFL4bvI5N905hHbg686f0o7uySa7cBdqMPawLCYxCONusjrByjYQ90+cvxaIOT5bs+gJ1gIbHaSZAouQm0AYbW4BmHC8+pOg3CMCALfdQPTys2jkr+McFxgop1KuyvjM2iYH9x1ZOB7aP+gRvq9Xq+xF9SuMM+n/4gNZWEwi6jVbrwgBNeFsxJtsIqjqPUrTDJSCpoqwrGUQwflaqRxoRjhtNoMS0lzTM7whA6NTLGgcqRrnxV8bjIJHGeFWamCdfb/Do2FlHMRm5MCq6m8Wlsmr6sNSzX2R5qlealoSpqLxiWHKoPLvwATVlCi+NwITApmvEIyxWawyvyYS7fEojIzca5OYF0cu13H6/bf7rcPX6ymswGegmegAxxwAA7Ba3AEBoBY2vpifbW+2Z/t7/YP+2dz1LZWPY/BpbDPfwO3o+vs X j X xi 2Rj L(yi, F(xi) + cj) + · |{cj }| + · 1 2 X j c2 j + ↵ · X j |cj | AAADO3iclVJNb9NAEF2brxI+msKRy4oK1AoarRvaphKgqnDgWCTSVoqtaL0ZJ6uu19buGEgs/6D+CH4C4kolLogb4to76yQH2nLJSCs9zcx7b3Y0ca6kRcbOPP/a9Rs3by3dbty5e+/+cnPlwaHNCiOgKzKVmeOYW1BSQxclKjjODfA0VnAUn7yp60cfwViZ6Q84ziFK+VDLRAqOLtVf8faTfhlOdXpmGEcla+12ttmLneesxVin3W47sMWC3c12RUOu8hGv1ibr9BUNYxhKXQpnbqvG5NmiKvQpndCXdGNhXhg2JouzarfXdGFa7cYcOUT4jGWGIzCfpIWqEYIezD/fb67WCnXQqyCYg9W99fOvG29/nB70mz/DQSaKFDQKxa3tBSzHqOQGpVC1dmEh5+KED6HnoOYp2KicTl7RJy4zoElm3NNIp9l/GSVPrR2nsetMOY7s5Vqd/F+tV2DSiUqp8wJBi5lRUiiKGa0vhw6kAYFq7AAXRrpZqRhxwwW6+7rgEqf1ToLLG7gKDjdbwXZr671bzj6ZxRJ5RB6TNRKQHbJH3pED0iXCO/W+ed+9M/+L/8v/7f+ZtfrenPOQXAj//C8p+Pmw f↵(z) = 8 > < > : z + ↵ z < ↵ z ↵ z > ↵ 0 otherwise AAAEfnictZNNb9MwGMe9NsAob904DiGLCdRqtDgp7ZID0sQQ4sBhINpNakrkuE5qzXmR7SCqKEc+wz4bRzjzIXCSHlhXTtMsRf7refw8/18ey37KmVQI/dxqNI1bt+9s323du//g4aP2zu5EJpkgdEwSnogzH0vKWUzHiilOz1JBceRzeuqfH5f5029USJbEX9QypbMIhzELGMFKh7ydxg/3HeUKd2QXvoE9NxCY5GaRWwV0OQ3UtFWHAi93K7epCP1ZjvqOPUKvD1+iPkL2YDDQYohMxxroOszTBda7z0LecWUW6Vo/yr8XHoMui+HnAoYeK9Oi+9Uq8rXOSK/RqOps2sjUwnFsy3JKIv1jc1wcbG668FgBezV35ya4vY/XJYcb0cu+JXzr4Eaorz3v/1CvRt5yBQsXqrvaZ/oK1h1GQwc5NbRdCe1oV/QhjiJt4LX3axCE4FVhrsT+0fuLp7+PJxcnXvuXO09IFtFYEY6lnJooVbMcC8UIp5opkzTF5ByHdKpljCMqZ3lFVcDnOjKHQSL0FytYRf+tyHEk5TLy9ckIq4Vcz5XBTblppgJ7lrM4zRSNSW0UZByqBJaPD86ZoETxpRaYCKZZIVlgfeNKP9FLLn5U6JmY6xO4KiZW3xz1h5/0cN6Cem2DPfAMdIAJDsER+ABOwBiQxp/mbnOv+cQAxgujZ7yqjza2VjWPwaVl2H8Bu3Njsw== (s) = 1 2 " f↵ P xi 2R gi 2 + P xi 2R hi f↵ P xi 2RL gi 2 + P xi 2RL hi + f↵ P xi 2RR gi 2 + P xi 2RR hi !#

Method 7: LightGBM Best-first expansions of regression tree recursion The successor to XGBoost that improves the e ciency and scalability when the feature dimension is high and data size is large. Best- rst expansion + Two novel techniques Gradient-based One-Side Sampling (GOSS): Reduce # of samples Exclude a signi cant proportion of data instances with small gradients Exclusive Feature Bundling (EFB): Reduce # of features Bundle mutually exclusive features (i.e., they rarely take nonzero values simultaneously) LightGBM (Ke+ NIPS ) Leaf-wise growth Level-wise growth c.f. depth-first, width-first Now this is widely used in many implementations e.g. scikit-learn Use max_leaf_nodes instead of max_depth for leaf-wise growth

XGBoost and LightGBM developed as open source implementations Also come with Scikit-learn AIPs Widely used methods have good implementations https://github.com/dmlc/xgboost https://github.com/Microsoft/LightGBM First Release 24 Apr 2017 (guolinke) First Release 27 Mar 2014 (tqchen) DMLC (Distributed Machine Learning Community) http://dmlc.ml DMTK (Distributed Machine Learning Toolkit) http://www.dmtk.io !pip install xgboost from xgboost import XGBClassifier from xgboost import XGBRegressor !pip install lightgbm from lightgbm import LGBMClassifier from lightgbm import LGBMRegressor

Code example Text text !pip install xgboost lightgbm from sklearn.ensemble import GradientBoostingClassifier from sklearn.ensemble import HistGradientBoostingClassifier from lightgbm import LGBMClassifier from xgboost import XGBClassifier from sklearn.datasets import load_breast_cancer from sklearn.model_selection import cross_validate def cv_5fold(clf, X, y, metrics=['accuracy', 'roc_auc']): scores = cross_validate(clf, X, y, cv=5, scoring=metrics, return_estimator=True) print(clf.__class__.__name__, {m: '%0.5f' % scores[f'test_{m}'].mean() for m in metrics}, 'fit_time', '%0.5f' % scores['fit_time'].mean()) return scores X, y = load_breast_cancer(return_X_y=True) result = cv_5fold(GradientBoostingClassifier(), X, y) result = cv_5fold(HistGradientBoostingClassifier(), X, y) result = cv_5fold(XGBClassifier(), X, y) result = cv_5fold(LGBMClassifier(), X, y) result = cv_5fold(LGBMClassifier(n_estimators=200, learning_rate=0.1, min_child_samples=5, reg_lambda=1.0), X, y) GradientBoostingClassifier {'accuracy': '0.95962', 'roc_auc': '0.99104'} fit_time 0.43648 HistGradientBoostingClassifier {'accuracy': '0.96660', 'roc_auc': '0.99268'} fit_time 0.35848 XGBClassifier {'accuracy': '0.96487', 'roc_auc': '0.99349'} fit_time 0.09564 LGBMClassifier {'accuracy': '0.96662', 'roc_auc': '0.99243'} fit_time 0.14272 LGBMClassifier {'accuracy': '0.97014', 'roc_auc': '0.99304'} fit_time 0.18123

Code example (with hyperparameter tuning) Text text from scipy.stats import uniform, randint from sklearn.utils.fixes import loguniform from sklearn.model_selection import RandomizedSearchCV model = RandomizedSearchCV(LGBMClassifier(), {"n_estimators": [100, 200, 500], "min_child_samples": [1, 5, 10, 50], "num_leaves": randint(3, 31), "reg_alpha": loguniform(1e-2, 1), "reg_lambda": loguniform(1e-2, 1), "learning_rate": loguniform(1e-3, 1), "subsample": uniform(0, 1), "colsample_bytree": uniform(0, 1)}, n_iter=20) result = cv_5fold(model, X, y) for e in result['estimator']: print(e.best_params_) RandomizedSearchCV {'accuracy': '0.97190', 'roc_auc': '0.99408'} fit_time 7.78688 {'colsample_bytree': 0.8157047506148358, 'learning_rate': 0.10355525231624815, 'min_child_sa {'colsample_bytree': 0.9631901921844805, 'learning_rate': 0.4912284442404967, 'min_child_sam {'colsample_bytree': 0.37151158305148924, 'learning_rate': 0.2702583980583705, 'min_child_sa {'colsample_bytree': 0.3435018330608688, 'learning_rate': 0.05038610774240793, 'min_child_sa {'colsample_bytree': 0.12849846893758976, 'learning_rate': 0.165067814006174, 'min_child_sam We should increase the number of iterations as much as possible for this large search grid (20 trials are too small)

Takeaways for Part 2: “Randomize” vs “Optimize” ‣ RandomForestRegressor max_depth=None, max_leaf_nodes=None ‣ ExtraTreesRegressor max_depth=None, max_leaf_nodes=None ‣ GradientBoostingRegressor max_depth=3, max_leaf_nodes=None ‣ HistGradientBoostingRegressor max_depth=None, max_leaf_nodes=31 ‣ XGBRegressor max_depth=3 ‣ LGBMRegressor num_leaves=31, max_depth=-1 • Use a large base tree (e.g. Fully-grown tree) • Each decision tree is trained to overfit. • Use a small base tree (Size-bounded tree) • Each decision tree is trained to underfit. “Weak" learner Scikit default values (Examples of Regressor) • Random Projection Ensemble (2017) • Random Composite Forest (2016) • Random Rotation Ensemble (2016) • Oblique Random Forests (2011) • Rotation Forests (2006) Other examples • Globally Induced Forest (ICML2017) • Regularized Greedy Forest (TPAMI 2014) • CatBoost (NeurIPS2018) • “Randomize” (Bagging, Random Forest) • “Optimize” (Boosting) Other examples

Remark 1: Integrating “Randomize” and “Optimize” strategies Modern implementations of the “Optimize” type can also use the “Randomize” type (Stochastic Gradient Boosting) • Friedman’s Stochastic Gradient Boosting (Comput Stat Data Anal, 2002) • Breiman’s Iterated Bagging (Mach Learn, 2001) • Ensemble on Random Patches (Louppe & Geurts, ECML PKDD 2012) The basic idea is to introduce 'Randomize' into the 'Optimize' process.

Remark 2: Control the complexity of the tree ensembles It is more common to control the complexity of base trees directly with structural constraints (i.e. max_depth, max_leaf_nodes, min_samples_leaf, etc) • Pre-pruning (Stopping) Learn trees/forests in a way that keeps the ensemble compact. RGF(Johnson & Zhang, 2014)ɺFeature-Budgeted Random Forest (Nan+ ICML2015)ɺGIF(Begon+ ICML2017)ɺBonsai(Kumar+ ICML2017)ɺADAPT-GBRT(Nan+ NIPS2017), etc. • Post-pruning (Pruning) After initially creating a set of trees, prune them to learn the appropriate trees/forests. ISLE (Friedman & Popescu 2003)ɺRuleFit (Friedman & Popescu 2008)ɺForest garrote (Meinshausen+ 2009)ɺL1-based compression (Joly+ 2012)ɺGlobal refinement (Ren+ CVPR 2015)ɺDART (Rashmi & Gilad-Bachrach, AISTATS 2015)ɺWavelet decomposition (Elisha & Dekel JMLR 2015)ɺBudgetPrune (Nan+ NIPS2016), etc. • Re-learning (Distillation) After initially creating a set of trees, relearn an equivalent but more compact trees/forests. CMM(Domingos, ICML1997)ɺNode Harvest (Meinshausen, 2010), etc. But there are also advanced methods for controlling the complexity of the entire forest

Outline We will jointly learn decision trees and ensemble learning! 1. Why we should learn decision trees and ensemble learning? 2. History of Decision Trees 3. Basics of Decision Trees 4. Ensemble Learning: Additive ensembles and decision forests 5. Random forest and Regularized gradient boosted trees 6. Practical data analysis with decision tree ensembles Part 1 Part 2

Practical data analysis using tree ensembles In many cases, prediction alone is not sufficient; analysis of the machine learning model is necessary. Visualization: Plotting the tree structure and accompanying information. Model Tuning: Adjusting hyperparameters for decision trees and decision forests. Model Interpretation: Variable Importance: Which input variables have a strong in uence on model predictions? Partial Dependence Plot (PDP): How does the prediction change when variable values are altered? SHapley Additive exPlanations (SHAP): Why was a particular prediction made? Uncertainty Quanti cation (UQ): Randomize type Law of total variance Optimize type Quantile Regression

dtreeviz: A visualization tool compatible with scikit-learn, XGBoost, and LightGBM. Visualization of Decision Trees https://github.com/parrt/dtreeviz While scikit-learn has built-in decision tree plotting capabilities using graphviz, dtreeviz o ers overwhelmingly better and more exible visualizations. Check out the Jupyter Notebook examples on GitHub for how to use dtreeviz. Parr & Grover, How to visualize decision trees https://explained.ai/decision-tree-viz/index.html https://github.com/parrt/dtreeviz/tree/master/notebooks Must-read: Detailed explanation on decision tree visualization by the authors of dtreeviz

Gallery (dtreevis) https://github.com/parrt/dtreeviz/blob/master/notebooks/examples.ipynb

Determine the number of trees (n_estimators) to create, say 100, 200, and 500 or so, and tune the other main hyperparameters through cross-validation. Model Tuning (hyperparameter tuning) • Decision Tree: max_leaf_nodes (or max_depth) + min_samples_leaf + Pruning by ccp_alpha • Decision Forest (Randomize-type): bootstrap + max_features • Decision Forest (Optimize-type): learning_rate (Adjust according to n_estimators) + with the Randomize type (subsample, colsample) + with regularization (1-norm regularization reg_alpha, 2-norm regularization reg_lambda) 1. Optuna 2. SMAC 3. Hyperopt 4. Scikit-Hyperband https://optuna.org/ https://github.com/automl/SMAC3 https://github.com/hyperopt/hyperopt https://github.com/thuijskens/scikit-hyperband Use Scikit's RandomSearchCV or GridSearchCV, as well as the following tuners for more systematic exploration. (Depending on the computation time and resources.)

Code example import optuna from lightgbm import LGBMRegressor from sklearn.metrics import mean_squared_error from sklearn.datasets import load_diabetes from sklearn.model_selection import train_test_split X, y = load_diabetes(return_X_y=True) X_train, X_val, y_train, y_val = train_test_split( X, y, test_size=0.33) def objective(trial): param = { "num_leaves": trial.suggest_int("num_leaves", 2, 256), "learning_rate": trial.suggest_float("learning_rate", 1e-5, 1.0, log=True), "min_child_samples": trial.suggest_int("min_child_samples", 1, 100), "subsample": trial.suggest_float("subsample", 0.4, 1.0), "subsample_freq": trial.suggest_int("subsample_freq", 1, 7), "colsample_bytree": trial.suggest_float("colsample_bytree", 0.4, 1.0), "reg_alpha": trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True), "reg_lambda": trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True), } model = LGBMRegressor(n_estimators=200, verbose=-1, **param) model.fit(X_train, y_train) y_pred = model.predict(X_val) return mean_squared_error(y_val, y_pred) optuna.logging.set_verbosity(optuna.logging.ERROR) study = optuna.create_study(direction="minimize") study.optimize(objective, n_trials=100) trial = study.best_trial best_params = trial.params model = LGBMRegressor(n_estimators=200, verbose=-1, **best_params) model.fit(X_train, y_train) y_pred = model.predict(X_val) mean_squared_error(y_val, y_pred) model = LGBMRegressor(n_estimators=200, verbose=-1) model.fit(X_train, y_train) y_pred = model.predict(X_val) mean_squared_error(y_val, y_pred) Compare

Model interpretation • The Mythos of Model Interpretability: The concept of "interpretability" is ambiguous and subjective, potentially dubious...? Three basic techniques for extracting information from decision tree ensembles. Variable importance Partial Dependence Plot (PDP) SHapley Additive exPlanations (SHAP) https://doi.org/10.1145/3236386.3241340 • Rashomon effect: The multiplicity of good ML models We can always have multiple different ML models having the same good prediction accuracy. • While decision trees are often said to have "high interpretability", they have issues with instability. The tree structures can change significantly due to random noise, making it largely unreliable. But obviously, decision tree ensembles would be not interpretable…

Variable Importance (Feature importance) is a convenient and user-friendly tool to understand which of the input variables used are important for predictions. Variable importances: MDI vs PFI • Feature Importance by Mean Decrease Impurity (MDI) Scoring how much each variable contributed to the prediction based on the internal information obtained during training from decision trees. • Permutation Feature Importance (PFI) Scoring a variable's importance by measuring the deterioration in accuracy when predicting with the variable versus predicting without it on any given data. In practice, it's preferable to use PFI over MDI. However, be aware that there are also drawbacks to consider. (masking by correlated features) Parr et al., Beware Default Random Forest Importances. https://explained.ai/rf-importance/

Nodes where variable is used for splitting Feature Importance 1: Mean Decrease Impurity (MDI) The weighted average of impurity improvement at nodes where variable is used. Since it's a relative scale normalized to a sum of 1 or 100, in actual calculations, you can simply add up the values of . AAACpnichVE9S8NQFD2NX7V+tOoiuBRLpVN5FUFxKro4ST/sB9RSk/haQ9MkJGmhFv+AroqDk4KD+DNc+gcc+hPEsYKLgzdpQLS03pC88869577zciVDVSybsZ5PmJicmp7xzwbm5hcWg6Gl5bylN02Z52Rd1c2iJFpcVTSesxVb5UXD5GJDUnlBqu87+UKLm5aia0d22+DlhljTlKoiizZRaa0SirA4cyM8DBIeiMCLlB7q4hin0CGjiQY4NNiEVYiw6CkhAQaDuDI6xJmEFDfPcYEAaZtUxalCJLZO3xrtSh6r0d7pablqmU5R6TVJGUaUvbIn1mdd9sze2NfIXh23h+OlTas00HKjErxczX7+q2rQauPsRzXWs40qdlyvCnk3XMa5hTzQt85v+9ndTLSzwR7YO/m/Zz32QjfQWh/yY5pn7sb4kcjL6D/m5L0KGmHi78CGQX4zniCc3ook97xh+rGGdcRoYttI4gAp5OgEjitc40aICYdCTigMSgWfp1nBrxBOvgHQN5dI AAACqXichVHLSsNAFD2Nr/ps1Y3gRiyKIshUBMVV0Y3LVq0tVpFknNqhaRKSaaEWf8CF2wquFFyIn+GmP+CinyAuFdy48CYNiIr1hmTOnHvPnTO5hmNKTzHWjmg9vX39A9HBoeGR0bFYfHxi37OrLhdZbpu2mzd0T5jSElkllSnyjiv0imGKnFHe8vO5mnA9aVt7qu6Io4p+asmi5LryKWtBLR7HE2yZBTHzGyRDkEAYaTvewiFOYIOjigoELCjCJnR49BSQBIND3BEaxLmEZJAXOMcQaatUJahCJ7ZM31PaFULWor3f0wvUnE4x6XVJOYM59sTu2StrsQf2zD7+7NUIevhe6rQaHa1wjmMXU7vv/6oqtCqUvlRdPSsUsR54leTdCRj/Fryjr501X3c3duYa8+yWvZD/G9Zmj3QDq/bG7zJi57qLH4O8/P3H/HxYQSNM/hzYb7C/spwknFlNpDbDYUYxjVks0MTWkMI20sjSCSVcookrbUnLaHntoFOqRULNJL6Fxj8BACiYKw== AAACpnichVE9S8NQFD2NX7V+tOoiuBRLpVN5FUFxKro4ST/sB9RSk/haQ9MkJGmhFv+AroqDk4KD+DNc+gcc+hPEsYKLgzdpQLS03pC88869577zciVDVSybsZ5PmJicmp7xzwbm5hcWg6Gl5bylN02Z52Rd1c2iJFpcVTSesxVb5UXD5GJDUnlBqu87+UKLm5aia0d22+DlhljTlKoiizZR6VYlFGFx5kZ4GCQ8EIEXKT3UxTFOoUNGEw1waLAJqxBh0VNCAgwGcWV0iDMJKW6e4wIB0japilOFSGydvjXalTxWo73T03LVMp2i0muSMowoe2VPrM+67Jm9sa+RvTpuD8dLm1ZpoOVGJXi5mv38V9Wg1cbZj2qsZxtV7LheFfJuuIxzC3mgb53f9rO7mWhngz2wd/J/z3rshW6gtT7kxzTP3I3xI5GX0X/MyXsVNMLE34ENg/xmPEE4vRVJ7nnD9GMN64jRxLaRxAFSyNEJHFe4xo0QEw6FnFAYlAo+T7OCXyGcfAPiR5dQ AAACpnichVE9S8NQFD3G7++qi+AiloqT3IqgOIkuTmLV2EItNYmvNTRfJK+FWvwDuioOTgoO4s9w6R9w8CeIo4KLgzdpQLSoNyTvvHPvue+8XN2zzEASPXUonV3dPb19/QODQ8Mjo4mx8b3ArfqGUA3Xcv2crgXCMh2hSlNaIuf5QrN1S2T1ynqYz9aEH5iusyvrnijYWtkxS6ahSaYysphI0jxFMd0O0jFIIo4tN9HEPg7hwkAVNgQcSMYWNAT85JEGwWOugAZzPiMzygucYIC1Va4SXKExW+FvmXf5mHV4H/YMIrXBp1j8+qycRooe6Y5eqUn39Ewfv/ZqRD1CL3Ve9ZZWeMXR08md939VNq8SR1+qPz1LlLAceTXZuxcx4S2Mlr52fPm6s7KdaszSDb2w/2t6oge+gVN7M24zYvvqDz86e/n9j4X5uIJHmP45sHawtzCfZpxZTK6uxcPswxRmMMcTW8IqNrAFlU8QOMM5LpQ5ZVNRlWyrVOmINRP4FsrBJ93Dl04= AAACpnichVE9S8NQFD3G7++qi+AiloqT3IqgOIkuTmLV2EItNYmvNTRfJK+FWvwDuioOTgoO4s9w6R9w8CeIo4KLgzdpQLSoNyTvvHPvue+8XN2zzEASPXUonV3dPb19/QODQ8Mjo4mx8b3ArfqGUA3Xcv2crgXCMh2hSlNaIuf5QrN1S2T1ynqYz9aEH5iusyvrnijYWtkxS6ahSaYysphI0jxFMd0O0jFIIo4tN9HEPg7hwkAVNgQcSMYWNAT85JEGwWOugAZzPiMzygucYIC1Va4SXKExW+FvmXf5mHV4H/YMIrXBp1j8+qycRooe6Y5eqUn39Ewfv/ZqRD1CL3Ve9ZZWeMXR08md939VNq8SR1+qPz1LlLAceTXZuxcx4S2Mlr52fPm6s7KdaszSDb2w/2t6oge+gVN7M24zYvvqDz86e/n9j4X5uIJHmP45sHawtzCfZpxZTK6uxcPswxRmMMcTW8IqNrAFlU8QOMM5LpQ5ZVNRlWyrVOmINRP4FsrBJ93Dl04= Impurity improvement at node # of all samples # of samples at node petal.width <= 0.8 gini = 0.67 samples = 150 petal.width <= 1.75 gini = 0.5 samples = 100 petal.length <= 4.95 gini = 0.17 samples = 54 petal.length <= 4.85 gini = 0.04 samples = 46 gini = 0.0 samples = 50 gini = 0.04 samples = 48 gini = 0.44 samples = 8 gini = 0.44 samples = 3 gini = 0.0 samples = 43 150×0.67– (50×0.0+100×0.5) 100×0.5- (54×0.17+46×0.04)

Feature Importance 2: Permutation Feature Importance (PFI) The decrease in prediction accuracy when the values of a specific variable are shuffled within the data. For importance features in predictions, the prediction accuracy should decrease when the feature values are randomly replaced with arbitrary values! →Features that cause a significant decline in prediction accuracy are considered important features. a trained model a trained model … … Accuracy A Accuracy B Feature importance of X3 = Accuracy A - Accuracy B

• PFI doesn't require retraining and can be calculated on either training or test data. However, careful interpretation is needed depending on which data the PFI was calculated on. • PFI involves averaging results over multiple shuffles due to the randomness of the shuffling, providing a more robust estimate. Advantages and Disadvantages of PFI Advantages • PFI assumes that the model has a high predictive accuracy. A poor model might undervalue an important feature that would have higher importance in a better model. • Masking Phenomenon: If there are multiple variables with high correlation, shuffling one variable might have its effects compensated for by the others. Consequently, using PFI can make the importance of all these variables appear lower. • A common way to handle this is to preprocess the data and represent highly correlated features with a single representative feature. Disadvantages

Example https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance.html

Partial Dependence Plot (PDP) https://scikit-learn.org/stable/auto_examples/ensemble/plot_partial_dependence.html Plot of empirical mean of change in output values when only the value of each variable is varied Example) Varying x2, while marginalizing the other variables (x1 and x3) at their empirical averages from the sample We can not only analyze the relative importance of each variable, but also see how they affect the output. AAACDnicbVDLSsNAFJ3UV62vqODGzdAiVISSFHwsi7pwWcE+oClhMp22Q2eSMDMRY8w/uPAL3Oranbj1F7r0T5w+Frb1wIXDOfdyD8cLGZXKsoZGZml5ZXUtu57b2Nza3jF39+oyiAQmNRywQDQ9JAmjPqkpqhhphoIg7jHS8AZXI79xT4SkgX+n4pC0Oer5tEsxUlpyzQOHI9UXPKleV1M3eXDLafHxGLpmwSpZY8BFYk9JoZJ3Tl6Glbjqmj9OJ8ARJ77CDEnZsq1QtRMkFMWMpDknkiREeIB6pKWpjziR7WScP4VHWunAbiD0+AqO1b8XCeJSxtzTm6O0ct4bif95rUh1L9oJ9cNIER9PHnUjBlUAR2XADhUEKxZrgrCgOivEfSQQVrqymS8eT3Un9nwDi6ReLtlnpdNbXc4lmCALDkEeFIENzkEF3IAqqAEMnsAreAPvxrPxYXwaX5PVjDG92QczML5/AaVHnz4= PDPx2 (z) AAACMXicbZDLSgMxFIYzXuu96tJNUIQKUmYUb4tC0YUuK1grdOqQSTNtaJIZkoxYxzyLe7c+g1tdd6dufQnTVsTbD4E//zmHk3xhwqjSrttzRkbHxicmc1PTM7Nz8wv5xaVzFacSkyqOWSwvQqQIo4JUNdWMXCSSIB4yUgs7R/167YpIRWNxprsJaXDUEjSiGGkbBfmDEvQjiXDmmUwY6KuUBxkteeZSwKjghzy7NjbwzCa82YRf922zEeTX3KI7EPxrvE+zVj6G935w26oE+Ve/GeOUE6ExQ0rVPTfRjQxJTTEjZtpPFUkQ7qAWqVsrECeqkQ2+aOC6TZowiqU9QsNB+n0iQ1ypLg9tJ0e6rX7X+uF/tXqqo/1GRkWSaiLwcFGUMqhj2OcFm1QSrFnXGoQltW+FuI0sMW2p/tgScmOZeL8J/DXnW0Vvt7hzauEcgqFyYAWsggLwwB4ogxNQAVWAwR14BE/g2Xlwes6L8zZsHXE+Z5bBDznvH2iurSI= = 1 n n X i=1 f(xi1, z, xi3) 2-variable PDP 1-variable PDPs

SHAP (SHapley Additive exPlanations) A model explanation method that decomposes the difference between a given prediction value and the dataset average into the "sum of contributions for each feature (SHAP values)”. (based on Cooperative game theory) a given prediction value the dataset average https://github.com/slundberg/shap

Example The dataset average is an annual income of 5 million yen, yet this individual was predicted to have an annual income of 10 million yen. 5 million 0 10 million dataset average Master's degree CS Specialty Position: Section Chief Don’t speak English +2 +2 +3 -2 Negative factors are also taken into account. Where does the difference of 5 million yen come from?

SHAP value The SHAP value of variable i for sample x = A weighted average of all possible differences Prediction when we use variable i Prediction when we don’t use variable i The total number of possible combinations AAAB93icbVDLSgNBEJyNrxhfUY9eBoPgKeyKr2NQEI8JmAckS5id9CZDZmaXmVlhWfIFXvXsTbz6OR79EyfJHkxiQUNR1U13VxBzpo3rfjuFtfWNza3idmlnd2//oHx41NJRoig0acQj1QmIBs4kNA0zHDqxAiICDu1gfD/128+gNIvkk0lj8AUZShYySoyVGg/9csWtujPgVeLlpIJy1Pvln94gookAaSgnWnc9NzZ+RpRhlMOk1Es0xISOyRC6lkoiQPvZ7NAJPrPKAIeRsiUNnql/JzIitE5FYDsFMSO97E3F/7xuYsJbP2MyTgxIOl8UJhybCE+/xgOmgBqeWkKoYvZWTEdEEWpsNgtbAjGxmXjLCayS1kXVu65eNS4rtbs8nSI6QafoHHnoBtXQI6qjJqII0At6RW9O6rw7H87nvLXg5DPHaAHO1y8/05Nq F AAAB93icbVDLSgNBEJyNrxhfUY9eBoPgKeyKr2PQi8cEzQOSJcxOepMhM7PLzKywLPkCr3r2Jl79HI/+iZNkDyaxoKGo6qa7K4g508Z1v53C2vrG5lZxu7Szu7d/UD48aukoURSaNOKR6gREA2cSmoYZDp1YAREBh3Ywvp/67WdQmkXyyaQx+IIMJQsZJcZKjcd+ueJW3RnwKvFyUkE56v3yT28Q0USANJQTrbueGxs/I8owymFS6iUaYkLHZAhdSyURoP1sdugEn1llgMNI2ZIGz9S/ExkRWqcisJ2CmJFe9qbif143MeGtnzEZJwYknS8KE45NhKdf4wFTQA1PLSFUMXsrpiOiCDU2m4UtgZjYTLzlBFZJ66LqXVevGpeV2l2eThGdoFN0jjx0g2roAdVRE1EE6AW9ojcndd6dD+dz3lpw8pljtADn6xdUSpN3 S The set of all features feature subset AAAB/nicbVDLSsNAFL2pr1pfVZdugkWom5KIr2XRjctK7QPaECbTSTt0ZhJmJmIJBb/Bra7diVt/xaV/4rTNwrYeuHA4517uvSeIGVXacb6t3Mrq2vpGfrOwtb2zu1fcP2iqKJGYNHDEItkOkCKMCtLQVDPSjiVBPGCkFQxvJ37rkUhFI/GgRzHxOOoLGlKMtJHaoV8vP/n1U79YcirOFPYycTNSggw1v/jT7UU44URozJBSHdeJtZciqSlmZFzoJorECA9Rn3QMFYgT5aXTe8f2iVF6dhhJU0LbU/XvRIq4UiMemE6O9EAtehPxP6+T6PDaS6mIE00Eni0KE2bryJ48b/eoJFizkSEIS2putfEASYS1iWhuS8DHJhN3MYFl0jyruJeVi/vzUvUmSycPR3AMZXDhCqpwBzVoAAYGL/AKb9az9W59WJ+z1pyVzRzCHKyvX+vRlf0= fS(xS) The prediction for feature values in set S that are marginalized over features that are not included in set S AAACa3icbVFdb9MwFHXCV1e+CnsDHiyqSdsDVYL4ekGaQJp4HCrdJtVR5Lg3rTXbCfYNonLzQ/fID+A3gNPlgW1cyfLROffcax0XtZIOk+Qiim/dvnP33mBneP/Bw0ePR0+enriqsQJmolKVPSu4AyUNzFCigrPaAteFgtPi/HOnn/4A62RlvuG6hkzzpZGlFBwDlY8cq1cyl/QjZa7RuZ92d+EA4Ts9ChhQS9M4yrxkbUtZabnwaetZIU2l/eZo0/rNdNNJCkqcD8v9n2EKE03dew7oK9qR04Mhs3K5wiwfjZNJsi16E6Q9GJO+jvPRL7aoRKPBoFDcuXma1Jh5blEKBe2QNQ5qLs75EuYBGq7BZX4bTkv3ArOgZWXDMUi37L8Oz7Vza12ETs1x5a5rHfk/bd5g+SHz0tQNghGXi8pGUaxolzRdSAsC1ToALqwMb6VixUN+GP7jypZCtyGT9HoCN8HJ60n6bvL265vx4ac+nQF5Tl6SfZKS9+SQfCHHZEYEuSB/okG0E/2Od+Nn8YvL1jjqPbvkSsV7fwFqF7wb i = X S✓F \{i} 1 |F | |S| ⇥ f(xS[{i} ) f(xS) ⇤ (in predictions with and without variable i)

treeSHAP: SHAP for trees is exact & practical! https://www.nature.com/articles/s42256-019-0138-9 https://arxiv.org/abs/1905.04610 For general ML models, exact computation is NP-Hard (infeasibly time-consuming) but for decision trees, it can be computed efficiently in polynomial time!

Visualizing SHAP Values (Local Explanation) Local Explanation (Decomposition of feature contributions for a specific prediction) Force Plot Waterfall Plot Local Bar Plot

Visualizing SHAP Values (Global Explanation) Beeswarm Plot Bar Plot Feature importance by SHAP values Dependence Plot Global Explanation (calculated by aggregating SHAP values for each data point) Partial dependency plot by SHAP values

Uncertainty Quantification (UQ) For the use for actual decision-making, not only the predicted values but also quantifying their reliability (uncertainty) are very important. A. The predicted stock price of Brand X for tomorrow is , yen (± . yen)! B. The predicted stock price of Brand X for tomorrow is , yen (± . yen)! Here we focus on how to get the prediction variances because useful quantities such as confidence intervals, prediction intervals and expected improvement are calculated from them 95% prediction interval (by normal approximation)

An Easy Solution: Use scikit-optimize (skopt) https://scikit-optimize.github.io/ We can easily obtain the prediction variances of tree ensembles by sklearn wrappers from the skopt library!

• A regression tree naturally possesses predictive variance → The "variance" of y in the training samples where x falls. • Since base trees in Randomize type are independent, the predictive variances from each regression tree can be combined to compute the predictive variance for the ensemble. Prediction variances for regression (Randomize type) Law of total variance: Total variance = Within-group variance + Between-group variance Hutter et al, Algorithm Runtime Prediction: Methods & Evaluation (2014) https://arxiv.org/abs/1211.0906

https://scikit-optimize.github.io/stable/_modules/skopt/learning/forest.html def _return_std(X, trees, predictions, min_variance): # This derives std(y | x) as described in 4.3.2 of arXiv:1211.0906 std = np.zeros(len(X)) for tree in trees: var_tree = tree.tree_.impurity[tree.apply(X)] # This rounding off is done in accordance with the # adjustment done in section 4.3.3 # of http://arxiv.org/pdf/1211.0906v2.pdf to account # for cases such as leaves with 1 sample in which there # is zero variance. var_tree[var_tree < min_variance] = min_variance mean_tree = tree.predict(X) std += var_tree + mean_tree ** 2 std /= len(trees) std -= predictions ** 2.0 std[std < 0.0] = 0.0 std = std ** 0.5 return std Hutter et al, Algorithm Runtime Prediction: Methods & Evaluation (2014) https://arxiv.org/abs/1211.0906 Prediction variances for regression (Randomize type)

• Since base trees in Optimize type are NOT independent, the predictive variances are calculated through quantile regression. For a normal distribution, the quantiles corresponding to the standard deviation are . (lower side) and . (upper side). Therefore, by regressing on these quantiles, we can obtain the standard deviation (or the variance). GradientBoostingRegressor(loss='quantile', alpha=α) Quantile Regression: Regression that targets a specific tail (q-quantile) of the data, rather than the center (mean) of the data. Prediction variances for regression (Optimize type)

• Use the Pinball loss as the loss function for optimization and perform quantile regression for τ=0.16, 0.5, 0.84 respectively. GradientBoostingRegressor (n_estimators=30, learning_rate=0.1) LGBMRegressor (n_estimators=30, learning_rate=0.1, max_leaf_nodes=8, min_child_samples=5) • The 0.5 quantile represents the mean, and the range from 0.16 to 0.84 can approximate one standard deviation. (assuming normal distribution) Prediction variances for regression (Optimize type) Use 1.96 σ for 95% PI

For your information: Toward Conformal Prediction https://github.com/scikit-learn-contrib/MAPIE regressor = ExtraTreesRegressor(max_leaf_nodes=32, bootstrap=True) MapieRegressor(regressor, method="plus", cv=-1) MapieRegressor(regressor, method="plus", cv=Subsample(n_resamplin gs=50)) MapieRegressor(regressor, method="plus", cv=-1) MapieRegressor(regress or, method="plus", cv=Subsample(n_resamp lings=50)) Jacknife+ 95% prediction intervals Jacknife+ after bootstrap 95% prediction intervals Jacknife+ 90% prediction intervals Jacknife+ after bootstrap 90% prediction intervals In recent years, Uncertainty Quantification (UQ) has been actively investigated as a method of Conformal Prediction (CP).

Summary We will jointly learn decision trees and ensemble learning! 1. Why we should learn decision trees and ensemble learning? 2. History of Decision Trees 3. Basics of Decision Trees 4. Ensemble Learning: Additive ensembles and decision forests 5. Random forest and Regularized gradient boosted trees 6. Practical data analysis with decision tree ensembles Part 1 Part 2

Outline We will jointly learn decision trees and ensemble learning! 1. Why we should learn decision trees and ensemble learning? 2. History of Decision Trees 3. Basics of Decision Trees 4. Ensemble Learning: Additive ensembles and decision forests 5. Random forest and Regularized gradient boosted trees 6. Practical data analysis with decision tree ensembles Part 1 Part 2