Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Modern Introduction to Decision Tree Ensembles

itakigawa
September 27, 2023

A Modern Introduction to Decision Tree Ensembles

Speaker: Ichi Takigawa (Kyoto University)

Video: Part 1 Part 2

Sep 11 – 22, 2023
Göttinger Rechenzentrum

Data Science Summer School 2023 @ Göttingen
https://events.gwdg.de/e/DS3-2023
https://www.eresearch.uni-goettingen.de/news-and-events/summer-schools/data-science-summer-school-2023/

HeKKSaGOn: German-Japanese University Consortium
Heidelberg – Kyoto – Karlsruhe – Sendai – Göttingen – Osaka
https://www.hekksagon.net/

itakigawa

September 27, 2023
Tweet

More Decks by itakigawa

Other Decks in Technology

Transcript

  1. https://itakigawa.github.io/
    A Modern Introduction to
    Decision Tree Ensembles
    Ichigaku Takigawa
    Institute for Liberal Arts and Sciences, Kyoto University
    Institute for Chemical Reaction Design and Discovery, Hokkaido University
    RIKEN Center for Advanced Intelligence Project
    Part 1

    View full-size slide

  2. About Me (Ichi Takigawa)
    • From Kyoto University, Japan
    • Research Interests:
    Machine Learning & Machine Discovery Algorithms
    • Machine Learning with Discrete Structures
    • Machine Discovery for Chemistry & Life Sciences
    • 34 years ago when I was a kid, I lived in Düsseldorf for 1 year
    https://itakigawa.github.io/

    View full-size slide

  3. Outline
    We will jointly learn decision trees and ensemble learning!
    1. Why we should learn decision trees and ensemble learning?
    2. History of Decision Trees
    3. Basics of Decision Trees
    4. Ensemble Learning: Additive ensembles and decision forests
    5. Random forest and Regularized gradient boosted trees
    6. Practical data analysis with decision tree ensembles
    Part 1
    Part 2

    View full-size slide

  4. Tree
    a forest = a set of trees
    Forest
    root
    leaf
    node
    pruning
    depth
    height
    Stump
    Quick note to complete beginners
    Technical Terms
    in Computer Science!

    View full-size slide

  5. Quick note to complete beginners
    We will get “decision tree ensembles” by integrating multiple
    “decision trees” in a certain way called “ensemble learning”.
    “ensemble learning”
    “decision trees”
    • CART
    • C4.5
    • Variations (M5, RETIS, OC1, MARS, GOSDT)
    “decision tree ensembles (decision forests)”
    • Random Forest
    • ExtraTrees (Extremely randomized trees)
    • Gradient boosted trees
    • XGBoost, LightGBM

    View full-size slide

  6. Outline
    We will jointly learn decision trees and ensemble learning!
    1. Why we should learn decision trees and ensemble learning?
    2. History of Decision Trees
    3. Basics of Decision Trees
    4. Ensemble Learning: Additive ensembles and decision forests
    5. Random forest and Regularized gradient boosted trees
    6. Practical data analysis with decision tree ensembles
    Part 1
    Part 2

    View full-size slide

  7. Trees as representations
    Understand by “classification” = Simple and intuitive idea for us!
    https://en.wikipedia.org/
    wiki/Tree_(data_structure)
    2
    7 5
    2 10 6
    5 11
    9
    4
    https://en.wikipedia.org/
    wiki/Tree_of_life_(Kabbalah)
    https://en.wikipedia.org/
    wiki/Tree_of_life_(biology)
    z 10
    z 1
    z 2
    z 3
    z 4
    z 5
    z 6
    z 7
    z 8
    z 9
    w 7
    w 8
    w 9
    w 10
    u5
    u 6
    u7
    u8
    t 2
    t 3
    m 1
    m 2
    m 3
    m 4
    m 5
    m 6
    m 7
    m 8
    m 9
    E10 F10
    m 10
    s 2
    i2
    i3
    k5
    k6
    k7
    k8
    l7
    l8
    a1
    a2
    a3
    a4
    a5
    a6
    a7
    a8
    a9
    f 6
    f 7
    f 8
    f 9
    a10 f 10
    A B C D E F G H I K L
    W.West lith. Hatton Garden
    I
    II
    III
    IV
    V
    VI
    VII
    VIII
    IX
    X
    XI
    XII
    XIII
    XIV
    a14 q14 p14 b14 f 14 o14 e 14 m 14 F 14 n14 r 14 w 14 y 14 v 14 z 14
    Tree structure (Data structure) Tree of Life, Phylogenetic trees, Family trees
    • Folders (File systems)
    • DOM (XML/HTML)
    • Class inheritance
    • Syntax tree
    • Search tree
    • Library classification
    • Probabilistic models
    :

    View full-size slide

  8. Why we should learn decision tree ensembles?
    The obsolete method, "decision trees,” which began its research around the
    1960s-70s and gained immense popularity in the 1980s-90s, has made a
    comeback in the modern era through "ensemble learning"!
    >

    ≤ >
    Classification
    trees
    Regression
    trees

    View full-size slide

  9. One of three major weapons in data science
    Decision tree ensemble is one of the three key weapons in modern data
    science alongside linear models and deep learning.
    https://www.kaggle.com/kaggle-survey-2021
    State of Data Science and Machine Learning 2021 (Kaggle Survey)
    Q17. Which of the following ML algorithms do you use on a regular basis?
    Linear or Logistic Regression
    Decision Trees or Random Forests
    Gradient Boosting Machines (xgboost, lightgbm, etc)
    Convolutional Neural Networks
    Dense Neural Networks (MLPs, etc)
    Bayesian Approaches
    Recurrent Neural Networks
    Transformer Networks (BERT, gpt-3, etc)
    None or Other
    Generative Adversarial Networks
    Evolutionary Approaches
    0 3,500 7,000 10,500 14,000
    13,852
    11,863
    7,566
    7,410
    4,468
    4,392
    4,228
    2,273
    1,953
    1,353
    963
    # respondents: 25,973
    Linear Models
    Decision Tree Ensembles
    Deep Learning

    View full-size slide

  10. Reason #1: Random Forest as post SVM
    Random forest showed better performance than SVM, the state of the art at
    that time, on various benchmarks particularly for high-dimensional data
    Do we need hundreds of classifiers to solve real world classification problems? (JMLR)

    https://jmlr.org/papers/v15/delgado14a.html
    An empirical evaluation of supervised learning in high dimensions (ICML 2008)

    https://doi.org/10.1145/1390156.1390169
    Report #1: 11 datasets × 10 methods (Caruana+, 2008)
    Report #2: 121 datasets × 179 methods (Fernández-Delgado+, 2014)
    # samples: 5.5K - 279.2K
    # features: 761 - 685K
    # samples: 16 - 130K
    # features: 3 - 262
    Very high-dimensional data
    Small-to-middle scale, but extremely diverse types of data

    View full-size slide

  11. Reason #2: Popularity in ML competitions
    In ML competitions like Kaggle, it has been widely recognized that
    decision tree ensembles work pretty well especially for tabular data.
    Well, we have too many ML algorithms, but what could be the “best method”?
    https://www.kaggle.com/
    • Kaggle is a data science competition platform and online community of data scientists
    and machine learning practitioners under Google LLC.
    • It’s also known for its substantial prize money for winners (from $10,000 to $1,000,000)
    All of the few masters who have ever been ranked No.1 in the world
    named the same ML method as their “most favorite”!

    View full-size slide

  12. Reason #2: Popularity in ML competitions
    Gradient Boosting Machines are the best!
    Before I knew of GBM, I was a big fan of neural networks.[2]
    It depends on the problem, but if I have to pick one,
    then it is GBM (its XGBoost flavor).[1]
    I like Gradient Boosting and Tree methods in general.[3]
    Gradient boosted trees by far! [4]
    [1] https://web.archive.org/web/20190925162927/http://blog.kaggle.com/2015/06/22/profiling-top-kagglers-owen-zhang-currently-1-in-the-world/
    [2] https://web.archive.org/web/20190914031256/http://blog.kaggle.com/2015/11/09/profiling-top-kagglers-gilberto-titericz-new-1-in-the-world/
    [3] https://web.archive.org/web/20191107132809/http://blog.kaggle.com/2016/02/10/profiling-top-kagglers-kazanova-new-1-in-the-world/
    [4] https://web.archive.org/web/20190913005952/http://blog.kaggle.com/2016/02/22/profiling-top-kagglers-leustagos-current-7-highest-1/
    That is “Gradient boosted (decision) trees” or GBDT!
    Aliases: Gradient boosting, GBM, MART, TreeNet, AnyBoost, …

    View full-size slide

  13. Reason #3: Research outcomes from Microsoft
    • Real-time Pose Recognition of Kinect (5ms per frame for Xbox360 GPU)
    • Decision Jungles (NIPS2013), LightGBM (NIPS 2017)
    • LambdaMART won Yahoo! Learning to Rank Challenge (ICML WS 2010)
    https://www.microsoft.com/en-us/research/project/
    human-pose-estimation-for-kinect/publications/
    http://research.microsoft.com/
    en-us/projects/decisionforests/
    Human Pose Estimation for Kinect Decision Forests
    https://doi.org/
    10.1007/978-1-4471-4929-3
    https://doi.org/10.1109/
    CVPR.2011.5995316

    View full-size slide

  14. Why we should learn decision tree ensembles?
    • What is a decision tree? (History and basic algorithm)
    • Why did it become so popular once in the past? (Advantages)
    Decision tree ensemble is one of the three key weapons in modern data
    science alongside linear models and deep learning.
    • Despite this, why did it fall out of favor? (Disadvantages)
    • What is ensemble learning? Why can it compensate for the shortcomings?
    • Practice and methods of decision tree ensembles
    [Part 1]
    [Part 2]

    View full-size slide

  15. A visual introduction to machine learning
    https://www.r2d3.us/visual-intro-to-machine-learning-part-1/

    View full-size slide

  16. A quick visual intro to decision trees
    Predict whether each property is in San Francisco or in New York.
    https://github.com/jadeyee/r2d3-part-1-data/blob/master/part_1_data.csv
    in_sf
    beds
    bath
    price
    year_built
    sqft
    price_per_sqft
    elevation
    Input
    variables
    (7)
    Output
    variables
    (1)
    Supervised
    Learning
    in_sf beds bath price year_built sqft price_per_sqft elevation
    1 1 1 865,000 1,993 960 901 17
    1 4 2.5 995,000 1,915 2,180 456 62
    0 0 1 319,000 1,941 500 638 10
    0 1 1 1,135,000 2,005 715 1,587 10
    0 2 1 545,000 1,939 1,049 520 39
    1 2 1.3 1,097,000 1,904 1,493 735 14
    0 0.5 1 384,900 1,962 540 713 10
    0 5 5 27,500,000 1,930 7,500 3,667 21
    1 3 2 1,650,000 1,922 2,025 815 106
    0: NY
    1: SF
    A tabular data
    i.e. Excel-like
    spreadsheet

    View full-size slide

  17. A quick visual intro to decision trees
    sf: 175
    ny: 224
    total: 399
    sf: 93
    ny: 0
    total: 93
    73
    >

    (224 + 93)/492
    = 0.6443
    Accuracy 64.43%
    sf: 268
    ny: 224
    total: 492
    SF would have a higher elevation. Let’s take the elevation variable.
    meters

    View full-size slide

  18. A quick visual intro to decision trees
    sf: 70
    ny: 187
    total: 257
    sf: 198
    ny: 37
    total: 246
    20
    >

    (187 + 198)/492
    = 0.7825
    Accuracy 78.25%
    sf: 268
    ny: 224
    total: 492
    SF would have a higher elevation. Let’s take the elevation variable.
    meters

    View full-size slide

  19. A quick visual intro to decision trees
    sf: 268
    ny: 224
    total: 492
    SF would have a higher elevation. Let’s take the elevation variable.
    meters

    View full-size slide

  20. A quick visual intro to decision trees
    sf: 85
    ny: 215
    total: 300
    sf: 183
    ny: 9
    total: 192
    30.5
    >

    (215 + 183)/492
    = 0.8089
    Accuracy 80.89%
    sf: 268
    ny: 224
    total: 492
    SF would have a higher elevation. Let’s take the elevation variable.
    meters

    View full-size slide

  21. A quick visual intro to decision trees
    ≤ >
    (215 + 183)/492
    = 0.8089
    Accuracy 80.89%
    183
    9
    215
    85
    Split all cases (492) into two groups (300 and 192 cases)!
    Here we have the simplest decision tree!
    (with a single split)
    “Decision Stump”

    View full-size slide

  22. A quick visual intro to decision trees
    in_sf beds bath price
    year_
    built
    sqft
    price_per
    _sqft
    eleva
    tion
    1 1 1 865,000 1,993 960 901 17
    1 4 2.5 995,000 1,915 2,180 456 62
    0 0 1 319,000 1,941 500 638 10
    0 1 1 1,135,000 2,005 715 1,587 10
    0 2 1 545,000 1,939 1,049 520 39
    1 2 1.3 1,097,000 1,904 1,493 735 14
    in_sf beds bath price
    year_
    built
    sqft
    price_per
    _sqft
    eleva
    tion
    1 1 1 865,000 1,993 960 901 17
    0 0 1 319,000 1,941 500 638 10
    0 1 1 1,135,000 2,005 715 1,587 10
    1 2 1.3 1,097,000 1,904 1,493 735 14
    0 0.5 1 384,900 1,962 540 713 10
    0 5 5 27,500,000 1,930 7,500 3,667 21
    in_sf beds bath price
    year_
    built
    sqft
    price_per
    _sqft
    eleva
    tion
    1 4 2.5 995,000 1,915 2,180 456 62
    0 2 1 545,000 1,939 1,049 520 39
    1 3 2 1,650,000 1,922 2,025 815 106
    1 1 1 699,000 1,907 932 750 59
    1 3 2 1,195,000 1,907 1,396 856 33
    1 4 3 1,895,000 2,001 2,041 928 61
    elevation > 30.5
    >

    True
    False
    492 cases
    300 cases 192 cases
    Variable selection
    Select a single variable
    Binary partitioning
    Split the data into two groups
    Split all cases (492) into two groups (300 and 192 cases)!

    View full-size slide

  23. A quick visual intro to decision trees
    in_sf beds bath price
    year_
    built
    sqft
    price_per
    _sqft
    eleva
    tion
    1 1 1 865,000 1,993 960 901 17
    0 0 1 319,000 1,941 500 638 10
    0 1 1 1,135,000 2,005 715 1,587 10
    1 2 1.3 1,097,000 1,904 1,493 735 14
    0 0.5 1 384,900 1,962 540 713 10
    0 5 5 27,500,000 1,930 7,500 3,667 21
    in_sf beds bath price
    year_
    built
    sqft
    price_per
    _sqft
    eleva
    tion
    1 4 2.5 995,000 1,915 2,180 456 62
    0 2 1 545,000 1,939 1,049 520 39
    1 3 2 1,650,000 1,922 2,025 815 106
    1 1 1 699,000 1,907 932 750 59
    1 3 2 1,195,000 1,907 1,396 856 33
    1 4 3 1,895,000 2,001 2,041 928 61
    300 cases 192 cases
    ≤ >
    Do the same thing again! (recursion)
    54
    63
    161
    22
    $

    View full-size slide

  24. A quick visual intro to decision trees
    ≤ >
    183
    9
    54
    63
    161
    22
    (63 + 161 + 183)/492
    = 0.8272
    Accuracy 82.72%
    A decision tree with 3 decision nodes!

    View full-size slide

  25. http://www.r2d3.us/visual-intro-
    to-machine-learning-part-1/
    A visual introduction to machine learning
    Repeat the recursion
    until we get the desired accuracy!
    A quick visual intro to decision trees

    View full-size slide

  26. A quick visual intro to decision trees
    ≤ >
    183
    9
    54
    63
    161
    22
    if elevation > 30.5
    return SF
    else
    if price_per_sqft > 1072.5
    return NY
    else
    return SF
    A record of any property
    Prediction SF or NY
    A decision tree = nested 'if' statements

    View full-size slide

  27. A quick visual intro to decision trees
    http://www.r2d3.us/visual-intro-
    to-machine-learning-part-1/
    A visual introduction to machine learning
    Just check the True/False
    conditions of the 'if' statements

    View full-size slide

  28. Outline
    We will jointly learn decision trees and ensemble learning!
    1. Why we should learn decision trees and ensemble learning?
    2. History of Decision Trees
    3. Basics of Decision Trees
    4. Ensemble Learning: Additive ensembles and decision forests
    5. Random forest and Regularized gradient boosted trees
    6. Practical data analysis with decision tree ensembles
    Part 1
    Part 2

    View full-size slide

  29. Brief history of decision trees
    1) Concept learning 2) Interaction detection 3) Nonlinear regression
    CLS (Hunt+ 1966)
    ID3 (Quinlan 1986)
    C4.5 (Quinlan 1993)
    C5.0/See5 (Quinlan)
    AID (Morgan & Sonquist 1963)
    Breiman & Meisel 1976
    Friedman 1979
    THAID (Messenger & Mandell, 1972)
    CHAID (Kass 1980)
    CART (Breiman+ 1984)
    M5 (Quinlan 1992)
    M5’ (Wang & Witten 1997)
    MARS (Friedman 1991)
    Cubist (Quinlan)
    RETIS (Karalic & Cestnik 1991)
    HTL (Torgo 1997)
    QUEST (Loh & Shih 1997)
    CRUISE (Kim & Loh, 2001)
    OC1 (Murthy+ 1993)
    Quinlan 1979
    Tree-shaped classification Tree-shaped regression
    Decision tree = A concept so simple that anyone could think of it
    In fact, this idea was explored in various contexts in the 1970s

    View full-size slide

  30. Learning of concepts and rules
    Decision trees are equivalent to If-then rules and have a close relationship
    with logic and logical circuits
    Decision Tree
    https://en.wikipedia.org/wiki/Decision_tree
    Decision Tables
    https://en.wikipedia.org/wiki/Decision_table
    Decision List
    https://en.wikipedia.org/wiki/Decision_list
    Decision Diagram
    https://en.wikipedia.org/wiki/Binary_decision_diagram
    if elevation > 30.5
    return SF
    else
    if price_per_sqft > 1072.5
    return NY
    else
    return SF
    A record of
    any property
    Prediction
    SF or NY

    View full-size slide

  31. Learning of concepts and rules
    In particular, the inductive reasoning problem called concept learning
    can be regarded as a noiseless version of today's supervised learning
    Swim Has ns Fly Has lungs Is a sh
    Herring yes yes no no ✔
    Cat no no no yes ✘
    Pigeon no no yes yes ✘
    Flying Fish yes yes yes no ✔
    Otter yes no no yes ✘
    Cod yes yes no no ✔
    Whale yes yes no yes ✘
    It assumes learning of logical expressions
    and hence variables are only boolean values
    By applying decision tree learning, one can obtain a consistent 'logical expression
    (If-then rule)' for the given examples!
    For example, NOT has lungs Is a sh
    Concept learning: Creating a description of
    the concept 'fish' from examples
    yes/no
    true/false

    View full-size slide

  32. Example: C5.0 for iris dataset
    The “Iris” dataset (3 classes, 4 variables, 150 samples)
    https://en.wikipedia.org/wiki/Iris_flower_data_set
    Target
    Variables Classification problem of 3 species
    setosa versicolor virginica
    Variables
    Petal
    Petal
    Sepal
    Sepal
    • Petal (Length, Width)
    • Sepal (Length, Width)
    The rules are not clear just
    by seeing many numbers…

    View full-size slide

  33. Example: C5.0 for iris dataset
    Let’s see what we get when we apply decision trees. (C5.0 in R)
    100 samples for training
    50 samples for test
    load the “iris” dataset
    inspect it
    C5.0 [Release 2.07 GPL Edition]
    Rules:
    Rule 1: (36, lift 2.7)
    Petal.Length <= 1.9
    -> class setosa [0.974]
    Rule 2: (34/3, lift 2.9)
    Petal.Length > 1.9
    Petal.Width <= 1.7
    -> class versicolor [0.889]
    Rule 3: (30, lift 2.9)
    Petal.Width > 1.7
    -> class virginica [0.969]

    View full-size slide

  34. Example: C5.0 for iris dataset
    Rules that humans can operate: they give a sense of “understanding”
    If petal is short, it s setosa
    If petal is wide, it s virginica
    If petal is long but narrow, it s versicolor
    ≤ 1.9 cm
    > 1.9 cm ≤ 1.7 cm
    > 1.7 cm
    https://www.kaggle.com/code/necibecan/iris-dataset-eda-n
    (Plus, we can ignore sepal)
    97.4%
    correct
    88.9%
    correct
    96.9%
    correct
    Rule 1: (36, lift 2.7)
    Petal.Length <= 1.9
    -> class setosa [0.974]
    Rule 2: (34/3, lift 2.9)
    Petal.Length > 1.9
    Petal.Width <= 1.7
    -> class versicolor [0.889]
    Rule 3: (30, lift 2.9)
    Petal.Width > 1.7
    -> class virginica [0.969]
    setosa
    versicolor virginica

    View full-size slide

  35. Example: C5.0 for iris dataset
    When applied to unseen data in training, “predictions” can be obtained
    Petal.Length
    1
    ≤ 1.9 > 1.9
    Node 2 (n = 36)
    setosa virginica
    0
    0.2
    0.4
    0.6
    0.8
    1
    Petal.Width
    3
    ≤ 1.7 > 1.7
    Node 4 (n = 34)
    setosa virginica
    0
    0.2
    0.4
    0.6
    0.8
    1
    Node 5 (n = 30)
    setosa virginica
    0
    0.2
    0.4
    0.6
    0.8
    1
    1 2 3 4 5 6
    0.5 1.0 1.5 2.0 2.5
    Petal.Length
    Petal.Width
    setosa
    versicolor
    virginica
    Decision tree:
    Petal.Length <= 1.9: setosa (36)
    Petal.Length > 1.9:
    :...Petal.Width > 1.7: versicolor (34/3)
    Petal.Width <= 1.7: virginica (30)

    View full-size slide

  36. Example: scikit-learn (Python)
    Text text
    !pip install -q dtreeviz
    !pip install -q svglib
    from sklearn import datasets
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import accuracy_score
    from sklearn.tree import DecisionTreeClassifier, export_text
    from dtreeviz.trees import dtreeviz
    iris = datasets.load_iris()
    X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.33)
    model = DecisionTreeClassifier(ccp_alpha=0.2)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print(100*accuracy_score(y_test, y_pred), '%')
    98 %
    model.classes_ = iris.target_names
    print(export_text(model, feature_names=iris.feature_names))
    |--- petal length (cm) <= 2.45
    | |--- class: setosa
    |--- petal length (cm) > 2.45
    | |--- petal width (cm) <= 1.75
    | | |--- class: versicolor
    | |--- petal width (cm) > 1.75
    | | |--- class: virginica
    model.classes_ = range(3)
    viz = dtreeviz(model, X_train, y_train,
    class_names=list(iris.target_names),
    feature_names=list(iris.feature_names),
    scale=2.0)
    display(viz)
    from svglib.svglib import svg2rlg
    from reportlab.graphics import renderPDF
    viz.save("treeout.svg")
    renderPDF.drawToFile(svg2rlg("treeout.svg"), "treeout.pdf")
    >

    View full-size slide

  37. Learning of concepts and rules
    AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGZiRVNvh8rijIXqiIrkThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOslho=
    X2
     ✓1
    yes no
    AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGRY0WT78eKosyFqswokThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOelho=
    X1
     ✓2
    yes no
    Blue
    AAAClnichVHLSsNAFD2N7/qquim4KZaKqzItRcWFiCK6rNZqoZWQxGkbTJOYTCta/AF/wIW4UFARP8APcOMPuOgniMsKblx4mwZExXrDZM6cuefOmbmqbeiuYKwRkLq6e3r7+geCg0PDI6OhsfFt16o6Gs9qlmE5OVVxuaGbPCt0YfCc7XClohp8R91fae3v1Ljj6pa5JY5svltRSqZe1DVFECWHxnJyMlIw+EGkIMpcKHJKDkVZnHkR+Q0SPojCj7QVekABe7CgoYoKOEwIwgYUuPTlkQCDTdwu6sQ5hHRvn+MEQdJWKYtThkLsPv1LtMr7rEnrVk3XU2t0ikHDIWUEMfbM7liTPbF79sI+/qxV92q0vBzRrLa13JZHT8OZ939VFZoFyl+qjp4Fipj3vOrk3faY1i20tr52fNbMLGzG6tPsir2S/0vWYI90A7P2pl1v8M3zDn5U8kIvRg1K/GzHb7CdjCdm46mNVHRp2W9VPyYxhRnqxxyWsI40slT/EBe4wa0UlhalVWmtnSoFfM0EvoWU/gQ6DJYd
    X2
     ✓4
    yes no
    Red AAAClnichVHLSsNAFD3GV62vVjeCm2JRXJWJ+MKFFEV0WVurBSshiVMNTZOYTCta/AF/wIW4UFARP8APcOMPuOgniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGGu0SK1t7R2dka5od09vX38sPrDu2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvaOVFf3+jxl3PsK01ceDwrYq6YxklQ1cFUUosXlDkRNHke4mi2OVCVaaUWJKlWBCJn0AOQRJhZOzYHYrYhg0dVVTAYUEQNqHCo28TMhgc4rZQJ84lZAT7HEeIkrZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHaX/eCj39q+qQrPA7qfqT88CJcwGXg3y7gSMfwu9qa8dnrzk5rKj9TF2wZ7J/zlrsHu6gVV71S9Xefb0Dz8aeaEXowbJ39vxE6xPpOTp1OTqZDK9ELYqgmGMYJz6MYM0VpBBnurv4wxXuJaGpHlpSVpupkotoWYQX0LKfAA5/pYd
    X1
     ✓5
    yes no
    Red AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4=
    X1
     ✓6
    yes no
    Red
    Blue
    AAAClnichVHLSsNAFD3GV3221Y3gplgUV2WiRcWFFEV0Wa3VgpWQxKmGpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxKWCGxfepAFRUW+YzJkz99w5M1dzTMMTjDVapNa29o7OSFd3T29ffzQWH1j37Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tPKCv79R465n2NaaOHD4VkXdsYySoauCKCUWLyhyomjyvURR7HKhKpNKLMlSLIjETyCHIIkwsnbsDkVsw4aOKirgsCAIm1Dh0bcJGQwOcVuoE+cSMoJ9jiN0k7ZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHSV6PJR7+1dVoVlg91P1p2eBEmYCrwZ5dwLGv4Xe1NcOT15ys6uj9TF2wZ7J/zlrsHu6gVV71S9X+OrpH3408kIvRg2Sv7fjJ1ifSMlTqfRKOpmZD1sVwTBGME79mEYGy8giT/X3cYYrXEtD0py0KC01U6WWUDOILyFlPwA1vpYb
    X1
     ✓3
    yes no
    Red
    AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykWHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm4/JcPJFJxFJLQav6MIFJzFA/kkhhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bz4+lh8=
    X1
     ✓7
    yes no
    Blue Blue
    if x2 ≤ θ1 then
    if x1 ≤ θ2 then
    return Blue
    else
    if x2 ≤ θ4 then
    return Red
    else
    if x1 ≤ θ5 then
    return Red
    else
    if x1 ≤ θ6 then
    return Blue
    else
    return Red
    else
    if x1 ≤ θ3 then
    if x2 ≤ θ7 then
    return Blue
    else
    return Blue
    else
    return Red
    Decision tree Logical rules

    View full-size slide

  38. For your information
    Knowledge representation as logical expressions was at the heart of
    symbolic and knowledge-based machine learning in the 90s
    • Programming Language Research
    • Logical Programming
    • Constraint Programming
    • Inductive Logic Programming (ILP)
    • Explanation-based Learning
    • Predicate Logic Learning
    • Propositional Logic Learning
    • Analogy
    • Inductive Reasoning
    • Grammar Inference (Inductive Language Learning)
    • Computational Learning
    • Gold's Limit Identification ('67)
    • Plotkin's Least General Generalization ('70)
    • Angluin's Inductive Inference ('83)
    • Angluin's Query Learning ('88)
    • Shapiro's Model Inference ('81)
    • Valiant's PAC Learning ('84)
    → Decision tree learning was a research topic that intersected with knowledge
    representation and logical reasoning. (a typical GOFAI = Good Old-Fashioned AI)

    View full-size slide

  39. Brief history of decision trees
    1) Concept learning 2) Interaction detection 3) Nonlinear regression
    CLS (Hunt+ 1966)
    ID3 (Quinlan 1986)
    C4.5 (Quinlan 1993)
    C5.0/See5 (Quinlan)
    AID (Morgan & Sonquist 1963)
    Breiman & Meisel 1976
    Friedman 1979
    THAID (Messenger & Mandell, 1972)
    CHAID (Kass 1980)
    CART (Breiman+ 1984)
    M5 (Quinlan 1992)
    M5’ (Wang & Witten 1997)
    MARS (Friedman 1991)
    Cubist (Quinlan)
    RETIS (Karalic & Cestnik 1991)
    HTL (Torgo 1997)
    QUEST (Loh & Shih 1997)
    CRUISE (Kim & Loh, 2001)
    OC1 (Murthy+ 1993)
    Quinlan 1979
    Tree-shaped classification Tree-shaped regression
    Decision tree = A concept so simple that anyone could think of it
    In fact, this idea was explored in various contexts in the 1970s

    View full-size slide

  40. Interaction detection in multivariate analysis
    An early idea toward the decision tree: AID = Automatic Interaction Detection
    Sonquist & Morgan, 1964
    Morgan & Sonquist, 1963
    https://doi.org/10.2307/2283276 https://www.amazon.com/dp/B0006BMGNW

    View full-size slide

  41. Interaction detection in multivariate analysis
    Sonquist & Morgan, 1964
    Morgan & Sonquist, 1963
    https://doi.org/10.2307/2283276 https://www.amazon.com/dp/B0006BMGNW
    THAID (Messenger & Mandell, 1972) CHAID (Kass, 1975, 1980)
    https://doi.org/10.1080/01621459.1972.10481290 https://doi.org/10.2307/2346565
    An early idea toward the decision tree: AID = Automatic Interaction Detection

    View full-size slide

  42. Interaction detection in multivariate analysis
    Independence of variables in a linear model ("independent" variables)
    AAACA3icbVC7SgNBFL0bXzG+opY2g0GITdiVoDZC0MYygnlAsoTZyWwyZGZ3nZkV1iWl32CrtZ3Y+iGW/omTZAuTeODC4Zx7OZfjRZwpbdvfVm5ldW19I79Z2Nre2d0r7h80VRhLQhsk5KFse1hRzgLa0Exz2o4kxcLjtOWNbiZ+65FKxcLgXicRdQUeBMxnBGsjuQm6Ql3FBgKXn057xZJdsadAy8TJSAky1HvFn24/JLGggSYcK9Vx7Ei7KZaaEU7HhW6saITJCA9ox9AAC6rcdPr0GJ0YpY/8UJoJNJqqfy9SLJRKhGc2BdZDtehNxP+8Tqz9SzdlQRRrGpBZkB9zpEM0aQD1maRE88QQTCQzvyIyxBITbXqaS/HE2HTiLDawTJpnFee8Ur2rlmrXWTt5OIJjKIMDF1CDW6hDAwg8wAu8wpv1bL1bH9bnbDVnZTeHMAfr6xc6xpfJ
    y = (z)
    AAACxniclVFdS8MwFE3r15xfUx99CQ5BUEbbdbo9CKIvPk5wm7iVkmbZDKZpSVJ1loK/0Hcf/Sem2wTnfNmBwOHcc3Iv9wYxo1JZ1qdhLi2vrK4V1osbm1vbO6XdvbaMEoFJC0csEvcBkoRRTlqKKkbuY0FQGDDSCZ6u83rnmQhJI36nRjHxQjTkdEAxUlrySx9v8AKmvfFHXTEMvNSqWGOcWhWnUa/VXU3Oaw237mbwxbcyeFJcwG9n8NW3F8s4ecZZLFPNM1Xol8o/PjhP7Ckpgymafumr149wEhKuMENSdm0rVl6KhKKYkazYSySJEX5CQ9LVlKOQSC8dj5XBI6304SAS+nEFx+rvRIpCKUdhoJ0hUo/yby0X/6t1EzWoeynlcaIIx5NGg4RBFcH8iLBPBcGKjTRBWFA9K8SPSCCs9KlnugRhpndi/93APGk7Ffus4t665cur6XYK4AAcgmNgg3NwCW5AE7QANlzjwcBG37wxuZmYLxOraUwz+2AG5vs3e/jKcg==
    z = w0 + w1x1 + w2x2 + w3x3
    AAAB+XicdVDLSgMxFM3UV62vqks3wSK4GmbK1La7ohuXFe0D2qFk0kwbmmSGJKOUoZ/gVtfuxK1f49I/MW1HsKIHLhzOuZd77wliRpV2nA8rt7a+sbmV3y7s7O7tHxQPj9oqSiQmLRyxSHYDpAijgrQ01Yx0Y0kQDxjpBJOrud+5J1LRSNzpaUx8jkaChhQjbaTbh4EzKJYc21kAOna5XqvUPEOqlbpniJtZJZChOSh+9ocRTjgRGjOkVM91Yu2nSGqKGZkV+okiMcITNCI9QwXiRPnp4tQZPDPKEIaRNCU0XKg/J1LElZrywHRypMfqtzcX//J6iQ5rfkpFnGgi8HJRmDCoIzj/Gw6pJFizqSEIS2puhXiMJMLapLOyJeAzk8n34/B/0i7b7oXt3XilxmWWTh6cgFNwDlxQBQ1wDZqgBTAYgUfwBJ6t1HqxXq23ZWvOymaOwQqs9y8luJSG
    w0
    AAAB+XicdVDLSgMxFM3UV62vqks3wSK4GmbK1La7ohuXFe0D2qFk0kwbmmSGJKOUoZ/gVtfuxK1f49I/MW1HsKIHLhzOuZd77wliRpV2nA8rt7a+sbmV3y7s7O7tHxQPj9oqSiQmLRyxSHYDpAijgrQ01Yx0Y0kQDxjpBJOrud+5J1LRSNzpaUx8jkaChhQjbaTbh4E7KJYc21kAOna5XqvUPEOqlbpniJtZJZChOSh+9ocRTjgRGjOkVM91Yu2nSGqKGZkV+okiMcITNCI9QwXiRPnp4tQZPDPKEIaRNCU0XKg/J1LElZrywHRypMfqtzcX//J6iQ5rfkpFnGgi8HJRmDCoIzj/Gw6pJFizqSEIS2puhXiMJMLapLOyJeAzk8n34/B/0i7b7oXt3XilxmWWTh6cgFNwDlxQBQ1wDZqgBTAYgUfwBJ6t1HqxXq23ZWvOymaOwQqs9y8nS5SH
    w1
    AAAB+XicdVDLSgMxFM3UV62vqks3wSK4GmbK1La7ohuXFe0D2qFk0kwbmmSGJKOUoZ/gVtfuxK1f49I/MW1HsKIHLhzOuZd77wliRpV2nA8rt7a+sbmV3y7s7O7tHxQPj9oqSiQmLRyxSHYDpAijgrQ01Yx0Y0kQDxjpBJOrud+5J1LRSNzpaUx8jkaChhQjbaTbh0F5UCw5trMAdOxyvVapeYZUK3XPEDezSiBDc1D87A8jnHAiNGZIqZ7rxNpPkdQUMzIr9BNFYoQnaER6hgrEifLTxakzeGaUIQwjaUpouFB/TqSIKzXlgenkSI/Vb28u/uX1Eh3W/JSKONFE4OWiMGFQR3D+NxxSSbBmU0MQltTcCvEYSYS1SWdlS8BnJpPvx+H/pF223Qvbu/FKjcssnTw4AafgHLigChrgGjRBC2AwAo/gCTxbqfVivVpvy9aclc0cgxVY718o3pSI
    w2
    AAAB+XicdVDLSgMxFM34rPVVdekmWARXw0yd2nZXdOOyon1AO5RMmmlDk8yQZJQy9BPc6tqduPVrXPonpu0IVvTAhcM593LvPUHMqNKO82GtrK6tb2zmtvLbO7t7+4WDw5aKEolJE0cskp0AKcKoIE1NNSOdWBLEA0bawfhq5rfviVQ0End6EhOfo6GgIcVIG+n2oX/eLxQd25kDOnapVi1XPUMq5ZpniJtZRZCh0S989gYRTjgRGjOkVNd1Yu2nSGqKGZnme4kiMcJjNCRdQwXiRPnp/NQpPDXKAIaRNCU0nKs/J1LElZrwwHRypEfqtzcT//K6iQ6rfkpFnGgi8GJRmDCoIzj7Gw6oJFiziSEIS2puhXiEJMLapLO0JeBTk8n34/B/0irZ7oXt3XjF+mWWTg4cgxNwBlxQAXVwDRqgCTAYgkfwBJ6t1HqxXq23ReuKlc0cgSVY718qcZSJ
    w3
    AAACA3icbVC7SgNBFJ2NrxhfUUubwSDEJuxKUMugjWUE84DsEmZnZ5Mh81hnZoWwpPQbbLW2E1s/xNI/cZJsYRIPXDiccy/ncsKEUW1c99sprK1vbG4Vt0s7u3v7B+XDo7aWqcKkhSWTqhsiTRgVpGWoYaSbKIJ4yEgnHN1O/c4TUZpK8WDGCQk4GggaU4yMlQJf0wFHVR9H0pz3yxW35s4AV4mXkwrI0eyXf/xI4pQTYTBDWvc8NzFBhpShmJFJyU81SRAeoQHpWSoQJzrIZk9P4JlVIhhLZUcYOFP/XmSIaz3mod3kyAz1sjcV//N6qYmvg4yKJDVE4HlQnDJoJJw2ACOqCDZsbAnCitpfIR4ihbCxPS2khHxiO/GWG1gl7Yuad1mr39crjZu8nSI4AaegCjxwBRrgDjRBC2DwCF7AK3hznp1358P5nK8WnPzmGCzA+foFLBmYXw==
    (·)
    When increases by ,
    increases by
    AAAB+XicbVA9TwJBEJ3DL8Qv1NJmIzGxIneEoCXRxhKjfCRwIXvLHmzY3bvs7hnJhZ9gq7WdsfXXWPpPXOAKAV8yyct7M5mZF8ScaeO6305uY3Nreye/W9jbPzg8Kh6ftHSUKEKbJOKR6gRYU84kbRpmOO3EimIRcNoOxrczv/1ElWaRfDSTmPoCDyULGcHGSg/P/Uq/WHLL7hxonXgZKUGGRr/40xtEJBFUGsKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LZVYUO2n81On6MIqAxRGypY0aK7+nUix0HoiAtspsBnpVW8m/ud1ExNe+ymTcWKoJItFYcKRidDsbzRgihLDJ5Zgopi9FZERVpgYm87SlkBMbSbeagLrpFUpe7Vy9b5aqt9k6eThDM7hEjy4gjrcQQOaQGAIL/AKb07qvDsfzueiNedkM6ewBOfrF73AlEA=
    x2
    AAAB93icbVA9SwNBEJ2LXzF+RS1tFoNgFe4kqGXQxjIB8wHJEfY2e8mS3b1jd084j/wCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzP/M4TVZpF8tGkMfUFHkkWMoKNlZrPg3LFrbpzoHXi5aQCORqD8k9/GJFEUGkIx1r3PDc2foaVYYTTaamfaBpjMsEj2rNUYkG1n80PnaILqwxRGClb0qC5+nciw0LrVAS2U2Az1qveTPzP6yUmvPUzJuPEUEkWi8KEIxOh2ddoyBQlhqeWYKKYvRWRMVaYGJvN0pZATG0m3moC66R9VfWuq7VmrVK/y9MpwhmcwyV4cAN1eIAGtIAAhRd4hTcndd6dD+dz0Vpw8plTWILz9QuRXZOd
    z AAAB+XicdVDLSgMxFM3UV62vqks3wSK4GmbK1La7ohuXFe0D2qFk0kwbmmSGJKOUoZ/gVtfuxK1f49I/MW1HsKIHLhzOuZd77wliRpV2nA8rt7a+sbmV3y7s7O7tHxQPj9oqSiQmLRyxSHYDpAijgrQ01Yx0Y0kQDxjpBJOrud+5J1LRSNzpaUx8jkaChhQjbaTbh0F5UCw5trMAdOxyvVapeYZUK3XPEDezSiBDc1D87A8jnHAiNGZIqZ7rxNpPkdQUMzIr9BNFYoQnaER6hgrEifLTxakzeGaUIQwjaUpouFB/TqSIKzXlgenkSI/Vb28u/uX1Eh3W/JSKONFE4OWiMGFQR3D+NxxSSbBmU0MQltTcCvEYSYS1SWdlS8BnJpPvx+H/pF223Qvbu/FKjcssnTw4AafgHLigChrgGjRBC2AwAo/gCTxbqfVivVpvy9aclc0cgxVY718o3pSI
    w2
    Interdependency between input variables are not considered
    “Interaction effect”
    Logistic Regression/Softmax Regression (Cox, 1958) ≈ Perceptron (Rosenblatt, 1959)"
    It has an effect independent of the values of other variables
    (linearity is a naive but very strong assumption)
    with a softmax activation and
    cross entropy loss
    AAACA3icbVC7SgNBFL0bXzG+opY2g0GITdiVoDZC0MYygnlAsoTZyWwyZGZ3nZkV1iWl32CrtZ3Y+iGW/omTZAuTeODC4Zx7OZfjRZwpbdvfVm5ldW19I79Z2Nre2d0r7h80VRhLQhsk5KFse1hRzgLa0Exz2o4kxcLjtOWNbiZ+65FKxcLgXicRdQUeBMxnBGsjuQm6Ql3FBgKXn057xZJdsadAy8TJSAky1HvFn24/JLGggSYcK9Vx7Ei7KZaaEU7HhW6saITJCA9ox9AAC6rcdPr0GJ0YpY/8UJoJNJqqfy9SLJRKhGc2BdZDtehNxP+8Tqz9SzdlQRRrGpBZkB9zpEM0aQD1maRE88QQTCQzvyIyxBITbXqaS/HE2HTiLDawTJpnFee8Ur2rlmrXWTt5OIJjKIMDF1CDW6hDAwg8wAu8wpv1bL1bH9bnbDVnZTeHMAfr6xc6xpfJ
    y = (z)

    View full-size slide

  43. Interaction detection in multivariate analysis
    Determination of the presence or absence of interaction
    Traditionally, a 'linear model + interaction term' is fitted and its significance is tested
    AAACqniclVHbSsNAEN3EW623qi+CL4tFEFpKkkZbX0T0QR9V7EXSEDbbbbu4ubC7UUvIT/k3PvonbmMFbX3pwMDhzJw5w4wfMyqkYXxo+tLyyupaYb24sbm1vVPa3WuLKOGYtHDEIt71kSCMhqQlqWSkG3OCAp+Rjv98Pal3XggXNAof5TgmboCGIR1QjKSivNJ7Baa9fIzDh76bGjUjj6pRs86bp01bgcbpud20M/jqpaaVZfDNM1VasFJcRGrVc6mlsr6gtG7m0nruXPRK5Z9+OA/MKSiDadx5pc9eP8JJQEKJGRLCMY1YuinikmJGsmIvESRG+BkNiaNgiAIi3DRfL4PHiunDQcRVhhLm7G9FigIhxoGvOgMkR2K2NiH/qzmJHDTdlIZxIkmIv40GCYMygpNHwT7lBEs2VgBhTtWuEI8QR1iqd/5x8YNM3cScvcA8aFs186xm39vly6vpdQrgEByBE2CCBrgEt+AOtADWDrQL7Ua71av6g/6kO9+tujbV7IM/ofe/AGC6wr0=
    +w12x1x2 + w23x2x3 + w31x3x1
    AAACxniclVFdS8MwFE3r15xfUx99CQ5BUEbbdbo9CKIvPk5wm7iVkmbZDKZpSVJ1loK/0Hcf/Sem2wTnfNmBwOHcc3Iv9wYxo1JZ1qdhLi2vrK4V1osbm1vbO6XdvbaMEoFJC0csEvcBkoRRTlqKKkbuY0FQGDDSCZ6u83rnmQhJI36nRjHxQjTkdEAxUlrySx9v8AKmvfFHXTEMvNSqWGOcWhWnUa/VXU3Oaw237mbwxbcyeFJcwG9n8NW3F8s4ecZZLFPNM1Xol8o/PjhP7Ckpgymafumr149wEhKuMENSdm0rVl6KhKKYkazYSySJEX5CQ9LVlKOQSC8dj5XBI6304SAS+nEFx+rvRIpCKUdhoJ0hUo/yby0X/6t1EzWoeynlcaIIx5NGg4RBFcH8iLBPBcGKjTRBWFA9K8SPSCCs9KlnugRhpndi/93APGk7Ffus4t665cur6XYK4AAcgmNgg3NwCW5AE7QANlzjwcBG37wxuZmYLxOraUwz+2AG5vs3e/jKcg==
    z = w0 + w1x1 + w2x2 + w3x3
    AAACOnicbZDLSgMxFIYz9VbrbdSlm2ARWpAy0xZ1WerGZQV7gXYYMmnahiYzQ5KRlqHv40P4DG7Vpe7ErQ9gphewrQdy+POdczjJ74WMSmVZ70ZqY3Nreye9m9nbPzg8Mo9PGjKIBCZ1HLBAtDwkCaM+qSuqGGmFgiDuMdL0hrdJvflIhKSB/6DGIXE46vu0RzFSGrlmNTdy7Us4cotJKuVhh6NQqgAu8yTZi1txQUoJzbtm1ipY04Drwp6LLJhHzTU/Ot0AR5z4CjMkZdu2QuXESCiKGZlkOpEkIcJD1CdtLX3EiXTi6V8n8EKTLuwFQh9fwSn9OxEjLuWYe7qTIzWQq7UE/ldrR6p348TUDyNFfDxb1IsY1F4kxsEuFQQrNtYCYUH1WyEeIIGw0vYubfH4RHtirzqwLhrFgn1VKN+Xs5Xq3J00OAPnIAdscA0q4A7UQB1g8ARewCt4M56NT+PL+J61poz5zClYCuPnF5txqJA=
    (x1, x2, x3) 7! (x1, x2, x3, x1x2, x2x3, x3x1)
    3 variables 6 variables
    “The most naive feature engineering”
    • Statistically compare the likelihood of models with and without interaction terms.
    • Since it is linear with respect to the parameters, the calculation for fitting is the same as linear.
    When increases by ,
    increases by
    AAAB+XicbVA9TwJBEJ3DL8Qv1NJmIzGxIneEoCXRxhKjfCRwIXvLHmzY3bvs7hnJhZ9gq7WdsfXXWPpPXOAKAV8yyct7M5mZF8ScaeO6305uY3Nreye/W9jbPzg8Kh6ftHSUKEKbJOKR6gRYU84kbRpmOO3EimIRcNoOxrczv/1ElWaRfDSTmPoCDyULGcHGSg/P/Uq/WHLL7hxonXgZKUGGRr/40xtEJBFUGsKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LZVYUO2n81On6MIqAxRGypY0aK7+nUix0HoiAtspsBnpVW8m/ud1ExNe+ymTcWKoJItFYcKRidDsbzRgihLDJ5Zgopi9FZERVpgYm87SlkBMbSbeagLrpFUpe7Vy9b5aqt9k6eThDM7hEjy4gjrcQQOaQGAIL/AKb07qvDsfzueiNedkM6ewBOfrF73AlEA=
    x2
    AAAB93icbVA9SwNBEJ2LXzF+RS1tFoNgFe4kqGXQxjIB8wHJEfY2e8mS3b1jd084j/wCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzP/M4TVZpF8tGkMfUFHkkWMoKNlZrPg3LFrbpzoHXi5aQCORqD8k9/GJFEUGkIx1r3PDc2foaVYYTTaamfaBpjMsEj2rNUYkG1n80PnaILqwxRGClb0qC5+nciw0LrVAS2U2Az1qveTPzP6yUmvPUzJuPEUEkWi8KEIxOh2ddoyBQlhqeWYKKYvRWRMVaYGJvN0pZATG0m3moC66R9VfWuq7VmrVK/y9MpwhmcwyV4cAN1eIAGtIAAhRd4hTcndd6dD+dz0Vpw8plTWILz9QuRXZOd
    z AAACXniclZHLSgMxFIbTUWut1o66EdwEiyAoZWaqtd0V3bisYC/QDkMmzbShmQtJRi3DvKEv4E5fwa0uTC8L27rxQODjP39yDn/ciFEhDeMto21sbmW3czv53b3CflE/OGyLMOaYtHDIQt51kSCMBqQlqWSkG3GCfJeRjju+m/Y7T4QLGgaPchIR20fDgHoUI6kkR/eS/uyRHh+6dmKUDVXV6uUUzJphKqjXa5ZVT+GzY6XwAv7Dn1iVNIUvTgVCRy/NvYYB18FcQAksqunoH/1BiGOfBBIzJETPNCJpJ4hLihlJ8/1YkAjhMRqSnsIA+UTYyWy1FJ4pZQC9kKsTSDhTf99IkC/ExHeV00dyJFZ7U/GvXi+WXs1OaBDFkgR4PsiLGZQhnIYLB5QTLNlEAcKcql0hHiGOsFRfsDTF9VOVibmawDq0rbJZLV8/XJUat4t0cuAEnIJzYIIb0AD3oAlaAINX8Am+wHfmXctqBa04t2qZxZ0jsFTa8Q/AB691
    w2 + w23x3

    View full-size slide

  44. Interaction detection in multivariate analysis
    • Adding every possible factors into the model, known as 'kitchen sink' regression, was
    considered a flawed method that only led to overfitting
    https://en.wikipedia.org/wiki/Kitchen_sink_regression
    • In multivariate analysis, it is necessary for the analyst to carefully select the interaction terms
    to be included in the model based on domain knowledge
    An early idea toward the decision tree: AID = Automatic Interaction Detection
    Can't we automate this combinatorial feature selection?
    It naturally considers interactions like a contingency table (also known as a cross tabulation)?

    View full-size slide

  45. Interaction detection in multivariate analysis
    An early idea toward the decision tree: AID = Automatic Interaction Detection
    AID (Morgan & Sonquist 1963) … A regression tree
    Theta AID (Messenger & Mandell, 1972) … A classification tree
    It was presented in the statistical community, but didn't receive much attention and faced
    quite harsh criticism...!
    • Overfitting issue: AID was pointed out to overfit in simulations (Einhorn, 1972).
    • Masking problem: When there are variables with high correlation, only one appears in the
    tree, leading to a spurious conclusion about the variable's importance (Doyle, 1973).
    • Lack of distinction between population and sample: It completely ignores the inherent
    'sampling variability' in the data (Bishop+, 1975).

    View full-size slide

  46. Brief history of decision trees
    1) Concept learning 2) Interaction detection 3) Nonlinear regression
    CLS (Hunt+ 1966)
    ID3 (Quinlan 1986)
    C4.5 (Quinlan 1993)
    C5.0/See5 (Quinlan)
    AID (Morgan & Sonquist 1963)
    Breiman & Meisel 1976
    Friedman 1979
    THAID (Messenger & Mandell, 1972)
    CHAID (Kass 1980)
    CART (Breiman+ 1984)
    M5 (Quinlan 1992)
    M5’ (Wang & Witten 1997)
    MARS (Friedman 1991)
    Cubist (Quinlan)
    RETIS (Karalic & Cestnik 1991)
    HTL (Torgo 1997)
    QUEST (Loh & Shih 1997)
    CRUISE (Kim & Loh, 2001)
    OC1 (Murthy+ 1993)
    Quinlan 1979
    Tree-shaped classification Tree-shaped regression
    Decision tree = A concept so simple that anyone could think of it
    In fact, this idea was explored in various contexts in the 1970s

    View full-size slide

  47. Nonlinear regression
    One might think decision trees are for classification, it also has roots in regression.
    “Regression” problem
    • The problem of fitting a curve to points.
    • Linear regression = fitting a "straight line".
    • However, there are many situations where what we want is
    clearly not a straight line … → Nonlinear regression
    Univariate (1D) regression
    KNeighborsRegressor
    “Nearest Neighbor Estimates” “Kernel Estimates” “Partitioning Estimates”
    DecisionTreeRegressor
    KernelRegression

    View full-size slide

  48. Nonlinear regression
    When solving this with polynomial regression, the optimal degree is hard to get…
    Polynomial Regression (4th) Polynomial Regression (10th) Polynomial Regression (30th)
    DecisionTreeRegressor (4 leaves) DecisionTreeRegressor (10 leaves) DecisionTreeRegressor (30 leaves)
    For regression trees, the degree of freedom simply controls the goodness of fit.

    View full-size slide

  49. Nonlinear regression
    Let’s consider a dumbest regression model, i.e., constant regression.
    (No matter what the value of the input is, it returns the mean of the training data)
    Indeed, this produces the safest predictions, which won’t be wrong by a huge margin.
    But ’always returning a constant value for any x’ is neither ML nor anything else...
    x
    The idea of regression trees: What about piecewise constants?
    Change the constant returned locally based on the value of x.
    x

    View full-size slide

  50. Nonlinear regression
    Text text
    Region averages
    Regression trees
    Find a splitting point in a certain way.
    For each region, it returns the region averages
    as predicted values.
    Apply this process recursively in each sub-region.
    We have a regression tree. (with 4 leaf nodes)

    View full-size slide

  51. Nonlinear regression
    Text text
    ≤ >
    DecisionTreeRegressor (10 leaves)
    For univariate data

    View full-size slide

  52. Nonlinear regression
    Text text
    https://explained.ai/decision-tree-viz/
    "Diabetes" Dataset
    The procedure is the same for
    multivariate data
    (as will be discussed later)

    View full-size slide

  53. Outline
    We will jointly learn decision trees and ensemble learning!
    1. Why we should learn decision trees and ensemble learning?
    2. History of Decision Trees
    3. Basics of Decision Trees
    4. Ensemble Learning: Additive ensembles and decision forests
    5. Random forest and Regularized gradient boosted trees
    6. Practical data analysis with decision tree ensembles
    Part 1
    Part 2

    View full-size slide

  54. Classification trees vs. Regression trees
    Basic learning algorithms are the same (except splitting criteria and leaf decisions)
    returns a categorical value
    (one of the prede ned categories)
    >

    ≤ >
    Classification
    trees
    Regression
    trees
    returns a numeric value
    (each leaf can have a di erent value)

    View full-size slide

  55. The three important aspects of decision trees
    1. Nested if-then rules 3. Piecewise constant prediction
    (i.e. histogram approximation)
    2. Recursive partitioning of
    the input space
    AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGZiRVNvh8rijIXqiIrkThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOslho=
    X2
     ✓1
    yes no
    AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGRY0WT78eKosyFqswokThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOelho=
    X1
     ✓2
    yes no
    Blue
    AAAClnichVHLSsNAFD2N7/qquim4KZaKqzItRcWFiCK6rNZqoZWQxGkbTJOYTCta/AF/wIW4UFARP8APcOMPuOgniMsKblx4mwZExXrDZM6cuefOmbmqbeiuYKwRkLq6e3r7+geCg0PDI6OhsfFt16o6Gs9qlmE5OVVxuaGbPCt0YfCc7XClohp8R91fae3v1Ljj6pa5JY5svltRSqZe1DVFECWHxnJyMlIw+EGkIMpcKHJKDkVZnHkR+Q0SPojCj7QVekABe7CgoYoKOEwIwgYUuPTlkQCDTdwu6sQ5hHRvn+MEQdJWKYtThkLsPv1LtMr7rEnrVk3XU2t0ikHDIWUEMfbM7liTPbF79sI+/qxV92q0vBzRrLa13JZHT8OZ939VFZoFyl+qjp4Fipj3vOrk3faY1i20tr52fNbMLGzG6tPsir2S/0vWYI90A7P2pl1v8M3zDn5U8kIvRg1K/GzHb7CdjCdm46mNVHRp2W9VPyYxhRnqxxyWsI40slT/EBe4wa0UlhalVWmtnSoFfM0EvoWU/gQ6DJYd
    X2
     ✓4
    yes no
    Red AAAClnichVHLSsNAFD3GV62vVjeCm2JRXJWJ+MKFFEV0WVurBSshiVMNTZOYTCta/AF/wIW4UFARP8APcOMPuOgniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGGu0SK1t7R2dka5od09vX38sPrDu2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvaOVFf3+jxl3PsK01ceDwrYq6YxklQ1cFUUosXlDkRNHke4mi2OVCVaaUWJKlWBCJn0AOQRJhZOzYHYrYhg0dVVTAYUEQNqHCo28TMhgc4rZQJ84lZAT7HEeIkrZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHaX/eCj39q+qQrPA7qfqT88CJcwGXg3y7gSMfwu9qa8dnrzk5rKj9TF2wZ7J/zlrsHu6gVV71S9Xefb0Dz8aeaEXowbJ39vxE6xPpOTp1OTqZDK9ELYqgmGMYJz6MYM0VpBBnurv4wxXuJaGpHlpSVpupkotoWYQX0LKfAA5/pYd
    X1
     ✓5
    yes no
    Red AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4=
    X1
     ✓6
    yes no
    Red
    Blue
    AAAClnichVHLSsNAFD3GV3221Y3gplgUV2WiRcWFFEV0Wa3VgpWQxKmGpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxKWCGxfepAFRUW+YzJkz99w5M1dzTMMTjDVapNa29o7OSFd3T29ffzQWH1j37Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tPKCv79R465n2NaaOHD4VkXdsYySoauCKCUWLyhyomjyvURR7HKhKpNKLMlSLIjETyCHIIkwsnbsDkVsw4aOKirgsCAIm1Dh0bcJGQwOcVuoE+cSMoJ9jiN0k7ZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHSV6PJR7+1dVoVlg91P1p2eBEmYCrwZ5dwLGv4Xe1NcOT15ys6uj9TF2wZ7J/zlrsHu6gVV71S9X+OrpH3408kIvRg2Sv7fjJ1ifSMlTqfRKOpmZD1sVwTBGME79mEYGy8giT/X3cYYrXEtD0py0KC01U6WWUDOILyFlPwA1vpYb
    X1
     ✓3
    yes no
    Red
    AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykWHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm4/JcPJFJxFJLQav6MIFJzFA/kkhhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bz4+lh8=
    X1
     ✓7
    yes no
    Blue Blue
    AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSTayFoyRO3IiMAtEDUfAibYQfoQInYIACLdCAgg4csQoS2PiVQQQCJnJV6CJnIWLuPoVzCKC2hVkUMyRkm/iv46rssTqunZq2q1bwFBWHhcoIxMgLuScD8kweyCv5/LNW163heOngLA+11KyFLlZyH/+qNJw5NL5VYz1zOIVd1ytD76bLOLdQhvr22dUgl8zGuuvklryh/xvSJ094A739rtxlaLY3xo+MXvDFsEHi73aMgsJmXNyOJzKJaGrfa5UfVmENNrAfO5CCI0hD3u3DJfTgWggKW0JS2BumCj5Psww/Qjj8Ak3rksg=
    ✓1
    AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRAooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4V4NLYTTWQS4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXtoWksD9KFTyuZhV+hHD0BVALksk=
    ✓2
    AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSLVELR0mcuBEZBaIHouBF2gg/QgVOwAAFWqABBR04YhUksPErgwgETOSq0EXOQsTcfQrnEEBtC7MoZkjINvFfx1XZY3VcOzVtV63gKSoOC5URiJEXck8G5Jk8kFfy+WetrlvD8dLBWR5qqVkLXazkPv5VaThzaHyrxnrmcAq7rleG3k2XcW6hDPXts6tBLpmNddfJLXlD/zekT57wBnr7XbnL0GxvjB8ZveCLYYPE3+0YBYXNuLgdT2QS0dS+1yo/rMIabGA/diAFR5CGvNuHS+jBtRAUtoSksDdMFXyeZhl+hHD4BVRLkss=
    ✓4
    AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQfIRYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixPS9wgTk1PTM95Z39y8fyEQXFzKW3rLlGlO1hXdLEqiRRWm0RxnXKFFw6SiKim0IDUP7f1Cm5oW07UT3jFoRRXrGqsxWeRIFcu8QblY3a4GwyRKnAiNgpgLwuBGSg8+QhlOQQcZWqACBQ04YgVEsPArQQwIGMhVoIuciYg5+xTOwYfaFmZRzBCRbeK/jquSy2q4tmtajlrGUxQcJipDECEv5J4MyDN5IK/k889aXaeG7aWDszTUUqMauFjJfvyrUnHm0PhWjfXMoQZ7jleG3g2HsW8hD/Xts6tBNpGJdNfJLXlD/zekT57wBlr7Xb5L00xvjB8JveCLYYNiv9sxCvKb0dhONJ6Oh5MHbqu8sAprsIH92IUkHEMKck4fLqEH14Jf2BISwv4wVfC4mmX4EcLRF1Zrksw=
    ✓5
    AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQgoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaCwRjWfi4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BViLks0=
    ✓6
    AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmQQooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGwnmsgkwqkDt1VeWIcN2MJ+7EIKjiENeacPl9CHa8EvxIWksD9KFTyuZhV+hHD0BVIrkso=
    ✓3
    AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQIoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGw3Gs/Ew6kDt1VeWIcN2MJ+JCAFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BVqrks4=
    ✓7
    AAAB+XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzQOSJcxOZpMh81hmZoWw5BO86tmbePVrPPonTpI9mMSChqKqm+6uKOHMWN//9gpr6xubW8Xt0s7u3v5B+fCoaVSqCW0QxZVuR9hQziRtWGY5bSeaYhFx2opGd1O/9Uy1YUo+2XFCQ4EHksWMYOukx3Yv6JUrftWfAa2SICcVyFHvlX+6fUVSQaUlHBvTCfzEhhnWlhFOJ6VuamiCyQgPaMdRiQU1YTY7dYLOnNJHsdKupEUz9e9EhoUxYxG5ToHt0Cx7U/E/r5Pa+CbMmExSSyWZL4pTjqxC079Rn2lKLB87golm7lZEhlhjYl06C1siMXGZBMsJrJLmRTW4ql4+XFZqt3k6RTiBUziHAK6hBvdQhwYQGMALvMKbl3nv3of3OW8tePnMMSzA+/oFiY2UHw==
    X1
    AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5CUMugjWVE8wHJEfY2e8mS3b1jd08IR36CrdZ2YuuvsfSfuLlcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzN/c4zVZpF8slMY+oLPJIsZAQbKz12B7VBueJW3QxonXg5qUCO5qD80x9GJBFUGsKx1j3PjY2fYmUY4XRW6ieaxphM8Ij2LJVYUO2n2akzdGGVIQojZUsalKl/J1IstJ6KwHYKbMZ61ZuL/3m9xIQ3fspknBgqyWJRmHBkIjT/Gw2ZosTwqSWYKGZvRWSMFSbGprO0JRAzm4m3msA6adeq3lW1/lCvNG7zdIpwBudwCR5cQwPuoQktIDCCF3iFNyd13p0P53PRWnDymVNYgvP1C4sglCA=
    X2

    View full-size slide

  56. Recursive partitioning of the input space
    AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSTayFoyRO3IiMAtEDUfAibYQfoQInYIACLdCAgg4csQoS2PiVQQQCJnJV6CJnIWLuPoVzCKC2hVkUMyRkm/iv46rssTqunZq2q1bwFBWHhcoIxMgLuScD8kweyCv5/LNW163heOngLA+11KyFLlZyH/+qNJw5NL5VYz1zOIVd1ytD76bLOLdQhvr22dUgl8zGuuvklryh/xvSJ094A739rtxlaLY3xo+MXvDFsEHi73aMgsJmXNyOJzKJaGrfa5UfVmENNrAfO5CCI0hD3u3DJfTgWggKW0JS2BumCj5Psww/Qjj8Ak3rksg=
    ✓1
    AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGZiRVNvh8rijIXqiIrkThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOslho=
    X2
     ✓1
    yes no
    AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRAooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4V4NLYTTWQS4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXtoWksD9KFTyuZhV+hHD0BVALksk=
    ✓2
    AAAClnichVHLSsNAFD3Gd3201Y3gplgUV2UiRcWFFEV02YfVgpWQxLENpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxGUFNy68TQOiot4wmTNn7rlzZq7mmIYnGGt0SJ1d3T29ff2hgcGh4XAkOrLh2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvanvLrf3NGnc9w7bWxaHDtytqyTJ2DV0VRCmRaEGRY0WT78eKosyFqswokThLMD9iP4EcgDiCSNuRexSxAxs6qqiAw4IgbEKFR98WZDA4xG2jTpxLyPD3OY4RIm2VsjhlqMTu0b9Eq62AtWjdqun5ap1OMWm4pIxhkj2xW9Zkj+yOPbP3X2vV/RotL4c0a20td5TwyVju7V9VhWaB8qfqT88Cu5j3vRrk3fGZ1i30tr52dNrMLWQn61Pskr2Q/wvWYA90A6v2ql9lePbsDz8aeaEXowbJ39vxE2zMJOTZRDKTjKeWglb1YRwTmKZ+zCGFNaSRp/oHOMc1bqQxaVFakVbbqVJHoBnFl5DSHzOelho=
    X1
     ✓2
    yes no
    Blue
    AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNagEi6AIlnmYByQh7K5jMmRf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCu5sF0WC8y+ycOXPPnTNzZVNlNiek7xMmJqemZ/yzgbn54EIovLhUsI2WpdC8YqiGVZIlm6pMp3nOuEpLpkUlTVZpUW4eOPvFNrVsZujHvGPSqibVdXbKFIkjVarwBuVSLVELR0mcuBEZBaIHouBF2gg/QgVOwAAFWqABBR04YhUksPErgwgETOSq0EXOQsTcfQrnEEBtC7MoZkjINvFfx1XZY3VcOzVtV63gKSoOC5URiJEXck8G5Jk8kFfy+WetrlvD8dLBWR5qqVkLXazkPv5VaThzaHyrxnrmcAq7rleG3k2XcW6hDPXts6tBLpmNddfJLXlD/zekT57wBnr7XbnL0GxvjB8ZveCLYYPE3+0YBYXNuLgdT2QS0dS+1yo/rMIabGA/diAFR5CGvNuHS+jBtRAUtoSksDdMFXyeZhl+hHD4BVRLkss=
    ✓4
    AAAClnichVHLSsNAFD2N7/qquim4KZaKqzItRcWFiCK6rNZqoZWQxGkbTJOYTCta/AF/wIW4UFARP8APcOMPuOgniMsKblx4mwZExXrDZM6cuefOmbmqbeiuYKwRkLq6e3r7+geCg0PDI6OhsfFt16o6Gs9qlmE5OVVxuaGbPCt0YfCc7XClohp8R91fae3v1Ljj6pa5JY5svltRSqZe1DVFECWHxnJyMlIw+EGkIMpcKHJKDkVZnHkR+Q0SPojCj7QVekABe7CgoYoKOEwIwgYUuPTlkQCDTdwu6sQ5hHRvn+MEQdJWKYtThkLsPv1LtMr7rEnrVk3XU2t0ikHDIWUEMfbM7liTPbF79sI+/qxV92q0vBzRrLa13JZHT8OZ939VFZoFyl+qjp4Fipj3vOrk3faY1i20tr52fNbMLGzG6tPsir2S/0vWYI90A7P2pl1v8M3zDn5U8kIvRg1K/GzHb7CdjCdm46mNVHRp2W9VPyYxhRnqxxyWsI40slT/EBe4wa0UlhalVWmtnSoFfM0EvoWU/gQ6DJYd
    X2
     ✓4
    yes no
    Red
    AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQfIRYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixPS9wgTk1PTM95Z39y8fyEQXFzKW3rLlGlO1hXdLEqiRRWm0RxnXKFFw6SiKim0IDUP7f1Cm5oW07UT3jFoRRXrGqsxWeRIFcu8QblY3a4GwyRKnAiNgpgLwuBGSg8+QhlOQQcZWqACBQ04YgVEsPArQQwIGMhVoIuciYg5+xTOwYfaFmZRzBCRbeK/jquSy2q4tmtajlrGUxQcJipDECEv5J4MyDN5IK/k889aXaeG7aWDszTUUqMauFjJfvyrUnHm0PhWjfXMoQZ7jleG3g2HsW8hD/Xts6tBNpGJdNfJLXlD/zekT57wBlr7Xb5L00xvjB8JveCLYYNiv9sxCvKb0dhONJ6Oh5MHbqu8sAprsIH92IUkHEMKck4fLqEH14Jf2BISwv4wVfC4mmX4EcLRF1Zrksw=
    ✓5
    AAAClnichVHLSsNAFD3GV62vVjeCm2JRXJWJ+MKFFEV0WVurBSshiVMNTZOYTCta/AF/wIW4UFARP8APcOMPuOgniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGGu0SK1t7R2dka5od09vX38sPrDu2VVX53ndNm23oKkeNw2L54UhTF5wXK5WNJNvaOVFf3+jxl3PsK01ceDwrYq6YxklQ1cFUUosXlDkRNHke4mi2OVCVaaUWJKlWBCJn0AOQRJhZOzYHYrYhg0dVVTAYUEQNqHCo28TMhgc4rZQJ84lZAT7HEeIkrZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHaX/eCj39q+qQrPA7qfqT88CJcwGXg3y7gSMfwu9qa8dnrzk5rKj9TF2wZ7J/zlrsHu6gVV71S9Xefb0Dz8aeaEXowbJ39vxE6xPpOTp1OTqZDK9ELYqgmGMYJz6MYM0VpBBnurv4wxXuJaGpHlpSVpupkotoWYQX0LKfAA5/pYd
    X1
     ✓5
    yes no
    Red
    AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQgoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaCwRjWfi4dSB2yovrMMGbGE/diEFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BViLks0=
    ✓6
    AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykVHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm43IynsgkYqmloFV9mMAkZqgfc0hhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bzwelh4=
    X1
     ✓6
    yes no
    Red
    Blue
    AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmQQooZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYi9eCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGwnmsgkwqkDt1VeWIcN2MJ+7EIKjiENeacPl9CHa8EvxIWksD9KFTyuZhV+hHD0BVIrkso=
    ✓3
    AAAClnichVHLSsNAFD3GV3221Y3gplgUV2WiRcWFFEV0Wa3VgpWQxKmGpklMphUt/oA/4EJcKKiIH+AHuPEHXPQTxKWCGxfepAFRUW+YzJkz99w5M1dzTMMTjDVapNa29o7OSFd3T29ffzQWH1j37Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tPKCv79R465n2NaaOHD4VkXdsYySoauCKCUWLyhyomjyvURR7HKhKpNKLMlSLIjETyCHIIkwsnbsDkVsw4aOKirgsCAIm1Dh0bcJGQwOcVuoE+cSMoJ9jiN0k7ZKWZwyVGLL9N+h1WbIWrT2a3qBWqdTTBouKRMYZY/shr2wB3bLntj7r7XqQQ3fywHNWlPLHSV6PJR7+1dVoVlg91P1p2eBEmYCrwZ5dwLGv4Xe1NcOT15ys6uj9TF2wZ7J/zlrsHu6gVV71S9X+OrpH3408kIvRg2Sv7fjJ1ifSMlTqfRKOpmZD1sVwTBGME79mEYGy8giT/X3cYYrXEtD0py0KC01U6WWUDOILyFlPwA1vpYb
    X1
     ✓3
    yes no
    Red
    AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVmRQIoZYEI2JJQ95JEDI7jrAhH1ldyBB4g9Y2lhgo4mF8QP8ABt/wIJPMJaY2Fh4d9nEKBHvZnbOnLnnzpm5kqEwixMy8AhT0zOzc95538KifykQXF4pWHrblGle1hXdLEmiRRWm0TxnXKElw6SiKim0KLUO7f1ih5oW07UT3jVoVRUbGqszWeRIlSq8SblYS9SCYRIlToTGQcwFYXAjrQcfoQKnoIMMbVCBggYcsQIiWPiVIQYEDOSq0EPORMScfQrn4ENtG7MoZojItvDfwFXZZTVc2zUtRy3jKQoOE5UhiJAXck+G5Jk8kFfy+WetnlPD9tLFWRppqVELXKzlPv5VqThzaH6rJnrmUIc9xytD74bD2LeQR/rO2dUwl8xGepvklryh/xsyIE94A63zLt9laLY/wY+EXvDFsEGx3+0YB4XtaGw3Gs/Ew6kDt1VeWIcN2MJ+JCAFx5CGvNOHS+jDteAXdoSksD9KFTyuZhV+hHD0BVqrks4=
    ✓7
    AAAClnichVHLSsNAFD3Gd321uhHcFIviqkykWHEhRRFd9mG1YCUkcWyDaRKTaUWLP+APuBAXCiriB/gBbvwBF/0EcVnBjQtv04CoqDdM5syZe+6cmas5puEJxhodUmdXd09vX39oYHBoeCQcGd3w7Kqr87xum7Zb0FSPm4bF88IQJi84Llcrmsk3tb3l1v5mjbueYVvr4tDh2xW1ZBm7hq4KopRwpKDI0aLJ96NFUeZCVZJKOMbizI/oTyAHIIYg0nb4HkXswIaOKirgsCAIm1Dh0bcFGQwOcduoE+cSMvx9jmOESFulLE4ZKrF79C/RaitgLVq3anq+WqdTTBouKaOYYk/sljXZI7tjz+z911p1v0bLyyHNWlvLHWXkZDz39q+qQrNA+VP1p2eBXcz7Xg3y7vhM6xZ6W187Om3mFrJT9Wl2yV7I/wVrsAe6gVV71a8yPHv2hx+NvNCLUYPk7+34CTZm4/JcPJFJxFJLQav6MIFJzFA/kkhhDWnkqf4BznGNG2lcWpRWpNV2qtQRaMbwJaT0Bz4+lh8=
    X1
     ✓7
    yes no
    Blue Blue
    if x2 ≤ θ1 then
    if x1 ≤ θ2 then
    return Blue
    else
    if x2 ≤ θ4 then
    return Red
    else
    if x1 ≤ θ5 then
    return Red
    else
    if x1 ≤ θ6 then
    return Blue
    else
    return Red
    else
    if x1 ≤ θ3 then
    if x2 ≤ θ7 then
    return Blue
    else
    return Blue
    else
    return Red
    AAAB+XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzQOSJcxOZpMh81hmZoWw5BO86tmbePVrPPonTpI9mMSChqKqm+6uKOHMWN//9gpr6xubW8Xt0s7u3v5B+fCoaVSqCW0QxZVuR9hQziRtWGY5bSeaYhFx2opGd1O/9Uy1YUo+2XFCQ4EHksWMYOukx3Yv6JUrftWfAa2SICcVyFHvlX+6fUVSQaUlHBvTCfzEhhnWlhFOJ6VuamiCyQgPaMdRiQU1YTY7dYLOnNJHsdKupEUz9e9EhoUxYxG5ToHt0Cx7U/E/r5Pa+CbMmExSSyWZL4pTjqxC079Rn2lKLB87golm7lZEhlhjYl06C1siMXGZBMsJrJLmRTW4ql4+XFZqt3k6RTiBUziHAK6hBvdQhwYQGMALvMKbl3nv3of3OW8tePnMMSzA+/oFiY2UHw==
    X1
    AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5CUMugjWVE8wHJEfY2e8mS3b1jd08IR36CrdZ2YuuvsfSfuLlcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzN/c4zVZpF8slMY+oLPJIsZAQbKz12B7VBueJW3QxonXg5qUCO5qD80x9GJBFUGsKx1j3PjY2fYmUY4XRW6ieaxphM8Ij2LJVYUO2n2akzdGGVIQojZUsalKl/J1IstJ6KwHYKbMZ61ZuL/3m9xIQ3fspknBgqyWJRmHBkIjT/Gw2ZosTwqSWYKGZvRWSMFSbGprO0JRAzm4m3msA6adeq3lW1/lCvNG7zdIpwBudwCR5cQwPuoQktIDCCF3iFNyd13p0P53PRWnDymVNYgvP1C4sglCA=
    X2
    AAAB+XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5CUMugjWVE8wHJEfY2e8mS3b1jd08IR36CrdZ2YuuvsfSfuLlcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzN/c4zVZpF8slMY+oLPJIsZAQbKz12B7VBueJW3QxonXg5qUCO5qD80x9GJBFUGsKx1j3PjY2fYmUY4XRW6ieaxphM8Ij2LJVYUO2n2akzdGGVIQojZUsalKl/J1IstJ6KwHYKbMZ61ZuL/3m9xIQ3fspknBgqyWJRmHBkIjT/Gw2ZosTwqSWYKGZvRWSMFSbGprO0JRAzm4m3msA6adeq3lW1/lCvNG7zdIpwBudwCR5cQwPuoQktIDCCF3iFNyd13p0P53PRWnDymVNYgvP1C4sglCA=
    X2
    AAAB+XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzQOSJcxOZpMh81hmZoWw5BO86tmbePVrPPonTpI9mMSChqKqm+6uKOHMWN//9gpr6xubW8Xt0s7u3v5B+fCoaVSqCW0QxZVuR9hQziRtWGY5bSeaYhFx2opGd1O/9Uy1YUo+2XFCQ4EHksWMYOukx3Yv6JUrftWfAa2SICcVyFHvlX+6fUVSQaUlHBvTCfzEhhnWlhFOJ6VuamiCyQgPaMdRiQU1YTY7dYLOnNJHsdKupEUz9e9EhoUxYxG5ToHt0Cx7U/E/r5Pa+CbMmExSSyWZL4pTjqxC079Rn2lKLB87golm7lZEhlhjYl06C1siMXGZBMsJrJLmRTW4ql4+XFZqt3k6RTiBUziHAK6hBvdQhwYQGMALvMKbl3nv3of3OW8tePnMMSzA+/oFiY2UHw==
    X1
    AAAClnichVHLSsNAFD3GV323uhHcFEvFVZmKqLiQoogu66NaaEtI4lhH0yQm00ot/oA/4EJcKKiIH+AHuPEHXPgJ4lLBjQtv04BoUW+YzJkz99w5M1d3TOFJxp5alNa29o7OUFd3T29f/0A4Mrjh2WXX4BnDNm03q2seN4XFM1JIk2cdl2sl3eSb+t5CfX+zwl1P2Na6rDq8UNKKltgWhiaJUsORrCqieZPvR/Nyh0tN3VXDMZZgfkSbQTIAMQSRtsN3yGMLNgyUUQKHBUnYhAaPvhySYHCIK6BGnEtI+PscR+gmbZmyOGVoxO7Rv0irXMBatK7X9Hy1QaeYNFxSRhFnj+yGvbIHdsue2cevtWp+jbqXKs16Q8sddeB4eO39X1WJZomdL9WfniW2MeN7FeTd8Zn6LYyGvnJ48ro2uxqvjbEL9kL+z9kTu6cbWJU343KFr57+4UcnL/Ri1KDkz3Y0g42JRHIqMbkyGUvNB60KYQSjGKd+TCOFZaSRofoHOMMVrpVhZU5ZVJYaqUpLoBnCt1DSnyS9loo=
    Xi
     ✓j
    yes no
    Learning a tree = Recursively find each split (stump)

    View full-size slide

  57. Recursively partitioning the data
    Text text
    virginica
    versicolor
    setosa

    View full-size slide

  58. Recursively partitioning the data
    Multi-dimensional data is viewed by projecting onto one dimension for each variable.
    virginica
    versicolor
    setosa
    Note: Decision trees do not look at
    multiple variables simultaneously!
    (that’s why it’s free from the curse of
    dimensionality…)

    View full-size slide

  59. Recursively partitioning the data
    virginica
    versicolor
    setosa
    Multi-dimensional data is viewed by projecting onto one dimension for each variable.
    Note: Decision trees do not look at
    multiple variables simultaneously!
    (that’s why it’s free from the curse of
    dimensionality…)

    View full-size slide

  60. Recursively partitioning the data
    virginica
    versicolor
    setosa
    Multi-dimensional data is viewed by projecting onto one dimension for each variable.
    Note: Decision trees do not look at
    multiple variables simultaneously!
    (that’s why it’s free from the curse of
    dimensionality…)

    View full-size slide

  61. Recursively partitioning the data
    virginica
    versicolor
    setosa
    Multi-dimensional data is viewed by projecting onto one dimension for each variable.
    Note: Decision trees do not look at
    multiple variables simultaneously!
    (that’s why it’s free from the curse of
    dimensionality…)
    threshold
    Take the threshold at
    the midpoint between
    neighboring points.

    View full-size slide

  62. Recursively partitioning the data
    virginica
    versicolor
    setosa
    Pick one of variables, and split the input space by a threshold on it
    threshold
    threshold
    Multi-dimensional data is viewed by projecting onto one dimension for each variable.

    View full-size slide

  63. Recursively partitioning the data
    Recursive bi-partitioning
    Apply the same process assuming that the subregion data was given from the beginning
    No need for further splitting
    (We stop resursion)
    setosa
    We do splitting once more
    (We have refinements)

    View full-size slide

  64. So how can we find the best split?
    Learning a stump = Finding the best split (a variable + a threshold)
    A toy example of two variable (petal.length & petal.width)
    These splits geometrically correspond to axis-parallel lines in the 2D input space
    = Simply examining all possible splits one by one!
    (Calculate the split criterion value for each split as we explain later)

    View full-size slide

  65. Finding the best splits
    In the iris dataset, two best splits are found based on the split criterion value (in this
    example, smaller is better). Choose one of them to split.
    iris ( )

    View full-size slide

  66. Finding the best splits (scikit-learn)
    Text text
    ≤ >
    Depth

    View full-size slide

  67. The split criterion: Impurity scores
    Since decision trees make constant predictions at leaf nodes, a 'good split' is one where the values
    within the region are as uniform as possible (i.e., with minimal mixing).
    Small impurity Large impurity Small impurity Large impurity
    Criteria for classification
    sklearn.tree.DecisionTreeClassifier sklearn.tree.DecisionTreeRegressor
    •"gini"
    The Gini impurity
    •"entropy", "log_loss"
    The Shannon information gain
    •"squared_error"
    The mean squared error
    •"absolute_error"
    The mean absolute error
    •"poisson"
    Poisson deviance
    •"friedman_mse"
    (A special one for gradient boosting)
    Criteria for regression

    View full-size slide

  68. Split criteria (impurities) for classification
    AAACJHicdVDLSgMxFM3UV62vqks3wSJU0DJT6mNTKLoR3FSwD+jUIZOmbZhkZkgyQhnmJ/wIv8Gtrt2JCzeCf2KmrWB9HAgczrmXm3PckFGpTPPNyMzNLywuZZdzK6tr6xv5za2mDCKBSQMHLBBtF0nCqE8aiipG2qEgiLuMtFzvPPVbt0RIGvjXahSSLkcDn/YpRkpLTv7A5kgNBY8pD5Niex9W4aEtI+7EXtVKbi5h6HjQZsEgJU6+YJbKR6YG/E2skjlGAUxRd/Ifdi/AESe+wgxJ2bHMUHVjJBTFjCQ5O5IkRNhDA9LR1EecyG48TpXAPa30YD8Q+vkKjtXvGzHiUo64qyfTDPKnl4p/eZ1I9U+7MfXDSBEfTw71IwZVANOKYI8KghUbaYKwoPqvEA+RQFjpImeuuDzRnXwFh/+TZrlkHZcqV5VC7WzaThbsgF1QBBY4ATVwAeqgATC4Aw/gETwZ98az8WK8TkYzxnRnG8zAeP8EUA2kiQ==
    imp(X) =
    K
    X
    k=1
    pk log pk
    AAACIHicdVDJSgNBFOyJW4zbqEcvjYmQU5iEuCAIQS8eI5gFMiH0dHqSJj0L3W/EMJlf8CP8Bq969iYe9eaf2FkE41LQUFS9pV85oeAKLOvNSC0sLi2vpFcza+sbm1vm9k5dBZGkrEYDEcimQxQT3Gc14CBYM5SMeI5gDWdwMfYbN0wqHvjXMAxZ2yM9n7ucEtBSx8yHnQE+PbNdSWhsA7uF2M7iwMVUEKVwbpBLknjUHCUdM2sVSoeWBv5NigVrgiyaodoxP+xuQCOP+TCZ1SpaIbRjIoFTwZKMHSkWEjogPdbS1CceU+14clGCD7TSxW4g9fMBT9TvHTHxlBp6jq70CPTVT28s/uW1InBP2jH3wwiYT6eL3EhgCPA4HtzlklEQQ00IlVz/FdM+0eGADnFui+ONM/k6HP9P6qVC8ahQvipnK+ezdNJoD+2jPCqiY1RBl6iKaoiiO/SAHtGTcW88Gy/G67Q0Zcx6dtEcjPdPMKijjg==
    pk :=
    # of class k
    |X|
    The ratio of samples from class k in X
    The Gini impurity
    The Shannon information gain
    Note: The Gini coefficient and entropy are more sensitive to probability
    changes than the misclassification rate, making ties less likely to occur.
    AAAB+XicdVDJSgNBEK1xjXGLevTSGARPw0yIy0UIehG8RDQLJEPo6fQkTbp7hu4eIQz5BK969iZe/RqP/omdRTAuDwoe71VRVS9MONPG896dhcWl5ZXV3Fp+fWNza7uws1vXcaoIrZGYx6oZYk05k7RmmOG0mSiKRchpIxxcjv3GPVWaxfLODBMaCNyTLGIEGyvdXp+XOoWi55aOPQv0m/iuN0ERZqh2Ch/tbkxSQaUhHGvd8r3EBBlWhhFOR/l2qmmCyQD3aMtSiQXVQTY5dYQOrdJFUaxsSYMm6veJDAuthyK0nQKbvv7pjcW/vFZqorMgYzJJDZVkuihKOTIxGv+NukxRYvjQEkwUs7ci0scKE2PTmdsSipHN5Otx9D+pl1z/xC3flIuVi1k6OdiHAzgCH06hAldQhRoQ6MEDPMKTkznPzovzOm1dcGYzezAH5+0TYKyUBg==
    K = 2
    AAAB+XicdVDJSgNBEK2JW4xb1KOXxiB4GmZCoh6DXjxGNAskQ+jp9CRNunuG7h4hDPkEr3r2Jl79Go/+iZ1FMC4PCh7vVVFVL0w408bz3p3cyura+kZ+s7C1vbO7V9w/aOo4VYQ2SMxj1Q6xppxJ2jDMcNpOFMUi5LQVjq6mfuueKs1ieWfGCQ0EHkgWMYKNlW6Tnt8rljy3XPUs0G/iu94MJVig3it+dPsxSQWVhnCsdcf3EhNkWBlGOJ0UuqmmCSYjPKAdSyUWVAfZ7NQJOrFKH0WxsiUNmqnfJzIstB6L0HYKbIb6pzcV//I6qYkugozJJDVUkvmiKOXIxGj6N+ozRYnhY0swUczeisgQK0yMTWdpSygmNpOvx9H/pFl2/TO3clMp1S4X6eThCI7hFHw4hxpcQx0aQGAAD/AIT07mPDsvzuu8NecsZg5hCc7bJ89KlEw=
    p1
    AAACP3icdVDLSgMxFM34rPVVdekmWIS6sMyU+tgUii4UuqlgH9DHkEnTNkwyMyQZoQzzSX6E3+BK0IUrd+LWnZm2gq16IHA4515uznECRqUyzSdjYXFpeWU1tZZe39jc2s7s7NalHwpMathnvmg6SBJGPVJTVDHSDARB3GGk4biXid+4I0JS37tVo4B0OBp4tE8xUlqyM1dtjtRQ8IjyIM41j2AJtmXI7cgtWXG3AgPbhTkLHidkzowqMZw63YKdyZr5wompAX8TK2+OkQVTVO3Ma7vn45ATT2GGpGxZZqA6ERKKYkbidDuUJEDYRQPS0tRDnMhONA4cw0Ot9GDfF/p5Co7VnxsR4lKOuKMnk3hy3kvEv7xWqPrnnYh6QaiIhyeH+iGDyodJe7BHBcGKjTRBWFD9V4iHSCCsdMczVxwe606+g8P/Sb2Qt07zxZtitnwxbScF9sEByAELnIEyuAZVUAMY3INH8AxejAfjzXg3PiajC8Z0Zw/MwPj8AhsprSI=
    imp(X) =
    K
    X
    k=1
    pk(1 pk) =
    K
    X
    k=1
    1 p2
    k
    AAACDnicbVDLSsNAFJ3UV62vqODGzWAR3FiS4mtTKLpxWcE+oI1hMp20Q2aSMDMRSsw/+A1ude1O3PoLLv0Tp20WtvXAhcM593Iux4sZlcqyvo3C0vLK6lpxvbSxubW9Y+7utWSUCEyaOGKR6HhIEkZD0lRUMdKJBUHcY6TtBTdjv/1IhKRReK9GMXE4GoTUpxgpLbnmQQ3a8BT2ZMLdNKjZGYzd4KHqmmWrYk0AF4mdkzLI0XDNn14/wgknocIMSdm1rVg5KRKKYkayUi+RJEY4QAPS1TREnEgnnfyfwWOt9KEfCT2hghP170WKuJQj7ulNjtRQzntj8T+vmyj/yklpGCeKhHga5CcMqgiOy4B9KghWbKQJwoLqXyEeIoGw0pXNpHg8053Y8w0skla1Yl9Uzu/OyvXrvJ0iOARH4ATY4BLUwS1ogCbA4Am8gFfwZjwb78aH8TldLRj5zT6YgfH1C0C3mu0=
    = 1
    X
    k=1
    p2
    k
    Example) Binary classi cation ( ) with and
    AAAB+XicbVDLTgJBEOzFF+IL9ehlIjHxRHaNryPRi0eMIiSwIbPDLEyYx2Zm1oRs+ASvevZmvPo1Hv0TB9iDgJV0UqnqTndXlHBmrO9/e4WV1bX1jeJmaWt7Z3evvH/wZFSqCW0QxZVuRdhQziRtWGY5bSWaYhFx2oyGtxO/+Uy1YUo+2lFCQ4H7ksWMYOukh6QbdMsVv+pPgZZJkJMK5Kh3yz+dniKpoNISjo1pB35iwwxrywin41InNTTBZIj7tO2oxIKaMJueOkYnTumhWGlX0qKp+nciw8KYkYhcp8B2YBa9ifif105tfB1mTCappZLMFsUpR1ahyd+oxzQllo8cwUQzdysiA6wxsS6duS2RGLtMgsUElsnTWTW4rF7cn1dqN3k6RTiCYziFAK6gBndQhwYQ6MMLvMKbl3nv3of3OWstePnMIczB+/oFr9eUOA==
    p1 AAAB+XicbVDLSgNBEOyNrxhfUY9eBoPgKewGX8egF48RzQOSJcxOZpMhM7PLzKwQlnyCVz17E69+jUf/xNlkDyaxoKGo6qa7K4g508Z1v53C2vrG5lZxu7Szu7d/UD48aukoUYQ2ScQj1QmwppxJ2jTMcNqJFcUi4LQdjO8yv/1MlWaRfDKTmPoCDyULGcHGSo9xv9YvV9yqOwNaJV5OKpCj0S//9AYRSQSVhnCsdddzY+OnWBlGOJ2WeommMSZjPKRdSyUWVPvp7NQpOrPKAIWRsiUNmql/J1IstJ6IwHYKbEZ62cvE/7xuYsIbP2UyTgyVZL4oTDgyEcr+RgOmKDF8YgkmitlbERlhhYmx6SxsCcTUZuItJ7BKWrWqd1W9fLio1G/zdIpwAqdwDh5cQx3uoQFNIDCEF3iFNyd13p0P53PeWnDymWNYgPP1C7FqlDk=
    p2
    0.5 × The Shannon information gain
    The Gini impurity
    Misclassification rate
    AAACB3icbVDLSgMxFM34rPXRUZdugkWoCGWm+NoIRTcuK9gHtMOQSTNtaJIJSUYopR/gN7jVtTtx62e49E9M21nY1gMXDufcy7mcSDKqjed9Oyura+sbm7mt/PbO7l7B3T9o6CRVmNRxwhLVipAmjApSN9Qw0pKKIB4x0owGdxO/+USUpol4NENJAo56gsYUI2Ol0C2UZOjDMyjDCryB/mnoFr2yNwVcJn5GiiBDLXR/Ot0Ep5wIgxnSuu170gQjpAzFjIzznVQTifAA9UjbUoE40cFo+vgYnlilC+NE2REGTtW/FyPEtR7yyG5yZPp60ZuI/3nt1MTXwYgKmRoi8CwoThk0CZy0ALtUEWzY0BKEFbW/QtxHCmFju5pLifjYduIvNrBMGpWyf1m+eDgvVm+zdnLgCByDEvDBFaiCe1ADdYBBCl7AK3hznp1358P5nK2uONnNIZiD8/ULzR+XTA==
    (p1 + p2 = 1)

    View full-size slide

  69. Split criteria (impurities) for regression
    The mean squared error (MSE)
    The mean absolute error (MAE)
    Poisson deviance (when y is a count value or frequency)
    AAACT3icdVDLSgMxFM3UV31XXboJFqGClhmpj40gunGpYK3QqUMmzbTBJDMkGWGI+TQ/wqVLcatrd2JaK/g8EDg5996cmxNnjCrt+w9eaWx8YnKqPD0zOze/sFhZWr5QaS4xaeKUpfIyRoowKkhTU83IZSYJ4jEjrfj6eFBv3RCpaCrOdZGRDkc9QROKkXZSVGmFHOm+5IbyzNZCU0TBZthNtdosIhHaDXgAw0QibAJrhA1VziNDDwJ75W7wtogo3Pp8gZMuRcLWTGE3bqNK1a9v7/gO8DcJ6v4QVTDCaVR5dLY450RozJBS7cDPdMcgqSlmxM6EuSIZwteoR9qOCsSJ6phhABauO6ULk1S6IzQcql8nDOJKFTx2nYNl1c/aQPyr1s51st8xVGS5JgJ/GCU5gzqFgzRhl0qCNSscQVhStyvEfeTy0i7zby4xty6Tz4/D/8nFdj3YrTfOGtXDo1E6ZbAK1kANBGAPHIITcAqaAIM78ASewYt37716b6VRa8kbkRXwDaXpd7h5tgI=
    imp({y1, . . . , yn
    }) =
    1
    n
    n
    X
    i=1
    |yi median(y)|
    AAACTXicdVBJSwMxGM3Ufa969BIsQgtaZorbRRC9eKxgVejUIZNmNJhkhiQjDCG/zB/h2WOvevYmYrqB64PAy/v2F2eMKu37z15pYnJqemZ2bn5hcWl5pby6dqnSXGLSwilL5XWMFGFUkJammpHrTBLEY0au4vvTfvzqgUhFU3Ghi4x0OLoVNKEYaSdF5VbIkb6T3FCe2WpoiijYDrupVttFJEJbg0cwTCTCJrBG2FDlPDL0KLA37gerRUThzrgDJ0jYalGr3TSicsWvN/Z8B/ibBHV/gAoYoRmVe24ozjkRGjOkVDvwM90xSGqKGbHzYa5IhvA9uiVtRwXiRHXM4HwLt5zShUkq3RMaDtSvFQZxpQoeu8z+qupnrC/+FWvnOjnsGCqyXBOBh4OSnEGdwr6XsEslwZoVjiAsqdsV4jvk3NLO8W9TYm6dJ+PD4f/kslEP9uu757uV45ORO7NgA2yCKgjAATgGZ6AJWgCDR9ADL+DVe/LevHfvY5ha8kY16+AbSjOfVbS1EQ==
    imp({y1, . . . , yn
    }) =
    1
    n
    n
    X
    i=1
    (yi mean(y))2
    AAACgHicdVFdaxQxFM2Mrda26vrxJpXgIsxiXWdKtUUolPbFxwrdtrBZh0w2sxuaZIbkjjCEvPsX+9g/4G8ws92CrXogcO695+benBS1FBbS9CqKH6ysPny09nh9Y/PJ02e95y/ObNUYxkeskpW5KKjlUmg+AgGSX9SGU1VIfl5cHnf18x/cWFHpU2hrPlF0pkUpGIWQyns/iaIwN8oJVfuEuDbPtsm0Arvd5pr4AT7ApDSUucw77YltVO7EQea/hwgTyUtI2lwEVs2WwhB6d3ur4lT7pB14/AF3uvf4fgUTI2ZzGOS9fjrc+ZQG4L9JNkwX6KMlTvLeddiTNYprYJJaO87SGiaOGhBMcr9OGstryi7pjI8D1VRxO3ELxzx+FzJTXFYmHA14kf2zw1FlbauKoOz2tfdrXfJftXED5f7ECV03wDW7GVQ2EkOFO/vxVBjOQLaBUGZE2BWzOQ2+QfikO1MK5YMntw/H/ydnO8Ps83D3227/8Gjpzhp6jd6iBGVoDx2ir+gEjRBDv6JX0Vb0Jo7jJP4YZzfSOFr2vER3EH/5DUesw0I=
    imp({y1, . . . , yn
    }) =
    1
    n
    n
    X
    i=1

    yi log
    yi
    mean(y)
    yi + mean(y)

    https://scikit-learn.org/stable/modules/tree.html#mathematical-formulation

    View full-size slide

  70. Impurity improvement for a good split
    If the 'impurity' decreases by splitting, then it's a 'good split'.
    The best split maximizes the amount of improvement in impurity before and after a split .
    AAAChHichVG7SgNBFD1ZNcb4SNRGsAmGiIWEiQ8UCwnaWOZhHhBD2F3HuGRf7G4CMfgD2ioWVgoW4gf4ATb+gEU+QSwj2Fh4s1kQDca7zM6ZM/fcOTNXMlXFdhhr+4Sh4RH/aGAsOD4xORUKT8/kbaNuyTwnG6phFSXR5qqi85yjOCovmhYXNUnlBam2290vNLhlK4a+7zRNXtbEqq4cKbLoEJW2K+EoizM3Iv0g4YEovEgZ4Ucc4BAGZNShgUOHQ1iFCJu+EhJgMIkro0WcRUhx9zlOESRtnbI4ZYjE1uhfpVXJY3Vad2varlqmU1QaFikjiLEXds867Jk9sFf2+Wetlluj66VJs9TTcrMSOpvLfvyr0mh2cPytGujZwRE2Xa8KeTddpnsLuadvnFx1sluZWGuR3bI38n/D2uyJbqA33uW7NM9cD/AjkRd6MWpQ4nc7+kF+JZ4gnF6LJne8VgUwjwUsUT82kMQeUshRfY5zXOBS8AvLwqqw3ksVfJ5mFj9C2P4CyLeP6g==
    impurity
    before the split
    AAAChHichVG7SgNBFD1ZNcb4SNRGsAmGiIWEiQ8UCwnaWOZhHhBD2F3HuGRf7G4CMfgD2ioWVgoW4gf4ATb+gEU+QSwj2Fh4s1kQDca7zM6ZM/fcOTNXMlXFdhhr+4Sh4RH/aGAsOD4xORUKT8/kbaNuyTwnG6phFSXR5qqi85yjOCovmhYXNUnlBam2290vNLhlK4a+7zRNXtbEqq4cKbLoEJW2K+EoizM3Iv0g4YEovEgZ4Ucc4BAGZNShgUOHQ1iFCJu+EhJgMIkro0WcRUhx9zlOESRtnbI4ZYjE1uhfpVXJY3Vad2varlqmU1QaFikjiLEXds867Jk9sFf2+Wetlluj66VJs9TTcrMSOpvLfvyr0mh2cPytGujZwRE2Xa8KeTddpnsLuadvnFx1sluZWGuR3bI38n/D2uyJbqA33uW7NM9cD/AjkRd6MWpQ4nc7+kF+JZ4gnF6LJne8VgUwjwUsUT82kMQeUshRfY5zXOBS8AvLwqqw3ksVfJ5mFj9C2P4CyLeP6g==
    AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhgUIwV0caShzwSJGR3HXDDvrK7kCDxB0xspbDSxML4AX6AjT9gwScYS0xsLLwsmxgl4t3Mzpkz99w5M1cyVcV2GOv5hInJqekZ/2xgbn5hMRhaWi7YRtOSeV42VMMqSaLNVUXneUdxVF4yLS5qksqLUuNgsF9scctWDP3IaZu8ool1XakpsugQldOr2WoowmLMjfAoiHsgAi/SRugRxziBARlNaODQ4RBWIcKmr4w4GEziKugQZxFS3H2OcwRI26QsThkisQ3612lV9lid1oOatquW6RSVhkXKMKLshd2zPntmD+yVff5Zq+PWGHhp0ywNtdysBi9Wcx//qjSaHZx+q8Z6dlDDrutVIe+mywxuIQ/1rbNuP7eXjXY22C17I/83rMee6AZ6612+y/Ds9Rg/EnmhF6MGxX+3YxQUtmLxnVgik4ik9r1W+bGGdWxSP5JI4RBp5Kl+HZe4QlfwCzFhW0gOUwWfp1nBjxBSX4V+kLM=
    nR
    AAAChnichVG7TgJBFD2sL8QHqI2JDZFgrMhgUIwV0cbCgoc8EiRkdx1xw76yu5Ag8QdMbKWw0sTC+AF+gI0/YMEnGEtMbCy8LJsYJeLdzM6ZM/fcOTNXMlXFdhjr+oSx8YnJKf90YGZ2bj4YWlgs2EbDknleNlTDKkmizVVF53lHcVReMi0uapLKi1J9r79fbHLLVgz90GmZvKKJNV05UWTRISqnVw+qoQiLMTfCwyDugQi8SBuhRxzhGAZkNKCBQ4dDWIUIm74y4mAwiaugTZxFSHH3Oc4RIG2DsjhliMTW6V+jVdljdVr3a9quWqZTVBoWKcOIshd2z3rsmT2wV/b5Z622W6PvpUWzNNBysxq8WM59/KvSaHZw+q0a6dnBCbZdrwp5N12mfwt5oG+edXq5nWy0vcZu2Rv5v2Fd9kQ30Jvv8l2GZ69H+JHIC70YNSj+ux3DoLARi2/FEplEJLXrtcqPFaxinfqRRAr7SCNP9Wu4xBU6gl+ICZtCcpAq+DzNEn6EkPoCeL6QrQ==
    nL
    AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqqZUCIRZhTgT7QdQFIbiR0AOPOMQRdMioowYODTZhFSIs+gqIgsEgrogWcSYhxdnnOIWPtHXK4pQhElulf4VWBZfVaN2taTlqmU5RaZikDCLMXtg967Bn9sBe2eeftVpOja6XJs1ST8uNkv9sPv3xr6pGs43jb9VAzzbK2HK8KuTdcJjuLeSevnFy1Ulvp8KtJXbL3sj/DWuzJ7qB1niX75I8dT3Aj0Re6MWoQdHf7egH2dVIdCMSS8ZC8V23VV4sYBHL1I9NxLGPBDJUn+McF7gURoUVYU1Y76UKHlczhx8h7HwBwQOP7g==
    n
    AAAChHichVG7SgNBFD1ZNcZ31EawCQbFQsJE4wMLCdpYWJjERCGK7K6TOLjZXXYngRj8AW0VCysFC/ED/AAbf8AinyCWEWwsvNksiAb1LrNz5sw9d87M1WxDuJKxekDp6OwKdod6evv6BwaHwsMjOdcqOzrP6pZhOTua6nJDmDwrhTT4ju1wtaQZfFs7Wmvub1e44wrL3JJVm++V1KIpCkJXJVGpjf1wlMWYF5F2EPdBFH5sWuEH7OIAFnSUUQKHCUnYgAqXvjziYLCJ20ONOIeQ8PY5TtBL2jJlccpQiT2if5FWeZ81ad2s6XpqnU4xaDikjGCSPbM71mBP7J69sI9fa9W8Gk0vVZq1lpbb+0OnY5n3f1UlmiUOv1R/epYoYMnzKsi77THNW+gtfeX4spFZTk/WptgNeyX/16zOHukGZuVNv03x9NUffjTyQi9GDYr/bEc7yM3G4guxRCoRTa76rQphHBOYpn4sIol1bCJL9TnOcI4LJajMKHPKfCtVCfiaUXwLZeUTeMOPzA==
    L AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqmSoFQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkChYOP0g==
    R
    AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqmS8FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCkkOP2A==
    X
    samples
    yes no
    AAAC93ichVFPSxtBFH9ZW01t1aiXgpelwZJFDBMRFaEg1YMHDxobDbgSZsdJMjj7h9lJIF3yBfwCHjwUCxWlJ09+AC9+gYJ+BOnRQi+l9GWzRVTUt8zOm9/8fm9+M88JpAg1IVcpo+fFy96+9Kv+128GBocywyMbod9QjJeYL31VdmjIpfB4SQsteTlQnLqO5JvO7mJnf7PJVSh875NuBXzbpTVPVAWjGqFKRtlLXGqaCy3zg2m7VNeVGwk3aOfKljlp2pJXdc6uKsoir7LSjry2zXZ8fZe6YpkT5n9S8RFS0bKVqNW1VclkSZ7EYT5MCkmShSRW/cwZ2LADPjBogAscPNCYS6AQ4rcFBSAQILYNEWIKMxHvc2hDP2obyOLIoIju4r+Gq60E9XDdqRnGaoanSBwKlSaMkx/khNyQC/KdXJM/j9aK4hodLy2cna6WB5Whvbfrv59VuThrqN+qnvSsoQpzsVeB3oMY6dyCdfXNz/s36/PF8eg9+Up+ov9DckXO8QZe8xf7tsaLB0/4cdALvhg2qHC/HQ+Tjal8YSY/vTadXfiYtCoNY/AOctiPWViAZViFEta/hL+pvlTaaBlfjCPjuEs1UolmFO6EcfoPe5i70A==
    (s) = imp(X)
    ⇣nL
    n
    · imp(L) +
    nR
    n
    · imp(R)

    samples samples
    Split
    impurity
    after the split
    (the weighted mean of two impurities)
    AAACunichVG7SgNBFD2u73fURrBZDIoihIn4QixEGwsLjUYDiSy7m4kO2Re7k4AGf8AfsLBSsBA/wFpsRGsLP0EsFWwsvNksiK94l905c+49d8/MNTxLBJKxxwalsam5pbWtvaOzq7unN9bXvxW4Jd/kadO1XD9j6AG3hMPTUkiLZzyf67Zh8W2juFzNb5e5HwjX2ZT7Ht+x9V1HFISpS6K02LSjrao5M+9KNWfrcs+3K8L2DsdWx9UJ1dFSv+VS41oszhIsDPUnSEYgjijW3NgVcsjDhYkSbHA4kIQt6AjoySIJBo+4HVSI8wmJMM9xiA7SlqiKU4VObJG+u7TLRqxD+2rPIFSb9BeLXp+UKkbYA7tgL+yWXbIn9v5nr0rYo+pln1ajpuWe1ns0uPH2r8qmVWLvU1XXs0QBc6FXQd69kKmewqzpywfHLxvzqZHKKDtjz+T/lD2yGzqBU341z9d56qSOH4O80I3RgJLfx/ETbE0mkjOJqfWp+OJSNKo2DGEYYzSPWSxiBWtIU/8TXOMO98qCYihCKdZKlYZIM4AvocgPt/Okbg==
    nL
    · imp(L) + nR
    · imp(R)
    AAACE3icdVDLSgMxFM34rPU16rKbYBHaTZkp9bERirpwWcE+oFNKJs20oclkSDLiMHThR/gNbnXtTtz6AS79E9OHYH0cuHA4517uvcePGFXacd6thcWl5ZXVzFp2fWNza9ve2W0oEUtM6lgwIVs+UoTRkNQ11Yy0IkkQ9xlp+sPzsd+8IVJREV7rJCIdjvohDShG2khdO+dxpAeSp5EUt8mo4F0QplFBFYvwtGvnnVL50DGAv4lbcibIgxlqXfvD6wkccxJqzJBSbdeJdCdFUlPMyCjrxYpECA9Rn7QNDREnqpNOnhjBA6P0YCCkqVDDifp9IkVcqYT7pnN8svrpjcW/vHasg5NOSsMo1iTE00VBzKAWcJwI7FFJsGaJIQhLam6FeIAkwtrkNrfF5yOTydfj8H/SKJfco1LlqpKvns3SyYAc2AcF4IJjUAWXoAbqAIM78AAewZN1bz1bL9brtHXBms3sgTlYb588FJ5E
    proxy( (s)) =
    Proxy minimization
    No matter what the split is, the values of and remain unchanged.
    So we need to minimize the following proxy criterion.
    AAACB3icbVDLSsNAFL2pr1ofjbp0M1iEuimJ+FoW3bisYB/QhjKZTtqhM0mYmQgl9AP8Bre6didu/QyX/omTNgvbeuDC4Zx7uYfjx5wp7TjfVmFtfWNzq7hd2tnd2y/bB4ctFSWS0CaJeCQ7PlaUs5A2NdOcdmJJsfA5bfvju8xvP1GpWBQ+6klMPYGHIQsYwdpIfbvcE1iPpEiZiKfVzlnfrjg1Zwa0StycVCBHo2//9AYRSQQNNeFYqa7rxNpLsdSMcDot9RJFY0zGeEi7hoZYUOWls+BTdGqUAQoiaSbUaKb+vUixUGoifLOZxVTLXib+53UTHdx4KQvjRNOQzB8FCUc6QlkLaMAkJZpPDMFEMpMVkRGWmGjT1cIXX0xNJ+5yA6ukdV5zr2qXDxeV+m3eThGO4QSq4MI11OEeGtAEAgm8wCu8Wc/Wu/Vhfc5XC1Z+cwQLsL5+AZ1Wma0=
    imp(X) AAAB93icbVDLSgNBEOyNrxhfUY9eFoPgKeyKr2PQi8cEzAOSJcxOepMhM7PLzKywhHyBVz17E69+jkf/xEmyBxMtaCiquunuChPOtPG8L6ewtr6xuVXcLu3s7u0flA+PWjpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2O72d++wmVZrF8NFmCgSBDySJGibFSQ/bLFa/qzeH+JX5OKpCj3i9/9wYxTQVKQznRuut7iQkmRBlGOU5LvVRjQuiYDLFrqSQCdTCZHzp1z6wycKNY2ZLGnau/JyZEaJ2J0HYKYkZ61ZuJ/3nd1ES3wYTJJDUo6WJRlHLXxO7sa3fAFFLDM0sIVcze6tIRUYQam83SllBMbSb+agJ/Seui6l9XrxqXldpdnk4RTuAUzsGHG6jBA9ShCRQQnuEFXp3MeXPenY9Fa8HJZ45hCc7nD37Lk5I=
    n

    View full-size slide

  71. Example 1: Calculating impurity improvement
    AAADIXichZHBTttAEIbHhhYaSgn0UqkXqxEVBxTNOo4TIyGh9tIDBwgNIIUost0ltXBsy3YigZUX6AtwaC8FcUB9AB6AS3urkDggnqDqkUpcOHTWcVVBVLore3/Pzjf7r8cKXCeKES8keWT0wcOx8Ue5iceTT6by0zPrkd8NbV63fdcPNy0z4q7j8XrsxC7fDEJudiyXb1g7r8X+Ro+HkeN7b+PdgDc7Zttzth3bjCnUyn9KttIijbBtNRMsYjrmsagaVZ0JUSkbpYrRV7y+sqgw2lNydxijStOg1DIiKzMSWFJ1o0RMa1lQJTY/xGCVqZo4R9eYJs4pG4xVWF8hqCYg3ci18oU/jpRhwTJRgGys+PkT2IJ34IMNXegABw9i0i6YENFsAAOEgGJNSCgWknLSfQ59yBHbpSxOGSZFd+jdpq9GFvXoW9SMUtqmU1x6QiIVmMVzPMYr/Ipf8Afe/LNWktYQXnZptQYsD1pTH56tXf+X6tAaw/u/1L2eY9iGaurVIe9BGhG3sAd8b2//am2hNpu8xAP8Sf4/4wWe0g283i/7aJXXPt7jxyIv9MeoQexuO4bFulpkelFb1QpLr7JWjcNzeAFz1I8KLMEbWIE62JIszUlMUuVD+Zv8XT4bpMpSxjyFW0O+/A3hp7rE
    n = 100, nL = 31, nR = 69
    AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqmS8FQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkCkkOP2A==
    X
    AAAChHichVG7SgNBFD1ZNcZ31EawCQbFQsJE4wMLCdpYWJjERCGK7K6TOLjZXXYngRj8AW0VCysFC/ED/AAbf8AinyCWEWwsvNksiAb1LrNz5sw9d87M1WxDuJKxekDp6OwKdod6evv6BwaHwsMjOdcqOzrP6pZhOTua6nJDmDwrhTT4ju1wtaQZfFs7Wmvub1e44wrL3JJVm++V1KIpCkJXJVGpjf1wlMWYF5F2EPdBFH5sWuEH7OIAFnSUUQKHCUnYgAqXvjziYLCJ20ONOIeQ8PY5TtBL2jJlccpQiT2if5FWeZ81ad2s6XpqnU4xaDikjGCSPbM71mBP7J69sI9fa9W8Gk0vVZq1lpbb+0OnY5n3f1UlmiUOv1R/epYoYMnzKsi77THNW+gtfeX4spFZTk/WptgNeyX/16zOHukGZuVNv03x9NUffjTyQi9GDYr/bEc7yM3G4guxRCoRTa76rQphHBOYpn4sIol1bCJL9TnOcI4LJajMKHPKfCtVCfiaUXwLZeUTeMOPzA==
    L AAAChHichVHLTsJAFD3UF+ID1I2JGyLBuDBkUHzEhSG6cclDHgkS0tYBG0rbtIUEiT+gW40LV5q4MH6AH+DGH3DBJxiXmLhx4aU0MUrE20znzJl77pyZKxmqYtmMtT3C0PDI6Jh33DcxOTXtD8zMZi29bso8I+uqbuYl0eKqovGMrdgqzxsmF2uSynNSda+7n2tw01J07cBuGrxYEyuaUlZk0SYqmSoFQizCnAj2g6gLQnAjoQcecYgj6JBRRw0cGmzCKkRY9BUQBYNBXBEt4kxCirPPcQofaeuUxSlDJLZK/wqtCi6r0bpb03LUMp2i0jBJGUSYvbB71mHP7IG9ss8/a7WcGl0vTZqlnpYbJf/ZfPrjX1WNZhvH36qBnm2UseV4Vci74TDdW8g9fePkqpPeToVbS+yWvZH/G9ZmT3QDrfEu3yV56nqAH4m80ItRg6K/29EPsquR6EYkloyF4rtuq7xYwCKWqR+biGMfCWSoPsc5LnApjAorwpqw3ksVPK5mDj9C2PkChYOP0g==
    R
    31 samples (o 31, o 0, o 0)
    gini = 0.0
    69 samples (o 0, o 36, o 33)
    gini = 0.499
    100 samples (value o 31, o 36, o 33)
    gini = 0.665
    AAAC53ichVHLShxBFL22mqgxcaIbwU3jYBghGW7P+BohIHHjIgtfo8L0ZOhua2YK+0V1zYA2/QNu3EXBlQEX4gfkA9z4A1n4CSJkY0AXWXi7p0ESiblFVZ06dc/tU31N3+aBRLzqUrp7el+87OsfeDX4+s1Q5u3wRuC1hMXKlmd7Yss0AmZzl5Ullzbb8gUzHNNmm+bOYny/2WYi4J67Lnd9VnWMhsvr3DIkUbWMqTuGbAon5I4f5UI9qVgRDbMaYr40R6P0HvPTiNq0RgCLhZlSMVI/R5PqR1VTP6i6zeoyp9eFYYVFLYqnLnijKSe/FCiFJLVMltYk1KdAS0EW0lj2Mt9Bh23wwIIWOMDABUnYBgMCGhXQAMEnrgohcYIQT+4ZRDBA2hZlMcowiN2htUGnSsq6dI5rBonaoq/YNAUpVZjAH3iGt3iJ53iNv/9ZK0xqxF52aTc7WubXhvZH1+7+q3Jol9B8VD3rWUId5hKvnLz7CRO/wuro23uHt2vzqxPhO/yGN+T/BK/wgl7gtn9Zpyts9fgZPyZ5oT9GDdL+bsdTsFHIazP5qZWp7MKntFV9MAbjkKN+zMICLMEylKn+JfyEO7hXuHKgfFWOOqlKV6oZgT9COXkAqJyxGg==
    imp(L) = 1

    31
    31
    ◆2
    = 0.0
    AAADOnichVE7b9NQFD42rxIeDbAgdbGIilIB1nHSOg1SpQILgqUP0kaK08g2N8lV/dL1TaRi+Q/wBxiYQGJAzIiFjYU/wNCVrerYSiwMHDtGqK1oj2Wf7373fMffvceJPB5LxF1FPXf+wsVLU5dLV65euz5dvnFzIw5HwmUtN/RC0XbsmHk8YC3JpcfakWC273hs09l+ku1vjpmIeRi8kDsR6/r2IOB97tqSqF75s+Xbcij8hPtRWk2svGNHDJxugjrmcR/1WnPRNDLQWGjWG81Ua6dz2pJmaA80y2N9WS1NktUXtpvUjTQxEFNL8MFQzm3VtHva0QLzrIL68YJSAZdQN82FXrny1512EhgFqEARK2H5C1jwEkJwYQQ+MAhAEvbAhpieDhiAEBHXhYQ4QYjn+wxSKJF2RFWMKmxit+k7oFWnYANaZz3jXO3SXzx6BSk1mMUf+BEP8Dt+wj38/d9eSd4j87JD2ZloWdSbfn17/deZKp+yhOE/1ameJfRhMffKyXuUM9kp3Il+/OrNwfrDtdnkLr7HffL/DnfxG50gGB+6H1bZ2ttT/DjkhW6MBmQcH8dJsFHTDVOfX52vLD8uRjUFM3AHqjSPBizDU1iBFrjKjPJIeaY8V7+qP9U9dX9SqiqF5hYcCfXwD3gkyyY=
    imp(X) = 1

    31
    100
    ◆2
    +

    36
    100
    ◆2
    +

    33
    100
    ◆2
    !
    = 0.665
    AAADGHichVFNaxNRFL0Zv2r8aNSN4MKHoZKgDnfSMe0IhaIbl21q2kImhpnxJXl0vnjzEqjD/AG3LgRdKbgQf4BbwYV2W3DRnyAuK4jgwpvJgLTF+oaZe96599w571039kWiEPdK2omTp06fmTlbPnf+wsXZyqXL60k0kh5ve5EfyU3XSbgvQt5WQvl8M5bcCVyfb7hbDyb5jTGXiYjCR2o75t3AGYSiLzxHEdWrPLcDRw1lkIogzmqpnXfsyIHbTVG3cNFomLdRbzRNw0QCdy3DWDAy1srqbIkxg91hts/7qlaeBrsvHS+db2Zp08psKQZDVX/cYLfYwfz8oXy5gEuom5bVq1RRx3yxo8AoQBWKtRJVPoANTyACD0YQAIcQFGEfHEjo6YABCDFxXUiJk4REnueQQZm0I6riVOEQu0XfAe06BRvSftIzydUe/cWnV5KSwRx+xXe4j1/wPX7D3//sleY9Jl62KbpTLY97s8+urv38ryqgqGD4V3WsZwV9WMy9CvIe58zkFN5UP376Yn/tXmsuvYlv8Dv5f417+IlOEI5/eG9XeevVMX5c8kI3RgMyDo/jKFhv6EZTN1fN6vL9YlQzcA1uQI3msQDL8BBWoE39f5Wul2qluvZS+6h91nampVqp0FyBA0vb/QOas7/3
    imp(R) = 1

    36
    69
    ◆2
    +

    33
    69
    ◆2
    !
    = 0.499
    (0.665 → 0.344)
    Impurity Improvement = 0.321
    AAACH3icbVDLSgMxFM34rPVVdekmWERdWGYKPhAKRRe6VLAP6IxDJs20oUlmSDJCGeYHBBd+gt/gVtfuRJcu/RExfSx8HbhwOOde7r0niBlV2rbfrYnJqemZ2dxcfn5hcWm5sLJaV1EiManhiEWyGSBFGBWkpqlmpBlLgnjASCPonQz8xjWRikbiUvdj4nHUETSkGGkj+YUtlyPdlTw9pYJm280deFSBzi50VcL9tFdxMhj7vauyXyjaJXsI+Jc4Y1Ksnt6cfLp3b+d+4cNtRzjhRGjMkFItx461lyKpKWYky7uJIjHCPdQhLUMF4kR56fChDG4apQ3DSJoSGg7V7xMp4kr1eWA6B+er395A/M9rJTo89FIq4kQTgUeLwoRBHcFBOrBNJcGa9Q1BWFJzK8RdJBHWJsMfWwKemUyc3wn8JfVyydkv7V2YcI7BCDmwDjbANnDAAaiCM3AOagCDW/AAHsGTdW89Wy/W66h1whrPrIEfsN6/AA6Ppeo=
    Gini(X) := 1
    X
    k=1
    p2
    k
    The Gini impurity
    (31/100 * 0.0 + 69/100 * 0.499) = 0.344
    AAAEH3icrVHLbtNAFL2xeZTyaAobJDYWUatEQJhJnIeRkCpgwaKLNiVtpDiKbHeSWPVL9iRSseYH+AGEWBWJBeID+AAWsEJsWOQTEMsisWHBtROESEnYdEb2nLlzzz1n5pqBY0eckHFGks+cPXd+6cLyxUuXr6xkV6/uRv4wtFjT8h0/bJlGxBzbY01uc4e1gpAZrumwPfPgYXK+N2JhZPveE34YsI5r9D27Z1sGx1B3NfNCf8QcbuSjgnJf0V2DD0I3tt1A5GM9Ld8O+2YnJkWSjtukWNLqVZqAWkUr1zShtERBuaPoDuvxvN4LDSueoWp1nBoyKoTQCkVAyqWqVhaK190UYjZ9vpInhG7t+3yh0flqm2j0lvJvi6ROS2qiWVWpmmhWNEprVCjosXH6HufKNURBD+3+gBe62dxvHeUkoFOQg+nY8rPvQId98MGCIbjAwAOO2AEDIpxtoEAgwFgHYoyFiOz0nIGAZeQOMYthhoHRA/z3cdeeRj3cJzWjlG2hioNfiEwF1sgX8oYck4/kLflKfs6tFac1Ei+HuJoTLgu6K8+u7/z4L8vFlcPgD2uhZw49qKdebfQepJHkFtaEP3r6/HjnXmMtXievyDf0f0TG5D3ewBt9t15vs8bLBX5M9IIvhg2is+04CXZLRVotqttqbuPBtFVLcANuQh77UYMNeAxb0ARLAmlduisR+Uj+IH+SP09SpcyUcw3+GvL4F7rTDME=
    (s) = imp(X)
    ⇣nL
    n
    · imp(L) +
    nR
    n
    · imp(R)

    0.665
    Petal.Length
    o
    o
    o
    o
    o
    o
    o
    o o
    o
    o
    oo o
    o
    o
    o
    o o
    o
    o
    o
    o
    o
    o
    o
    o
    o o
    o
    o
    o
    o
    o
    o
    o o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o o
    o o
    o
    o
    o
    o o o
    o o o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    o
    oo
    o o
    o
    o
    o
    o
    o
    o
    o o
    o
    o
    o
    o
    o
    oo o
    1 2 3 4 5 6
    setosa
    versicolor
    virginica

    View full-size slide

  72. Example 2: Changes in proxy for Iris dataset
    proxy of the impurity improvement
    impurity improvement

    View full-size slide

  73. Class probabilities
    Text text
    Random Forest
    Gaussian Process Classifier
    Logistic Regression
    Output a probability between and
    (predict_proba)
    P(class=red)
    P(class=blue)
    = 1 - P(class=red)
    Class probability in classification

    View full-size slide

  74. How to get final predicted values at leaf nodes
    Decision tree (class probability) = Histogram approximation by data-dependent partitions
    P(class=red) P(class=blue)
    = 1 - P(class=red)
    predict_proba(x)
    red blue
    8/27 = 0.296
    19/27 = 0.704
    0.0
    1.0
    8/27
    19/27
    19
    8
    35 2

    View full-size slide

  75. How to get final predicted values at leaf nodes
    predict_proba(x)
    /
    /
    /
    /
    Class probabilities
    = [ . , . , . . . ]
    The distribution of y for the training samples in the region where x fell
    Decision tree (class probability) = Histogram approximation by data-dependent partitions

    View full-size slide

  76. How to get final predicted values at leaf nodes
    The sample average
    of the local region
    • A very conservative prediction (constant prediction = histogram approximation)
    based on very sloppy/rough partitions
    • In grid partitions, we have many regions that do not contain sample points. But, with data-
    dependent splitting, each region always contains sample points even in high dimensions!
    Regression tree = Histogram approximation by data-dependent partitions

    View full-size slide

  77. How to get final predicted values at leaf nodes
    ≤ >
    Return the average y of the training samples in
    the region where x fell as the predicted value
    Regression tree = Histogram approximation by data-dependent partitions

    View full-size slide

  78. Stopping condition: When should the recursion be stopped?
    Stopping condition for recursion
    • When the depth of the tree reaches the specified limit (max_depth).
    • When the number of leaf nodes reaches the specified limit (max_leaf_nodes).
    • When the improvement in impurity after splitting is sufficiently small (min_impurity_decrease).
    • When the impurity of the region is sufficiently low (min_impurity_split).
    • When the number of samples in the region is sufficiently small (min_samples_split).
    • When the number of samples in the leaf node reaches the lower limit (min_samples_leaf).
    • When the sample size ratio of the node after splitting reaches the lower limit
    (min_weight_fraction_leaf).
    Or stop to prevent the partitions from becoming too fine or unbalanced
    Don't stop (fully grown trees)
    ➡ Recursion continues until the training error becomes zero.
    In the worst case, it stops at a leaf node with a sample size of 1

    View full-size slide

  79. Stop based on the size of the tree.
    • max_depth
    • max_leaf_nodes
    • min_impurity_decrease
    • min_impurity_split
    • min_samples_leaf
    • min_samples_split
    • min_weight_fraction_leaf
    AAAClnichVHLSsNAFD2N73fUjeAmWCquZCKC4kJEEV22am2hLSGJow3Ny2Ra0eIP+AMuxIWCivgBfoAbf8CFnyAuFdy48DYNiIp6w2TOnLnnzpm5hm9boWDsMSG1tLa1d3R2dff09vUPyINDm6FXDUyeNT3bC/KGHnLbcnlWWMLmeT/gumPYPGdUlhr7uRoPQstzN8S+z0uOvuNa25apC6I0eTCvqUrR5rtKUZS50DVVk5NskkWh/ARqDJKII+3JtyhiCx5MVOGAw4UgbENHSF8BKhh84kqoExcQsqJ9jkN0k7ZKWZwydGIr9N+hVSFmXVo3aoaR2qRTbBoBKRWk2AO7Zi/snt2wJ/b+a616VKPhZZ9mo6nlvjZwNLL+9q/KoVmg/Kn607PANmYjrxZ59yOmcQuzqa8dHL+sz62l6uPsnD2T/zP2yO7oBm7t1bzI8LWTP/wY5IVejBqkfm/HT7A5NakSzkwnFxbjVnViFGOYoH7MYAGrSCNL9fdwiktcSSPSvLQsrTRTpUSsGcaXkNIfLpKWEA==
    AAAClnichVHLSsNAFD3GV62vVjeCm2JRXMmkCooLKYrosg+rBSshiWMbTJOYTCu1+AP+gAtxoaAifoAf4MYfcOEniEsFNy68SQOiot4wmTNn7rlzZq7mmIYnGHtsk9o7Oru6Iz3R3r7+gcFYfGjds2uuzgu6bdpuUVM9bhoWLwhDmLzouFytaibf0HaX/P2NOnc9w7bWRMPhW1W1bBk7hq4KopRYvKikEiWT7yVKosKFqkwrsSSbYkEkfgI5BEmEkbFjtyhhGzZ01FAFhwVB2IQKj75NyGBwiNtCkziXkBHscxwiStoaZXHKUIndpX+ZVpsha9Har+kFap1OMWm4pExgnD2wa/bC7tkNe2Lvv9ZqBjV8Lw2atZaWO8rg0Uj+7V9VlWaByqfqT88CO5gLvBrk3QkY/xZ6S18/OH7Jz+fGmxPsnD2T/zP2yO7oBlb9Vb/I8tzJH3408kIvRg2Sv7fjJ1hPTcmEszPJ9GLYqghGMYZJ6scs0lhFBgWqv49TXOJKGpEWpGVppZUqtYWaYXwJKfMBNQCWEw==
    AAAClnichVHLSsNAFD2Nr1pfrW4KbopFcSXTIigupCiiyz6sFqyEJE7bYF4m00ot/oA/4EJcKKiIH+AHuPEHXPgJ4lLBjQtv04CoqDdM5syZe+6cmas6hu4Jxh5DUld3T29fuD8yMDg0PBKNjW54dt3VeFGzDdstqYrHDd3iRaELg5cclyumavBNdXe5vb/Z4K6n29a6aDp821Sqll7RNUUQJUdjJTmdKBt8L1EWNS4UOS1Hk2yG+ZH4CVIBSCKIrB29RRk7sKGhDhMcFgRhAwo8+raQAoND3DZaxLmEdH+f4xAR0tYpi1OGQuwu/au02gpYi9btmp6v1ugUg4ZLygQm2QO7Zi/snt2wJ/b+a62WX6PtpUmz2tFyRx45ihfe/lWZNAvUPlV/ehaoYN73qpN3x2fat9A6+sbB8UthIT/ZmmLn7Jn8n7FHdkc3sBqv2kWO50/+8KOSF3oxalDqezt+go30TIpwbjaZWQpaFcY4JjBN/ZhDBmvIokj193GKS1xJcWlRWpFWO6lSKNCM4UtI2Q8y4JYS
    AAAClnichVHLSsNAFD3GV323uim4KZaKK5lIQXEhRRFdtmproS0hiaOGpklMppVa/AF/wIW4UFARP8APcOMPuOgniEsFNy68TQOiRb1hMmfO3HPnzFzNMQ1PMNbskrp7evv6QwODQ8Mjo2PhyHjOs6uuzrO6bdpuXlM9bhoWzwpDmDzvuFytaCbf1sorrf3tGnc9w7a2RN3hpYq6Zxm7hq4KopRwJK/IsaLJD2JFsc+FqiSVcJzNMj9inUAOQBxBpO3wPYrYgQ0dVVTAYUEQNqHCo68AGQwOcSU0iHMJGf4+xzEGSVulLE4ZKrFl+u/RqhCwFq1bNT1frdMpJg2XlDEk2BO7Za/skd2xZ/bxa62GX6PlpU6z1tZyRxk7iW6+/6uq0Cyw/6X607PALhZ8rwZ5d3ymdQu9ra8dnb5uLm4kGtPskr2Q/wvWZA90A6v2pl9l+MbZH3408kIvRg2Sf7ajE+TmZmXCmWQ8tRy0KoRJTGGG+jGPFNaRRpbqH+Ic17iRotKStCqttVOlrkAzgW8hpT8BNPKWEw==
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhVmRYlYBW0s8zAPiCHsrpO4ZF/sbgIx+AOCrSmsFCzED/ADbPwBi3yCWEawsfBmsyAajHeZnTNn7rlzZq5saarjMtYLCBOTU9MzwdnQ3PzCYjiytFxwzKat8LxiaqZdkiWHa6rB867qarxk2VzSZY0X5cbBYL/Y4rajmsaR27Z4RZfqhlpTFcklKqdVxWokxhLMi+goEH0Qgx9pM/KIY5zAhIImdHAYcAlrkODQV4YIBou4CjrE2YRUb5/jHCHSNimLU4ZEbIP+dVqVfdag9aCm46kVOkWjYZMyijh7Yfesz57ZA3tln3/W6ng1Bl7aNMtDLbeq4YvV3Me/Kp1mF6ffqrGeXdSw63lVybvlMYNbKEN966zbz+1l450NdsveyP8N67EnuoHRelfuMjx7PcaPTF7oxahB4u92jILCVkIknNmOpfb9VgWxhnVsUj+SSOEQaeSpfh2XuEJXCAoJYUdIDlOFgK9ZwY8QUl84LpCH
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmQYlYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKqUSScG4rltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBOk6QiA== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmPohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYLCGPjE5NTwenQzOzcfDiysFi0jaYl84JsqIZVlkSbq4rOC47iqLxsWlzUJJWXpMa+u19qcctWDP3QaZu8qol1XTlRZNEhKq/WNmuRGEswL6LDIOmDGPzIGJFHHOEYBmQ0oYFDh0NYhQibvgqSYDCJq6JDnEVI8fY5zhEibZOyOGWIxDboX6dVxWd1Wrs1bU8t0ykqDYuUUcTZC7tnffbMHtgr+/yzVser4Xpp0ywNtNyshS+W8x//qjSaHZx+q0Z6dnCCHc+rQt5Nj3FvIQ/0rbNuP7+bi3fW2C17I/83rMee6AZ6612+y/Lc9Qg/EnmhF6MGJX+3YxgUNxJJwtmtWHrPb1UQK1jFOvUjhTQOkEGB6tdxiSt0haCQELaF1CBVCPiaJfwIIf0FPG6QiQ==
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmEolYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4VwqltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBPo6Qig== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmYohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4dx2LLPntyqINaxjk/qRRgYHyKJI9Ru4xBV6QlBICCkhPUwVAr5mBT9CyHwBQK6Qiw==
    AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmjaAEi6AIlnmYByQh7K6TZMi+2J0EYvAHLG0sYqNgIX6AH2DjD1jkE8Qygo2FdzcLosF4l9k5c+aeO2fmyqbKbE7IwCdMTc/MzvnnAwuLwaVQeHmlYBttS6F5xVANqyRLNlWZTvOccZWWTItKmqzSotw6dPaLHWrZzNBPeNekVU1q6KzOFIkjVarwJuVSLVELR0mcuBEZB6IHouBF2gg/QgVOwQAF2qABBR04YhUksPErgwgETOSq0EPOQsTcfQrnEEBtG7MoZkjItvDfwFXZY3VcOzVtV63gKSoOC5URiJEXck+G5Jk8kFfy+WetnlvD8dLFWR5pqVkLXazlPv5VaThzaH6rJnrmUIc91ytD76bLOLdQRvrO2dUwl8zGepvklryh/xsyIE94A73zrtxlaLY/wY+MXvDFsEHi73aMg8J2XESc2YmmDrxW+WEdNmAL+7ELKTiGNOTdPlxCH66FoJAQksL+KFXweZpV+BHC0RdPP5LB
    AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhXuqqAEi6AIlnmYByQh7K4Ts2Rf7E4CMfgDljYWsVGwED/AD7DxByzyCWIZwcbCm82CaDDeZXbOnLnnzpm5sqWpDkfs+YSJyanpGf9sYG4+uBAKLy7lHbNpKyynmJppF2XJYZpqsBxXucaKls0kXdZYQW4cDPYLLWY7qmkc87bFKrp0aqg1VZE4UcUyrzMuVcVqOIpxdCMyCkQPRMGLlBl+hDKcgAkKNEEHBgZwwhpI4NBXAhEQLOIq0CHOJqS6+wzOIUDaJmUxypCIbdD/lFYljzVoPajpuGqFTtFo2KSMQAxf8B77+IwP+Iqff9bquDUGXto0y0Mts6qhi5Xsx78qnWYO9W/VWM8carDrelXJu+Uyg1soQ33r7KqfTWRinXW8xTfyf4M9fKIbGK135S7NMt0xfmTyQi9GDRJ/t2MU5DfjIuH0djS577XKD6uwBhvUjx1IwhGkIOf24RK6cC0EhS0hIewNUwWfp1mGHyEcfgFK/5K/
    AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmo6AEi6AIlnmYByQh7K6TZMi+2J0EYvAHLG0sYqNgIX6AH2DjD1jkE8Qygo2FdzcLosF4l9k5c+aeO2fmyqbKbE7IwCdMTc/MzvnnAwuLwaVQeHmlYBttS6F5xVANqyRLNlWZTvOccZWWTItKmqzSotw6dPaLHWrZzNBPeNekVU1q6KzOFIkjVarwJuVSLVELR0mcuBEZB6IHouBF2gg/QgVOwQAF2qABBR04YhUksPErgwgETOSq0EPOQsTcfQrnEEBtG7MoZkjItvDfwFXZY3VcOzVtV63gKSoOC5URiJEXck+G5Jk8kFfy+WetnlvD8dLFWR5pqVkLXazlPv5VaThzaH6rJnrmUIc91ytD76bLOLdQRvrO2dUwl8zGepvklryh/xsyIE94A73zrtxlaLY/wY+MXvDFsEHi73aMg0IiLiLO7ERTB16r/LAOG7CF/diFFBxDGvJuHy6hD9dCUNgWksL+KFXweZpV+BHC0RdNH5LA
    AAACi3ichVG7SgNBFL1ZXzEaE7URbIIhYhVmNaAEi6AIlnmYByQh7K6TZMi+2J0EYvAHLG0sYqNgIX6AH2DjD1jkE8Qygo2FdzcLosF4l9k5c+aeO2fmyqbKbE7IwCdMTc/MzvnnAwuLwaVQeHmlYBttS6F5xVANqyRLNlWZTvOccZWWTItKmqzSotw6dPaLHWrZzNBPeNekVU1q6KzOFIkjVarwJuVSLVELR0mcuBEZB6IHouBF2gg/QgVOwQAF2qABBR04YhUksPErgwgETOSq0EPOQsTcfQrnEEBtG7MoZkjItvDfwFXZY3VcOzVtV63gKSoOC5URiJEXck+G5Jk8kFfy+WetnlvD8dLFWR5pqVkLXazlPv5VaThzaH6rJnrmUIc91ytD76bLOLdQRvrO2dUwl8zGepvklryh/xsyIE94A73zrtxlaLY/wY+MXvDFsEHi73aMg8J2XEScSURTB16r/LAOG7CF/diFFBxDGvJuHy6hD9dCUNgRksL+KFXweZpV+BHC0RdRX5LC
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhVmRYlYBW0s8zAPiCHsrpO4ZF/sbgIx+AOCrSmsFCzED/ADbPwBi3yCWEawsfBmsyAajHeZnTNn7rlzZq5saarjMtYLCBOTU9MzwdnQ3PzCYjiytFxwzKat8LxiaqZdkiWHa6rB867qarxk2VzSZY0X5cbBYL/Y4rajmsaR27Z4RZfqhlpTFcklKqdVxWokxhLMi+goEH0Qgx9pM/KIY5zAhIImdHAYcAlrkODQV4YIBou4CjrE2YRUb5/jHCHSNimLU4ZEbIP+dVqVfdag9aCm46kVOkWjYZMyijh7Yfesz57ZA3tln3/W6ng1Bl7aNMtDLbeq4YvV3Me/Kp1mF6ffqrGeXdSw63lVybvlMYNbKEN966zbz+1l450NdsveyP8N67EnuoHRelfuMjx7PcaPTF7oxahB4u92jILCVkIknNmOpfb9VgWxhnVsUj+SSOEQaeSpfh2XuEJXCAoJYUdIDlOFgK9ZwY8QUl84LpCH
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmQYlYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKqUSScG4rltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBOk6QiA== AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmPohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYLCGPjE5NTwenQzOzcfDiysFi0jaYl84JsqIZVlkSbq4rOC47iqLxsWlzUJJWXpMa+u19qcctWDP3QaZu8qol1XTlRZNEhKq/WNmuRGEswL6LDIOmDGPzIGJFHHOEYBmQ0oYFDh0NYhQibvgqSYDCJq6JDnEVI8fY5zhEibZOyOGWIxDboX6dVxWd1Wrs1bU8t0ykqDYuUUcTZC7tnffbMHtgr+/yzVser4Xpp0ywNtNyshS+W8x//qjSaHZx+q0Z6dnCCHc+rQt5Nj3FvIQ/0rbNuP7+bi3fW2C17I/83rMee6AZ6612+y/Lc9Qg/EnmhF6MGJX+3YxgUNxJJwtmtWHrPb1UQK1jFOvUjhTQOkEGB6tdxiSt0haCQELaF1CBVCPiaJfwIIf0FPG6QiQ==
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmEolYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4VwqltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBPo6Qig==
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmYohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqqDWU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4dx2LLPntyqINaxjk/qRRgYHyKJI9Ru4xBV6QlBICCkhPUwVAr5mBT9CyHwBQK6Qiw==
    depth
    leaf
    yes no
    yes no yes no
    yes no
    Stopping condition: When should the recursion be stopped?
    Stop based on impurity
    Stop based on the number of samples in the region

    View full-size slide

  80. • max_depth
    • max_leaf_nodes
    • min_samples_leaf
    • min_samples_split
    • min_weight_fraction_leaf
    How many regions (leaf nodes) should it be divided into?
    How many sample points should each region (leaf node)
    contain at a minimum?
    Stopping condition: When should the recursion be stopped?
    Stop based on the size of the tree
    Stop based on impurity
    Stop based on the number of samples in the region
    • min_impurity_decrease
    • min_impurity_split

    View full-size slide

  81. Scikit-learn's default setting is to let the tree grow fully.
    from sklearn.tree import DecisionTreeClassifier
    DecisionTreeClassifier().get_params()
    from sklearn.tree import DecisionTreeRegressor
    DecisionTreeRegressor().get_params()
    {'ccp_alpha': 0.0,
    'class_weight': None,
    'criterion': 'gini',
    'max_depth': None,
    'max_features': None,
    'max_leaf_nodes': None,
    'min_impurity_decrease': 0.0,
    'min_samples_leaf': 1,
    'min_samples_split': 2,
    'min_weight_fraction_leaf': 0.0,
    'random_state': None,
    'splitter': 'best'}
    {'ccp_alpha': 0.0,
    'criterion': 'squared_error',
    'max_depth': None,
    'max_features': None,
    'max_leaf_nodes': None,
    'min_impurity_decrease': 0.0,
    'min_samples_leaf': 1,
    'min_samples_split': 2,
    'min_weight_fraction_leaf': 0.0,
    'random_state': None,
    'splitter': 'best'}
    Note: In Scikit-learn, if max_leaf_nodes is set, it expands child nodes in a best-first manner.
    If not set, it expands in a depth-first manner.
    Stopping condition: When should the recursion be stopped?

    View full-size slide

  82. In other words, be careful because even for completely random training data,
    we can (easily) have the zero training error! (Of course, it’s overfitting though)
    from sklearn.tree import DecisionTreeClassifier
    from sklearn.metrics import accuracy_score
    import numpy as np
    X = np.random.rand(100, 3)
    y = np.random.randint(2, size=100)
    model = DecisionTreeClassifier()
    model.fit(X, y)
    y_pred = model.predict(X)
    from sklearn.tree import DecisionTreeRegressor
    from sklearn.metrics import mean_squared_error
    import numpy as np
    X = np.random.rand(100, 3)
    y = np.random.rand(100)
    model = DecisionTreeRegressor()
    model.fit(X, y)
    y_pred = model.predict(X)
    accuracy_score(y, y_pred)
    model.get_n_leaves()
    mean_squared_error(y, y_pred)
    model.get_n_leaves()
    0.0
    100
    1.0
    36
    model.get_depth()
    20
    model.get_depth()
    10
    Stopping condition: When should the recursion be stopped?

    View full-size slide

  83. Optimality concern: Is the greedy method OK?
    TDIDT (Top-Down Induction of Decision Trees) plus pruning is the mainstream for
    learning a single decision tree (to prevent the horizon effect)
    • TDIDT is a greedy method: The best split is chosen at each stage of recursion,
    and the final solution is a combination of these partial solutions.
    → It does not consider global optimality
    • But learning an optimal tree out of all possible trees is known to be NP-Hard
    • Since the optimal tree for the training data is almost never optimal for the test
    data, the actual benefit obtained by working hard on costly combinatorial
    optimization is small
    This is why classic TDIDT (+post-pruning) is still the standard approach: starting
    with building a larger tree, and then prune unnecessary branches to shape the tree

    View full-size slide

  84. Controlling the model complexity of decision trees
    If the model complexity is too low, it results in underfitting.
    If the model complexity is too high, it results in overfitting.
    Underfitting Overfitting
    The basics of decision trees: Control by pruning the overgrown decision tree.
    Caveat: However, when used in ensemble learning, pruning is often not performed
    for each individual decision tree. Controlling the overall complexity of the ensemble
    is important.

    View full-size slide

  85. Cost-Complexity Pruning (CCP)
    AAACBHicbVC7SgNBFJ31GeMramkzGITYhF0JaiME01hGyAuSNdydTJIhM7PLzKwQlrR+g63WdmLrf1j6J06SLUzigQuHc+7lXE4QcaaN6347a+sbm1vbmZ3s7t7+wWHu6Lihw1gRWichD1UrAE05k7RumOG0FSkKIuC0GYwqU7/5RJVmoayZcUR9AQPJ+oyAsdJjpdsBHg2hULvAt7iby7tFdwa8SryU5FGKajf30+mFJBZUGsJB67bnRsZPQBlGOJ1kO7GmEZARDGjbUgmCaj+ZfT3B51bp4X6o7EiDZ+rfiwSE1mMR2E0BZqiXvan4n9eOTf/GT5iMYkMlmQf1Y45NiKcV4B5TlBg+tgSIYvZXTIaggBhb1EJKICa2E2+5gVXSuCx6V8XSQylfvkvbyaBTdIYKyEPXqIzuURXVEUEKvaBX9OY8O+/Oh/M5X11z0psTtADn6xdZn5fL
    C↵(T) =
    ccp_alpha in scikit-learn
    Pruning of decision trees
    Q
    Q Q
    Yes No
    Yes No Yes No
    Q
    Q
    Yes No
    Yes No

    https://scikit-learn.org/stable/auto_examples/tree/plot_cost_complexity_pruning.html
    Prediction Error + α × # of leaf nodes
    Minimizing a penalized loss to prevent overfitting:
    4 leaf nodes 3 leaf nodes
    Just revert to the state before that split

    View full-size slide

  86. Weakest link pruning in CART
    • Determine the value of α for regularization by cross-validation
    • We can guarantee that for each α, we can always find a tree that
    minimizes in the following sequence of trees.
    Prune the leaf where the increase in error is the smallest
    a single-split tree
    (decision stump)
    AAACAXicbVC7SgNBFL3rM8ZX1NJmMAixCbviqwymsYyQF2yWMDuZJENmZ5aZWSEsqfwGW63txNYvsfRPnCRbmMQDFw7n3Mu994QxZ9q47reztr6xubWd28nv7u0fHBaOjptaJorQBpFcqnaINeVM0IZhhtN2rCiOQk5b4ag69VtPVGkmRd2MYxpEeCBYnxFsrORXux3M4yEu1S+6haJbdmdAq8TLSBEy1LqFn05PkiSiwhCOtfY9NzZBipVhhNNJvpNoGmMywgPqWypwRHWQzk6eoHOr9FBfKlvCoJn6dyLFkdbjKLSdETZDvexNxf88PzH9uyBlIk4MFWS+qJ9wZCSa/o96TFFi+NgSTBSztyIyxAoTY1Na2BJGE5uJt5zAKmlelr2b8vXjVbFyn6WTg1M4gxJ4cAsVeIAaNICAhBd4hTfn2Xl3PpzPeeuak82cwAKcr18aiJcx
    C↵(T)

    View full-size slide

  87. Generalizing decision trees?
    x1
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuK9gHtUDJppg1NMkOSEcvYT3Cre3fi1p9x65eYtrPQ1gMXDufcy7mcMBHcWN//8gorq2vrG8XN0tb2zu5eef+gaeJUU9agsYh1OySGCa5Yw3IrWDvRjMhQsFY4up76rQemDY/VvR0nLJBkoHjEKbFOunvs4V654lf9GdAywTmpQI56r/zd7cc0lUxZKogxHewnNsiItpwKNil1U8MSQkdkwDqOKiKZCbLZqxN04pQ+imLtRlk0U39fZEQaM5ah25TEDs2iNxX/9UK5kGyjyyDjKkktU3QeHKUC2RhNe0B9rhm1YuwIoZq73xEdEk2odW2VXCl4sYJl0jyrYr+Kb88rtau8niIcwTGcAoYLqMEN1KEBFAbwDC/w6j15b9679zFfLXj5zSH8gff5A0z4lEY=
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuK9gHtUDJppg1NMkOSEcvYT3Cre3fi1p9x65eYtrPQ1gMXDufcy7mcMBHcWN//8gorq2vrG8XN0tb2zu5eef+gaeJUU9agsYh1OySGCa5Yw3IrWDvRjMhQsFY4up76rQemDY/VvR0nLJBkoHjEKbFOunvs4V654lf9GdAywTmpQI56r/zd7cc0lUxZKogxHewnNsiItpwKNil1U8MSQkdkwDqOKiKZCbLZqxN04pQ+imLtRlk0U39fZEQaM5ah25TEDs2iNxX/9UK5kGyjyyDjKkktU3QeHKUC2RhNe0B9rhm1YuwIoZq73xEdEk2odW2VXCl4sYJl0jyrYr+Kb88rtau8niIcwTGcAoYLqMEN1KEBFAbwDC/w6j15b9679zFfLXj5zSH8gff5A0z4lEY=
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuK9gHtUDJppg1NMkOSEcvYT3Cre3fi1p9x65eYtrPQ1gMXDufcy7mcMBHcWN//8gorq2vrG8XN0tb2zu5eef+gaeJUU9agsYh1OySGCa5Yw3IrWDvRjMhQsFY4up76rQemDY/VvR0nLJBkoHjEKbFOunvs4V654lf9GdAywTmpQI56r/zd7cc0lUxZKogxHewnNsiItpwKNil1U8MSQkdkwDqOKiKZCbLZqxN04pQ+imLtRlk0U39fZEQaM5ah25TEDs2iNxX/9UK5kGyjyyDjKkktU3QeHKUC2RhNe0B9rhm1YuwIoZq73xEdEk2odW2VXCl4sYJl0jyrYr+Kb88rtau8niIcwTGcAoYLqMEN1KEBFAbwDC/w6j15b9679zFfLXj5zSH8gff5A0z4lEY=
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuK9gHtUDJppg1NMkOSEcvYT3Cre3fi1p9x65eYtrPQ1gMXDufcy7mcMBHcWN//8gorq2vrG8XN0tb2zu5eef+gaeJUU9agsYh1OySGCa5Yw3IrWDvRjMhQsFY4up76rQemDY/VvR0nLJBkoHjEKbFOunvs4V654lf9GdAywTmpQI56r/zd7cc0lUxZKogxHewnNsiItpwKNil1U8MSQkdkwDqOKiKZCbLZqxN04pQ+imLtRlk0U39fZEQaM5ah25TEDs2iNxX/9UK5kGyjyyDjKkktU3QeHKUC2RhNe0B9rhm1YuwIoZq73xEdEk2odW2VXCl4sYJl0jyrYr+Kb88rtau8niIcwTGcAoYLqMEN1KEBFAbwDC/w6j15b9679zFfLXj5zSH8gff5A0z4lEY=
    x2
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047KifUA7lEyaaUOTzJBkxDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31vO+0Mrq2vrGZmGruL2zu7dfOjhsmjjVlDVoLGLdDolhgivWsNwK1k40IzIUrBWOrqd+64Fpw2N1b8cJCyQZKB5xSqyT7h571V6p7FW8GfAy8XNShhz1Xum7249pKpmyVBBjOr6X2CAj2nIq2KTYTQ1LCB2RAes4qohkJshmr07wqVP6OIq1G2XxTP19kRFpzFiGblMSOzSL3lT81wvlQrKNLoOMqyS1TNF5cJQKbGM87QH3uWbUirEjhGrufsd0SDSh1rVVdKX4ixUsk2a14nsV//a8XLvK6ynAMZzAGfhwATW4gTo0gMIAnuEFXtETekPv6GO+uoLymyP4A/T5A06MlEc=
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047KifUA7lEyaaUOTzJBkxDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31vO+0Mrq2vrGZmGruL2zu7dfOjhsmjjVlDVoLGLdDolhgivWsNwK1k40IzIUrBWOrqd+64Fpw2N1b8cJCyQZKB5xSqyT7h571V6p7FW8GfAy8XNShhz1Xum7249pKpmyVBBjOr6X2CAj2nIq2KTYTQ1LCB2RAes4qohkJshmr07wqVP6OIq1G2XxTP19kRFpzFiGblMSOzSL3lT81wvlQrKNLoOMqyS1TNF5cJQKbGM87QH3uWbUirEjhGrufsd0SDSh1rVVdKX4ixUsk2a14nsV//a8XLvK6ynAMZzAGfhwATW4gTo0gMIAnuEFXtETekPv6GO+uoLymyP4A/T5A06MlEc=
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047KifUA7lEyaaUOTzJBkxDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31vO+0Mrq2vrGZmGruL2zu7dfOjhsmjjVlDVoLGLdDolhgivWsNwK1k40IzIUrBWOrqd+64Fpw2N1b8cJCyQZKB5xSqyT7h571V6p7FW8GfAy8XNShhz1Xum7249pKpmyVBBjOr6X2CAj2nIq2KTYTQ1LCB2RAes4qohkJshmr07wqVP6OIq1G2XxTP19kRFpzFiGblMSOzSL3lT81wvlQrKNLoOMqyS1TNF5cJQKbGM87QH3uWbUirEjhGrufsd0SDSh1rVVdKX4ixUsk2a14nsV//a8XLvK6ynAMZzAGfhwATW4gTo0gMIAnuEFXtETekPv6GO+uoLymyP4A/T5A06MlEc=
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047KifUA7lEyaaUOTzJBkxDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31vO+0Mrq2vrGZmGruL2zu7dfOjhsmjjVlDVoLGLdDolhgivWsNwK1k40IzIUrBWOrqd+64Fpw2N1b8cJCyQZKB5xSqyT7h571V6p7FW8GfAy8XNShhz1Xum7249pKpmyVBBjOr6X2CAj2nIq2KTYTQ1LCB2RAes4qohkJshmr07wqVP6OIq1G2XxTP19kRFpzFiGblMSOzSL3lT81wvlQrKNLoOMqyS1TNF5cJQKbGM87QH3uWbUirEjhGrufsd0SDSh1rVVdKX4ixUsk2a14nsV//a8XLvK6ynAMZzAGfhwATW4gTo0gMIAnuEFXtETekPv6GO+uoLymyP4A/T5A06MlEc=
    n0
    AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubd8mS3b1jd08IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSq4sb7/5RVWVtfWN4qbpa3tnd298v5B0ySZZthgiUh0K6IGBVfYsNwKbKUaqYwEPkTD66n/8Ija8ETd21GKoaR9xWPOqHXSner63XLFr/ozkGUS5KQCOerd8nenl7BMorJMUGPagZ/acEy15UzgpNTJDKaUDWkf244qKtGE49mrE3LilB6JE+1GWTJTf1+MqTRmJCO3KakdmEVvKv7rRXIh2caX4ZirNLOo2Dw4zgSxCZn2QHpcI7Ni5AhlmrvfCRtQTZl1bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/O4iUOw==
    AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubd8mS3b1jd08IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSq4sb7/5RVWVtfWN4qbpa3tnd298v5B0ySZZthgiUh0K6IGBVfYsNwKbKUaqYwEPkTD66n/8Ija8ETd21GKoaR9xWPOqHXSner63XLFr/ozkGUS5KQCOerd8nenl7BMorJMUGPagZ/acEy15UzgpNTJDKaUDWkf244qKtGE49mrE3LilB6JE+1GWTJTf1+MqTRmJCO3KakdmEVvKv7rRXIh2caX4ZirNLOo2Dw4zgSxCZn2QHpcI7Ni5AhlmrvfCRtQTZl1bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/O4iUOw==
    AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubd8mS3b1jd08IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSq4sb7/5RVWVtfWN4qbpa3tnd298v5B0ySZZthgiUh0K6IGBVfYsNwKbKUaqYwEPkTD66n/8Ija8ETd21GKoaR9xWPOqHXSner63XLFr/ozkGUS5KQCOerd8nenl7BMorJMUGPagZ/acEy15UzgpNTJDKaUDWkf244qKtGE49mrE3LilB6JE+1GWTJTf1+MqTRmJCO3KakdmEVvKv7rRXIh2caX4ZirNLOo2Dw4zgSxCZn2QHpcI7Ni5AhlmrvfCRtQTZl1bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/O4iUOw==
    AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubd8mS3b1jd08IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSq4sb7/5RVWVtfWN4qbpa3tnd298v5B0ySZZthgiUh0K6IGBVfYsNwKbKUaqYwEPkTD66n/8Ija8ETd21GKoaR9xWPOqHXSner63XLFr/ozkGUS5KQCOerd8nenl7BMorJMUGPagZ/acEy15UzgpNTJDKaUDWkf244qKtGE49mrE3LilB6JE+1GWTJTf1+MqTRmJCO3KakdmEVvKv7rRXIh2caX4ZirNLOo2Dw4zgSxCZn2QHpcI7Ni5AhlmrvfCRtQTZl1bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/O4iUOw==
    dn0
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XPpr65QqyGw1UrdYgsmvIdd26IejCrTcc6NhojgpYouWXPwdhQrKYCk04VqrvoFR7OZaaEU6npUGmaIrJGA9p31CBY6q8fH7uFJ4ZJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuZUL4/h/+Tjms7yHZuq5Xm1TKeIjgBp+AcOOASNMENaIE2IGAMHsETeLYerBfr1XpbtBas5cwx+AXr/QsAXZZn
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XPpr65QqyGw1UrdYgsmvIdd26IejCrTcc6NhojgpYouWXPwdhQrKYCk04VqrvoFR7OZaaEU6npUGmaIrJGA9p31CBY6q8fH7uFJ4ZJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuZUL4/h/+Tjms7yHZuq5Xm1TKeIjgBp+AcOOASNMENaIE2IGAMHsETeLYerBfr1XpbtBas5cwx+AXr/QsAXZZn
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XPpr65QqyGw1UrdYgsmvIdd26IejCrTcc6NhojgpYouWXPwdhQrKYCk04VqrvoFR7OZaaEU6npUGmaIrJGA9p31CBY6q8fH7uFJ4ZJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuZUL4/h/+Tjms7yHZuq5Xm1TKeIjgBp+AcOOASNMENaIE2IGAMHsETeLYerBfr1XpbtBas5cwx+AXr/QsAXZZn
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XPpr65QqyGw1UrdYgsmvIdd26IejCrTcc6NhojgpYouWXPwdhQrKYCk04VqrvoFR7OZaaEU6npUGmaIrJGA9p31CBY6q8fH7uFJ4ZJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuZUL4/h/+Tjms7yHZuq5Xm1TKeIjgBp+AcOOASNMENaIE2IGAMHsETeLYerBfr1XpbtBas5cwx+AXr/QsAXZZn
    `5
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKosegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AI70lYk=
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKosegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AI70lYk=
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKosegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AI70lYk=
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKosegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AI70lYk=
    `6
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AJCIlYo=
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AJCIlYo=
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AJCIlYo=
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKqMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL2LXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d16pXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AJCIlYo=
    n4
    AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWakoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSeui6ntV/65WqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AEHYlD8=
    AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWakoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSeui6ntV/65WqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AEHYlD8=
    AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWakoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSeui6ntV/65WqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AEHYlD8=
    AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWakoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSeui6ntV/65WqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AEHYlD8=
    dn4
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSYxxXZXdOOygn1AG8JkMm2HTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+YcqY0Qh9WYW19Y3OruF3a2d3bPygfHrVVkklCWyThieyGWFHOBG1ppjntppLiOOS0E46vZ37nnkrFEnGnJyn1YzwUbMAI1kbqREEuAm8alCvIrteR51UhsqvIdd2aIejCrdUd6NhojgpYohmUP/tRQrKYCk04VqrnoFT7OZaaEU6npX6maIrJGA9pz1CBY6r8fH7uFJ4ZJYKDRJoSGs7VnxM5jpWaxKHpjLEeqVVvJv7phfHKZj2o+TkTaaapIIvFg4xDncBZFjBikhLNJ4ZgIpm5HZIRlphok1jJhPL9OfyftF3bQbZz61UaV8t4iuAEnIJz4IBL0AA3oAlagIAxeARP4Nl6sF6sV+tt0VqwljPH4Bes9y8GsZZr
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSYxxXZXdOOygn1AG8JkMm2HTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+YcqY0Qh9WYW19Y3OruF3a2d3bPygfHrVVkklCWyThieyGWFHOBG1ppjntppLiOOS0E46vZ37nnkrFEnGnJyn1YzwUbMAI1kbqREEuAm8alCvIrteR51UhsqvIdd2aIejCrdUd6NhojgpYohmUP/tRQrKYCk04VqrnoFT7OZaaEU6npX6maIrJGA9pz1CBY6r8fH7uFJ4ZJYKDRJoSGs7VnxM5jpWaxKHpjLEeqVVvJv7phfHKZj2o+TkTaaapIIvFg4xDncBZFjBikhLNJ4ZgIpm5HZIRlphok1jJhPL9OfyftF3bQbZz61UaV8t4iuAEnIJz4IBL0AA3oAlagIAxeARP4Nl6sF6sV+tt0VqwljPH4Bes9y8GsZZr
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSYxxXZXdOOygn1AG8JkMm2HTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+YcqY0Qh9WYW19Y3OruF3a2d3bPygfHrVVkklCWyThieyGWFHOBG1ppjntppLiOOS0E46vZ37nnkrFEnGnJyn1YzwUbMAI1kbqREEuAm8alCvIrteR51UhsqvIdd2aIejCrdUd6NhojgpYohmUP/tRQrKYCk04VqrnoFT7OZaaEU6npX6maIrJGA9pz1CBY6r8fH7uFJ4ZJYKDRJoSGs7VnxM5jpWaxKHpjLEeqVVvJv7phfHKZj2o+TkTaaapIIvFg4xDncBZFjBikhLNJ4ZgIpm5HZIRlphok1jJhPL9OfyftF3bQbZz61UaV8t4iuAEnIJz4IBL0AA3oAlagIAxeARP4Nl6sF6sV+tt0VqwljPH4Bes9y8GsZZr
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSYxxXZXdOOygn1AG8JkMm2HTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+YcqY0Qh9WYW19Y3OruF3a2d3bPygfHrVVkklCWyThieyGWFHOBG1ppjntppLiOOS0E46vZ37nnkrFEnGnJyn1YzwUbMAI1kbqREEuAm8alCvIrteR51UhsqvIdd2aIejCrdUd6NhojgpYohmUP/tRQrKYCk04VqrnoFT7OZaaEU6npX6maIrJGA9pz1CBY6r8fH7uFJ4ZJYKDRJoSGs7VnxM5jpWaxKHpjLEeqVVvJv7phfHKZj2o+TkTaaapIIvFg4xDncBZFjBikhLNJ4ZgIpm5HZIRlphok1jJhPL9OfyftF3bQbZz61UaV8t4iuAEnIJz4IBL0AA3oAlagIAxeARP4Nl6sF6sV+tt0VqwljPH4Bes9y8GsZZr
    n1
    AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubvWTJ7t6x+04IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSqFRd//8gorq2vrG8XN0tb2zu5eef+gaZPMMN5giUxMK6KWS6F5AwVK3koNpyqS/CEaXk/9h0durEj0PY5SHira1yIWjKKT7nQ36JYrftWfgSyTICcVyFHvlr87vYRlimtkklrbDvwUwzE1KJjkk1InszylbEj7vO2oporbcDx7dUJOnNIjcWLcaCQz9ffFmCprRypym4riwC56U/FfL1ILyRhfhmOh0wy5ZvPgOJMEEzLtgfSE4QzlyBHKjHC/EzaghjJ0bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/PRyUPA==
    AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubvWTJ7t6x+04IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSqFRd//8gorq2vrG8XN0tb2zu5eef+gaZPMMN5giUxMK6KWS6F5AwVK3koNpyqS/CEaXk/9h0durEj0PY5SHira1yIWjKKT7nQ36JYrftWfgSyTICcVyFHvlr87vYRlimtkklrbDvwUwzE1KJjkk1InszylbEj7vO2oporbcDx7dUJOnNIjcWLcaCQz9ffFmCprRypym4riwC56U/FfL1ILyRhfhmOh0wy5ZvPgOJMEEzLtgfSE4QzlyBHKjHC/EzaghjJ0bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/PRyUPA==
    AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubvWTJ7t6x+04IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSqFRd//8gorq2vrG8XN0tb2zu5eef+gaZPMMN5giUxMK6KWS6F5AwVK3koNpyqS/CEaXk/9h0durEj0PY5SHira1yIWjKKT7nQ36JYrftWfgSyTICcVyFHvlr87vYRlimtkklrbDvwUwzE1KJjkk1InszylbEj7vO2oporbcDx7dUJOnNIjcWLcaCQz9ffFmCprRypym4riwC56U/FfL1ILyRhfhmOh0wy5ZvPgOJMEEzLtgfSE4QzlyBHKjHC/EzaghjJ0bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/PRyUPA==
    AAAB+nicbVA9SwNBFHwXv2L8ilraLAbBKtyJoGXQxjKiiYHkCHubvWTJ7t6x+04IMT/BVns7sfXP2PpL3CRXaOLAg2HmPeYxUSqFRd//8gorq2vrG8XN0tb2zu5eef+gaZPMMN5giUxMK6KWS6F5AwVK3koNpyqS/CEaXk/9h0durEj0PY5SHira1yIWjKKT7nQ36JYrftWfgSyTICcVyFHvlr87vYRlimtkklrbDvwUwzE1KJjkk1InszylbEj7vO2oporbcDx7dUJOnNIjcWLcaCQz9ffFmCprRypym4riwC56U/FfL1ILyRhfhmOh0wy5ZvPgOJMEEzLtgfSE4QzlyBHKjHC/EzaghjJ0bZVcKcFiBcukeVYN/Gpwe16pXeX1FOEIjuEUAriAGtxAHRrAoA/P8AKv3pP35r17H/PVgpffHMIfeJ8/PRyUPA==
    dn1
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XvjP1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcB8pZo
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XvjP1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcB8pZo
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XvjP1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcB8pZo
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8XvjP1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcB8pZo
    `1
    AAAB/XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNlJJxkzM7vMzAphCf6CV717E69+i1e/xEmyB00saCiquqmmokRwY33/y1tZXVvf2CxsFbd3dvf2SweHDROnmmGdxSLWrYgaFFxh3XIrsJVopDIS2IxGN1O/+Yja8Fjd23GCoaQDxfucUeukRgeF6AbdUtmv+DOQZRLkpAw5at3Sd6cXs1SiskxQY9qBn9gwo9pyJnBS7KQGE8pGdIBtRxWVaMJs9u2EnDqlR/qxdqMsmam/LzIqjRnLyG1Kaodm0ZuK/3qRXEi2/asw4ypJLSo2D+6ngtiYTKsgPa6RWTF2hDLN3e+EDammzLrCiq6UYLGCZdI4rwR+Jbi7KFev83oKcAwncAYBXEIVbqEGdWDwAM/wAq/ek/fmvXsf89UVL785gj/wPn8AiKSVhQ==
    AAAB/XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNlJJxkzM7vMzAphCf6CV717E69+i1e/xEmyB00saCiquqmmokRwY33/y1tZXVvf2CxsFbd3dvf2SweHDROnmmGdxSLWrYgaFFxh3XIrsJVopDIS2IxGN1O/+Yja8Fjd23GCoaQDxfucUeukRgeF6AbdUtmv+DOQZRLkpAw5at3Sd6cXs1SiskxQY9qBn9gwo9pyJnBS7KQGE8pGdIBtRxWVaMJs9u2EnDqlR/qxdqMsmam/LzIqjRnLyG1Kaodm0ZuK/3qRXEi2/asw4ypJLSo2D+6ngtiYTKsgPa6RWTF2hDLN3e+EDammzLrCiq6UYLGCZdI4rwR+Jbi7KFev83oKcAwncAYBXEIVbqEGdWDwAM/wAq/ek/fmvXsf89UVL785gj/wPn8AiKSVhQ==
    AAAB/XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNlJJxkzM7vMzAphCf6CV717E69+i1e/xEmyB00saCiquqmmokRwY33/y1tZXVvf2CxsFbd3dvf2SweHDROnmmGdxSLWrYgaFFxh3XIrsJVopDIS2IxGN1O/+Yja8Fjd23GCoaQDxfucUeukRgeF6AbdUtmv+DOQZRLkpAw5at3Sd6cXs1SiskxQY9qBn9gwo9pyJnBS7KQGE8pGdIBtRxWVaMJs9u2EnDqlR/qxdqMsmam/LzIqjRnLyG1Kaodm0ZuK/3qRXEi2/asw4ypJLSo2D+6ngtiYTKsgPa6RWTF2hDLN3e+EDammzLrCiq6UYLGCZdI4rwR+Jbi7KFev83oKcAwncAYBXEIVbqEGdWDwAM/wAq/ek/fmvXsf89UVL785gj/wPn8AiKSVhQ==
    AAAB/XicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNlJJxkzM7vMzAphCf6CV717E69+i1e/xEmyB00saCiquqmmokRwY33/y1tZXVvf2CxsFbd3dvf2SweHDROnmmGdxSLWrYgaFFxh3XIrsJVopDIS2IxGN1O/+Yja8Fjd23GCoaQDxfucUeukRgeF6AbdUtmv+DOQZRLkpAw5at3Sd6cXs1SiskxQY9qBn9gwo9pyJnBS7KQGE8pGdIBtRxWVaMJs9u2EnDqlR/qxdqMsmam/LzIqjRnLyG1Kaodm0ZuK/3qRXEi2/asw4ypJLSo2D+6ngtiYTKsgPa6RWTF2hDLN3e+EDammzLrCiq6UYLGCZdI4rwR+Jbi7KFev83oKcAwncAYBXEIVbqEGdWDwAM/wAq/ek/fmvXsf89UVL785gj/wPn8AiKSVhQ==
    `2
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukVat6btW7u6jUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/ijiVhg==
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukVat6btW7u6jUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/ijiVhg==
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukVat6btW7u6jUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/ijiVhg==
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukVat6btW7u6jUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/ijiVhg==
    n2
    AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWaKoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSatW9b2qf3dRqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AD6wlD0=
    AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWaKoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSatW9b2qf3dRqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AD6wlD0=
    AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWaKoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSatW9b2qf3dRqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AD6wlD0=
    AAAB+nicbVDLSgMxFL2pr1pfVZdugkVwVWaKoMuiG5cV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sZ73hQpr6xubW8Xt0s7u3v5B+fCoZeJUU9aksYh1JySGCa5Y03IrWCfRjMhQsHY4vpn57UemDY/Vg50kLJBkqHjEKbFOulf9Wr9c8areHHiV+DmpQI5Gv/zdG8Q0lUxZKogxXd9LbJARbTkVbFrqpYYlhI7JkHUdVUQyE2TzV6f4zCkDHMXajbJ4rv6+yIg0ZiJDtymJHZllbyb+64VyKdlGV0HGVZJapugiOEoFtjGe9YAHXDNqxcQRQjV3v2M6IppQ69oquVL85QpWSatW9b2qf3dRqV/n9RThBE7hHHy4hDrcQgOaQGEIz/ACr+gJvaF39LFYLaD85hj+AH3+AD6wlD0=
    dn2
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8Xvjv1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcDh5Zp
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8Xvjv1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcDh5Zp
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8Xvjv1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcDh5Zp
    AAAB/nicdVDLSsNAFJ3UV62vqks3g0VwFSaxxXZXdOOygn1AG8JkMmmHTiZhZiKUUPAX3Orenbj1V9z6JU4fghY9cOFwzr3ce0+QcqY0Qh9WYW19Y3OruF3a2d3bPygfHnVUkklC2yThiewFWFHOBG1rpjntpZLiOOC0G4yvZ373nkrFEnGnJyn1YjwULGIEayN1Qz8Xvjv1yxVkNxqoWq1BZNeQ67p1Q9CFW2840LHRHBWwRMsvfw7ChGQxFZpwrFTfQan2ciw1I5xOS4NM0RSTMR7SvqECx1R5+fzcKTwzSgijRJoSGs7VnxM5jpWaxIHpjLEeqVVvJv7pBfHKZh3VvZyJNNNUkMXiKONQJ3CWBQyZpETziSGYSGZuh2SEJSbaJFYyoXx/Dv8nHdd2kO3cVivNq2U8RXACTsE5cMAlaIIb0AJtQMAYPIIn8Gw9WC/Wq/W2aC1Yy5lj8AvW+xcDh5Zp
    `3
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL3zXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d1GpXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AIvMlYc=
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL3zXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d1GpXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AIvMlYc=
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL3zXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d1GpXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AIvMlYc=
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegF48RzAOSJcxOepMxM7vLzKwQluAveNW7N/Hqt3j1S5wke9DEgoaiqptqKkgE18Z1v5zCyura+kZxs7S1vbO7V94/aOo4VQwbLBaxagdUo+ARNgw3AtuJQioDga1gdDP1W4+oNI+jezNO0Jd0EPGQM2qs1OyiEL3zXrniVt0ZyDLxclKBHPVe+bvbj1kqMTJMUK07npsYP6PKcCZwUuqmGhPKRnSAHUsjKlH72ezbCTmxSp+EsbITGTJTf19kVGo9loHdlNQM9aI3Ff/1ArmQbMIrP+NRkhqM2Dw4TAUxMZlWQfpcITNibAllitvfCRtSRZmxhZVsKd5iBcukeVb13Kp3d1GpXef1FOEIjuEUPLiEGtxCHRrA4AGe4QVenSfnzXl3PuarBSe/OYQ/cD5/AIvMlYc=
    `4
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukdVH13Kp3V6vUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/jWCViA==
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukdVH13Kp3V6vUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/jWCViA==
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukdVH13Kp3V6vUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/jWCViA==
    AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKexKQI9BLx4jmAckS5id9CZjZmeXmVkhLMFf8Kp3b+LVb/HqlzhJ9qCJBQ1FVTfVVJAIro3rfjmFtfWNza3idmlnd2//oHx41NJxqhg2WSxi1QmoRsElNg03AjuJQhoFAtvB+Gbmtx9RaR7LezNJ0I/oUPKQM2qs1OqhEP1av1xxq+4cZJV4OalAjka//N0bxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bP7tlJxZZUDCWNmRhszV3xcZjbSeRIHdjKgZ6WVvJv7rBdFSsgmv/IzLJDUo2SI4TAUxMZlVQQZcITNiYgllitvfCRtRRZmxhZVsKd5yBaukdVH13Kp3V6vUr/N6inACp3AOHlxCHW6hAU1g8ADP8AKvzpPz5rw7H4vVgpPfHMMfOJ8/jWCViA==
    n3
    AAAB+nicbVC7SgNBFL0bXzG+opY2g0GwCrsqaBm0sYxoHpAsYXYymwyZxzIzK4Q1n2CrvZ3Y+jO2fomTZAtNPHDhcM69nMuJEs6M9f0vr7Cyura+UdwsbW3v7O6V9w+aRqWa0AZRXOl2hA3lTNKGZZbTdqIpFhGnrWh0M/Vbj1QbpuSDHSc0FHggWcwItk66l73zXrniV/0Z0DIJclKBHPVe+bvbVyQVVFrCsTGdwE9smGFtGeF0UuqmhiaYjPCAdhyVWFATZrNXJ+jEKX0UK+1GWjRTf19kWBgzFpHbFNgOzaI3Ff/1IrGQbOOrMGMySS2VZB4cpxxZhaY9oD7TlFg+dgQTzdzviAyxxsS6tkqulGCxgmXSPKsGfjW4u6jUrvN6inAEx3AKAVxCDW6hDg0gMIBneIFX78l78969j/lqwctvDuEPvM8fQESUPg==
    AAAB+nicbVC7SgNBFL0bXzG+opY2g0GwCrsqaBm0sYxoHpAsYXYymwyZxzIzK4Q1n2CrvZ3Y+jO2fomTZAtNPHDhcM69nMuJEs6M9f0vr7Cyura+UdwsbW3v7O6V9w+aRqWa0AZRXOl2hA3lTNKGZZbTdqIpFhGnrWh0M/Vbj1QbpuSDHSc0FHggWcwItk66l73zXrniV/0Z0DIJclKBHPVe+bvbVyQVVFrCsTGdwE9smGFtGeF0UuqmhiaYjPCAdhyVWFATZrNXJ+jEKX0UK+1GWjRTf19kWBgzFpHbFNgOzaI3Ff/1IrGQbOOrMGMySS2VZB4cpxxZhaY9oD7TlFg+dgQTzdzviAyxxsS6tkqulGCxgmXSPKsGfjW4u6jUrvN6inAEx3AKAVxCDW6hDg0gMIBneIFX78l78969j/lqwctvDuEPvM8fQESUPg==
    AAAB+nicbVC7SgNBFL0bXzG+opY2g0GwCrsqaBm0sYxoHpAsYXYymwyZxzIzK4Q1n2CrvZ3Y+jO2fomTZAtNPHDhcM69nMuJEs6M9f0vr7Cyura+UdwsbW3v7O6V9w+aRqWa0AZRXOl2hA3lTNKGZZbTdqIpFhGnrWh0M/Vbj1QbpuSDHSc0FHggWcwItk66l73zXrniV/0Z0DIJclKBHPVe+bvbVyQVVFrCsTGdwE9smGFtGeF0UuqmhiaYjPCAdhyVWFATZrNXJ+jEKX0UK+1GWjRTf19kWBgzFpHbFNgOzaI3Ff/1IrGQbOOrMGMySS2VZB4cpxxZhaY9oD7TlFg+dgQTzdzviAyxxsS6tkqulGCxgmXSPKsGfjW4u6jUrvN6inAEx3AKAVxCDW6hDg0gMIBneIFX78l78969j/lqwctvDuEPvM8fQESUPg==
    AAAB+nicbVC7SgNBFL0bXzG+opY2g0GwCrsqaBm0sYxoHpAsYXYymwyZxzIzK4Q1n2CrvZ3Y+jO2fomTZAtNPHDhcM69nMuJEs6M9f0vr7Cyura+UdwsbW3v7O6V9w+aRqWa0AZRXOl2hA3lTNKGZZbTdqIpFhGnrWh0M/Vbj1QbpuSDHSc0FHggWcwItk66l73zXrniV/0Z0DIJclKBHPVe+bvbVyQVVFrCsTGdwE9smGFtGeF0UuqmhiaYjPCAdhyVWFATZrNXJ+jEKX0UK+1GWjRTf19kWBgzFpHbFNgOzaI3Ff/1IrGQbOOrMGMySS2VZB4cpxxZhaY9oD7TlFg+dgQTzdzviAyxxsS6tkqulGCxgmXSPKsGfjW4u6jUrvN6inAEx3AKAVxCDW6hDg0gMIBneIFX78l78969j/lqwctvDuEPvM8fQESUPg==
    dn3
    AAAB/nicdVDLSsNAFJ34rPVVdelmsAiuwiRtsd0V3bisYB/QhjCZTNqhk0mYmQglFPwFt7p3J279Fbd+idOHoEUPXDiccy/33hOknCmN0Ie1tr6xubVd2Cnu7u0fHJaOjjsqySShbZLwRPYCrChngrY105z2UklxHHDaDcbXM797T6ViibjTk5R6MR4KFjGCtZG6oZ8LvzL1S2VkNxqoWq1BZNeQ67p1Q1DFrTcc6NhojjJYouWXPgdhQrKYCk04VqrvoFR7OZaaEU6nxUGmaIrJGA9p31CBY6q8fH7uFJ4bJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuaUL4/h/+Tjms7yHZuq+Xm1TKeAjgFZ+ACOOASNMENaIE2IGAMHsETeLYerBfr1XpbtK5Zy5kT8AvW+xcFHJZq
    AAAB/nicdVDLSsNAFJ34rPVVdelmsAiuwiRtsd0V3bisYB/QhjCZTNqhk0mYmQglFPwFt7p3J279Fbd+idOHoEUPXDiccy/33hOknCmN0Ie1tr6xubVd2Cnu7u0fHJaOjjsqySShbZLwRPYCrChngrY105z2UklxHHDaDcbXM797T6ViibjTk5R6MR4KFjGCtZG6oZ8LvzL1S2VkNxqoWq1BZNeQ67p1Q1DFrTcc6NhojjJYouWXPgdhQrKYCk04VqrvoFR7OZaaEU6nxUGmaIrJGA9p31CBY6q8fH7uFJ4bJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuaUL4/h/+Tjms7yHZuq+Xm1TKeAjgFZ+ACOOASNMENaIE2IGAMHsETeLYerBfr1XpbtK5Zy5kT8AvW+xcFHJZq
    AAAB/nicdVDLSsNAFJ34rPVVdelmsAiuwiRtsd0V3bisYB/QhjCZTNqhk0mYmQglFPwFt7p3J279Fbd+idOHoEUPXDiccy/33hOknCmN0Ie1tr6xubVd2Cnu7u0fHJaOjjsqySShbZLwRPYCrChngrY105z2UklxHHDaDcbXM797T6ViibjTk5R6MR4KFjGCtZG6oZ8LvzL1S2VkNxqoWq1BZNeQ67p1Q1DFrTcc6NhojjJYouWXPgdhQrKYCk04VqrvoFR7OZaaEU6nxUGmaIrJGA9p31CBY6q8fH7uFJ4bJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuaUL4/h/+Tjms7yHZuq+Xm1TKeAjgFZ+ACOOASNMENaIE2IGAMHsETeLYerBfr1XpbtK5Zy5kT8AvW+xcFHJZq
    AAAB/nicdVDLSsNAFJ34rPVVdelmsAiuwiRtsd0V3bisYB/QhjCZTNqhk0mYmQglFPwFt7p3J279Fbd+idOHoEUPXDiccy/33hOknCmN0Ie1tr6xubVd2Cnu7u0fHJaOjjsqySShbZLwRPYCrChngrY105z2UklxHHDaDcbXM797T6ViibjTk5R6MR4KFjGCtZG6oZ8LvzL1S2VkNxqoWq1BZNeQ67p1Q1DFrTcc6NhojjJYouWXPgdhQrKYCk04VqrvoFR7OZaaEU6nxUGmaIrJGA9p31CBY6q8fH7uFJ4bJYRRIk0JDefqz4kcx0pN4sB0xliP1Ko3E//0gnhls47qXs5EmmkqyGJxlHGoEzjLAoZMUqL5xBBMJDO3QzLCEhNtEiuaUL4/h/+Tjms7yHZuq+Xm1TKeAjgFZ+ACOOASNMENaIE2IGAMHsETeLYerBfr1XpbtK5Zy5kT8AvW+xcFHJZq
    • Recursive 'univariate partitioning' → Is any bisection acceptable?
    • Constant prediction in leaves → Are any non-constant leaves acceptable?
    AAACq3ichVE9S8NQFD2N35+tugguxVKpg/IiguJUdHGs1rZilZLE1xqbL5K0WKt/wMlN1EnBQfwZLv0DDv0J4qjg4uBNGhAtrTck77xz77nvvFzZ0lTHZawZEnp6+/oHBoeGR0bHxsORicmsY1ZshWcUUzPtXVlyuKYaPOOqrsZ3LZtLuqzxnFze8PK5Krcd1TR23JrFD3SpZKhFVZFcorKpRO3sZL4QibFF5ke0HYgBiCGIlBlpYB+HMKGgAh0cBlzCGiQ49OQhgsEi7gB14mxCqp/nOMcwaStUxalCIrZM3xLt8gFr0N7r6fhqhU7R6LVJGUWcvbBH9s4a7Im9sq+Ovep+D89LjVa5peVWIXwxnf78V6XT6uLoR9XVs4siVn2vKnm3fMa7hdLSV0+v3tNr2/H6HLtnb+T/jjXZM93AqH4oD1t8+7aLH5m8dP5jXj6ooBGKfwfWDrJLiyLhreVYcj0Y5iBmMIsETWwFSWwihQydcIxLXONGWBDSwp6w3yoVQoFmCr9C4N89C5ka
    Any model of

    View full-size slide

  88. Extensions: Model trees and Multivariate trees
    ■ OC1/Oblique Tree (Murthy+ 1994)
    ■ Perceptron Tree (Utgoff , 1988)
    ■ Large Margin Tree (Wu+ 1999; Bennett+ 2000)
    ■ Margin Tree (Tibshirani & Hastie 2007)
    ■ Geometric Decision Tree (Manwani & Sastry, 2012)
    ■ HHCART (Wickramarachchi+, 2016)
    ■ M5/M5’ (Quinlan, 1992; Wang & Witten 1997)
    ■ RETIS (Karalic & Cestnik 1991)
    ■ SECRET (Dobra & Gehrke, 2002)
    ■ SMOTI (Malerba+ 2004)
    ■ MAUVE (Vens & Blockeel, 2006)
    ■ LLRT (Vogel, Asparouhov, Scheffer, 2007)
    ■ Cubist (Quinlan, 2011)
    AAAClnichVHLSsNAFD3Gd3201Y3gplgqrmQiguJCiiK6rNZqoZWQxLEdmiYxmVa0+AP+gAtxoaAifoAf4MYfcOEniEsFNy68TQOiot4wmTNn7rlzZq7hWsKXjD22Ke0dnV3dPb2Rvv6BwWgsPrThOzXP5DnTsRwvb+g+t4TNc1JIi+ddj+tVw+KbRmWxub9Z554vHHtd7rt8q6qXbLEjTF0SpcXieU0kiiW+myjKMpe6pmqxJJtkQSR+AjUESYSRcWK3KGIbDkzUUAWHDUnYgg6fvgJUMLjEbaFBnEdIBPsch4iQtkZZnDJ0Yiv0L9GqELI2rZs1/UBt0ikWDY+UCaTYA7tmL+ye3bAn9v5rrUZQo+lln2ajpeWuFj0ayb79q6rSLFH+VP3pWWIHs4FXQd7dgGnewmzp6wfHL9m5tVRjnJ2zZ/J/xh7ZHd3Arr+aF6t87eQPPwZ5oRejBqnf2/ETbExNqoRXp5PphbBVPRjFGCaoHzNIYwUZ5Kj+Hk5xiStlRJlXlpTlVqrSFmqG8SWUzAedy5ZD
    AAAClnichVHLSsNAFD2Nr/psq5uCm2BRXMlEBMWFFEV02VarBSshiWONTZOYTCu1+AP+gAtxoaAifoAf4MYfcOEniEsFNy68TQOiot4wmTNn7rlzZq7uWqYvGHuMSG3tHZ1d0e6e3r7+gVg8MbjmO1XP4HnDsRyvoGs+t0yb54UpLF5wPa5VdIuv6+WF5v56jXu+6dirou7yzYpWss1t09AEUWo8UVB35WKJ78lFscOFpk6q8RSbYEHIP4ESghTCyDjxWxSxBQcGqqiAw4YgbEGDT98GFDC4xG2iQZxHyAz2OQ7RQ9oqZXHK0Igt079Eq42QtWndrOkHaoNOsWh4pJQxyh7YNXth9+yGPbH3X2s1ghpNL3Wa9ZaWu2rsKLny9q+qQrPAzqfqT88C25gJvJrk3Q2Y5i2Mlr52cPyyMpsbbYyxc/ZM/s/YI7ujG9i1V+Miy3Mnf/jRyQu9GDVI+d6On2BtckIhnJ1KpefDVkUxjBGMUz+mkcYyMshT/X2c4hJXUlKakxalpVaqFAk1Q/gSUuYDohmWRQ==
    AAAClnichVHLSsNAFD3Gd3201Y3gplgqrmSqguJCRBFd1mof0JaQxLENTZOYTCta/AF/wIW4UFARP8APcOMPuPATxGUFNy68TQOixXrDZM6cuefOmbmqbeiuYOylS+ru6e3rHxgMDA2PjAZD4bG0a1Udjac0y7CcrKq43NBNnhK6MHjWdrhSUQ2eUcvrzf1MjTuubpm74sjmhYpSNPV9XVMEUXIonJXLkXyRH0TyosSFIs/LoSibZV5E2kHcB1H4kbBCD8hjDxY0VFEBhwlB2IACl74c4mCwiSugTpxDSPf2OU4QIG2VsjhlKMSW6V+kVc5nTVo3a7qeWqNTDBoOKSOIsWd2xxrsid2zV/b5Z626V6Pp5YhmtaXlthw8ndj5+FdVoVmg9K3q6FlgH0ueV5282x7TvIXW0teOzxo7y8lYfZpdsTfyf8le2CPdwKy9a9fbPHnewY9KXujFqEHx3+1oB+m52Tjh7YXo6prfqgFMYgoz1I9FrGILCaSo/iEucINbaUJakTakzVaq1OVrxvEjpMQXpmeWRw==
    AAAClnichVHLSsNAFD2Nr1pfrW4EN8VScSVTKSgupCiiy1atLbQlJHFsg2kSk2lFiz/gD7gQFwoq4gf4AW78ARf9BHGp4MaFt2lAtFhvmMyZM/fcOTNXtQ3dFYw1A1JPb1//QHAwNDQ8MjoWjozvuFbN0XhWswzLyauKyw3d5FmhC4PnbYcrVdXgOXV/tbWfq3PH1S1zWxzZvFRVyqa+p2uKIEoOR/KyES2W+UG0KCpcKHJSDsfYHPMi2gkSPojBj7QVfkARu7CgoYYqOEwIwgYUuPQVkACDTVwJDeIcQrq3z3GCEGlrlMUpQyF2n/5lWhV81qR1q6brqTU6xaDhkDKKOHtmd+yNPbF79sI+/6zV8Gq0vBzRrLa13JbHTie3Pv5VVWkWqHyrunoW2MOi51Un77bHtG6htfX147O3raXNeGOGXbFX8n/JmuyRbmDW37XrDN887+JHJS/0YtSgxO92dIKd+bkE4UwyllrxWxXEFKYxS/1YQAobSCNL9Q9xgRvcSpPSsrQmrbdTpYCvmcCPkNJfqrWWSQ==
    AAACmnichVHLSsNAFD3Gd31VRRB0USxVQZCJCIorqRvFjVprC1pCEqc1mBfJtKDBH/AHXLiq4EL7AX6AG3/AhZ8gLiu4ceFtGhAt1hsmc+bMPXfOzNVc0/AFYy8dUmdXd09vX39sYHBoeCQ+OnbgO2VP51ndMR0vr6k+Nw2bZ4UhTJ53Pa5amslz2ulGYz9X4Z5vOPa+OHN5wVJLtlE0dFUQpcQncoo8lzjSrCB/kVggwIWqyEo8yRZZGIlWIEcgiSh2nPgDjnAMBzrKsMBhQxA2ocKn7xAyGFziCgiI8wgZ4T7HBWKkLVMWpwyV2FP6l2h1GLE2rRs1/VCt0ykmDY+UCaTYM7tjdfbEauyVff5ZKwhrNLyc0aw1tdxVRi4nMx//qiyaBU6+VW09CxSxGno1yLsbMo1b6E195fyqnlnbSwWz7Ia9kf8qe2GPdAO78q7f7vK96zZ+NPJCL0YNkn+3oxUcLC3KhHeXk+vpqFV9mMIM5qkfK1jHJnaQpfoBqrhHTZqW0tKWtN1MlToizTh+hLT/BZu5lww=
    AAACmnichVHLSsNAFD3G97NVEQRdFIsPEMpUBcVVqRvFTVutLVQJSZzW0LxIpgUN/oA/4MKVggv1A/wAN/6Ai36CuFRw48LbNCBa1Bsmc+bMPXfOzFUdQ/cEY40OqbOru6e3r39gcGh4JBIdHdvz7Jqr8bxmG7ZbVBWPG7rF80IXBi86LldM1eAFtbrR3C/UuevptrUrjh1+YCoVSy/rmiKIkqMTBXl5Pravmn7xNLZIgAtFXpajcZZgQcTaQTIEcYSRsaP32MchbGiowQSHBUHYgAKPvhKSYHCIO4BPnEtID/Y5TjFA2hplccpQiK3Sv0KrUshatG7W9AK1RqcYNFxSxjDLntgNe2WP7I49s49fa/lBjaaXY5rVlpY7cuRscuf9X5VJs8DRl+pPzwJlrAVedfLuBEzzFlpLXz85f91Zz836c+yKvZD/S9ZgD3QDq/6mXWd57uIPPyp5oRejBiV/tqMd7C0lkoSzK/FUOmxVH6YwgwXqxypS2EQGearv4xK3uJOmpbS0JW23UqWOUDOObyHtfgKkXZcQ
    AAACmnichVHLSsNAFD2N7/qqiiDoolh8gFCmRVBciW4UN9W2tqASkjitwbxIpgUN/oA/4MJVBRfaD/AD3PgDLvoJ4lLBjQtv04Bosd4wmTNn7rlzZq7qGLonGGtEpK7unt6+/oHo4NDwyGhsbHzfsyuuxvOabdhuUVU8bugWzwtdGLzouFwxVYMX1NPN5n6hyl1Pt62cOHP4kamULb2ka4ogSo5NFuT0QvxQNf3iRXyJABeKnJZjCZZkQcTbQSoECYSRsWMPOMQxbGiowASHBUHYgAKPvgOkwOAQdwSfOJeQHuxzXCBK2gplccpQiD2lf5lWByFr0bpZ0wvUGp1i0HBJGccce2Z37I09sTp7YZ9/1vKDGk0vZzSrLS135NHLqezHvyqTZoGTb1VHzwIlrAZedfLuBEzzFlpLXz2/esuu7c358+yGvZL/GmuwR7qBVX3Xbnf53nUHPyp5oRejBqV+t6Md7KeTKcK7y4n1jbBV/ZjGLBapHytYxxYyyFN9HzXcoy7NSBvStrTTSpUioWYCP0LKfQGgC5cO
    AAACmnichVHLSsNAFD2N7/qqiiDoolh8gFCmUlBciW4UN9W2tqASkjitwbxIpgUN/oA/4MJVBRfaD/AD3PgDLvoJ4lLBjQtv04Bosd4wmTNn7rlzZq7qGLonGGtEpK7unt6+/oHo4NDwyGhsbHzfsyuuxvOabdhuUVU8bugWzwtdGLzouFwxVYMX1NPN5n6hyl1Pt62cOHP4kamULb2ka4ogSo5NFuT0QvxQNf3iRXyJABeKnJZjCZZkQcTbQSoECYSRsWMPOMQxbGiowASHBUHYgAKPvgOkwOAQdwSfOJeQHuxzXCBK2gplccpQiD2lf5lWByFr0bpZ0wvUGp1i0HBJGccce2Z37I09sTp7YZ9/1vKDGk0vZzSrLS135NHLqezHvyqTZoGTb1VHzwIlrAZedfLuBEzzFlpLXz2/esuu7c358+yGvZL/GmuwR7qBVX3Xbnf53nUHPyp5oRejBqV+t6Md7C8nU4R304n1jbBV/ZjGLBapHytYxxYyyFN9HzXcoy7NSBvStrTTSpUioWYCP0LKfQGor5cS
    AAACmnichVHLSsNAFD3G97NVEQRdFIsPEMpUFMVVqRvFTVutLVQJSZzW0LxIpgUN/oA/4MKVggv1A/wAN/6Ai36CuFRw48LbNCBa1Bsmc+bMPXfOzFUdQ/cEY40OqbOru6e3r39gcGh4JBIdHdvz7Jqr8bxmG7ZbVBWPG7rF80IXBi86LldM1eAFtbrR3C/UuevptrUrjh1+YCoVSy/rmiKIkqMTBXllPravmn7xNLZIgAtFXpGjcZZgQcTaQTIEcYSRsaP32MchbGiowQSHBUHYgAKPvhKSYHCIO4BPnEtID/Y5TjFA2hplccpQiK3Sv0KrUshatG7W9AK1RqcYNFxSxjDLntgNe2WP7I49s49fa/lBjaaXY5rVlpY7cuRscuf9X5VJs8DRl+pPzwJlrAVedfLuBEzzFlpLXz85f91Zz836c+yKvZD/S9ZgD3QDq/6mXWd57uIPPyp5oRejBiV/tqMd7C0lkoSzy/FUOmxVH6YwgwXqxypS2EQGearv4xK3uJOmpbS0JW23UqWOUDOObyHtfgKtAZcU
    AAACnnichVG7SgNBFD2u7/hI1EaxWQw+qjArgmIlCmIjGjUmYGTZXSfJkn25OwloEHt/wMJKwUIELf0AG3/Awk8QSwUbC282C6JivMvsnDlzz50zc3XPMgPB2FOL1NrW3tHZ1R3r6e3rjycGBrcDt+IbPGO4luvndC3glunwjDCFxXOezzVbt3hWLy/V97NV7gem62yJA4/v2lrRMQumoQmi1MRIVlUm5bxu13JHcr7I9+W8KHGhqYqaSLIUC0P+DZQIJBHFupu4Qx57cGGgAhscDgRhCxoC+naggMEjbhc14nxCZrjPcYQYaSuUxSlDI7ZM/yKtdiLWoXW9ZhCqDTrFouGTUsY4e2RX7JU9sGv2zD7+rFULa9S9HNCsN7TcU+Mnw5vv/6psmgVKX6qmngUKmAu9muTdC5n6LYyGvnp4+ro5vzFem2AX7IX8n7Mndk83cKpvxmWab5w18aOTF3oxapDysx2/wfZ0SiGcnkkuLEat6sIoxjBF/ZjFAlawjgzVP8YlbnArydKytCqtNVKllkgzhG8h5T4BYfKZHA==
    AAACnnichVG7SgNBFD2u72eiNorNYvBRyWwQFCtREBvxmQcksuyuk2RxX+5OAhrE3h+wsFKwEEFLP8DGH7DIJ4hlBBsLbzYLoqLeZXbOnLnnzpm5umeZgWCs1iK1trV3dHZ19/T29Q/E4oND6cAt+wZPGa7l+lldC7hlOjwlTGHxrOdzzdYtntH3lxv7mQr3A9N1dsShx3dtreiYBdPQBFFqfDSjJqfkvG5Xs8dyvsgP5LwocaGpSTWeYDMsDPknUCKQQBQbbvweeezBhYEybHA4EIQtaAjoy0EBg0fcLqrE+YTMcJ/jGD2kLVMWpwyN2H36F2mVi1iH1o2aQag26BSLhk9KGRPsiV2zOntkN+yZvf9aqxrWaHg5pFlvarmnxk5Htt/+Vdk0C5Q+VX96FihgPvRqkncvZBq3MJr6ytFZfXtha6I6yS7ZC/m/YDX2QDdwKq/G1SbfOv/Dj05e6MWoQcr3dvwE6eSMQnhzNrG4FLWqC2MYxzT1Yw6LWMUGUlT/BFe4xZ0kSyvSmrTeTJVaIs0wvoSU/QBmSJke
    AAACnnichVG7SgNBFD2u7/iK2ig2i8FHFSZGUKxEQWzEZ0zAhGV3ncQl+3J3EtAg9v6AhZVCChG09ANs/AELP0EsFWwsvNksiAbjXWbnzJl77pyZq7mm4QvGnluk1rb2js6u7khPb1//QHRwaNd3Sp7OU7pjOl5GU31uGjZPCUOYPON6XLU0k6e14nJtP13mnm849o44cnnOUgu2kTd0VRClREfTSnJKzmpWJXMiZwv8UM6KAy5UJalEYyzOgpAbQSIEMYSx4UTvkcU+HOgowQKHDUHYhAqfvj0kwOASl0OFOI+QEexznCBC2hJlccpQiS3Sv0CrvZC1aV2r6QdqnU4xaXiklDHBntg1e2OP7Ia9sM8/a1WCGjUvRzRrdS13lYGzke2Pf1UWzQIH36qmngXymA+8GuTdDZjaLfS6vnx8/ra9sDVRmWRX7JX8X7Jn9kA3sMvvenWTb1008aORF3oxalDidzsawe5MPEF4cza2uBS2qgtjGMc09WMOi1jFBlJU/xRV3OJOkqUVaU1ar6dKLaFmGD9CynwBap6ZIA==
    AAACnnichVG7SgNBFD1Z389EbRSbxeCjkokEFCtREBvRJOYBiSy76yRZ3Je7k4AGsfcHLKwULETQ0g+w8Qcs/ASxVLCx8GazICrqXWbnzJl77pyZq7mm4QvGHiNSW3tHZ1d3T29f/8BgNDY0nPOdmqfzrO6YjlfQVJ+bhs2zwhAmL7geVy3N5Hltd6W5n69zzzcce0vsu3zbUiu2UTZ0VRClxMbySnJaLmlWo3Aolyp8Ty6JKheqklRicTbLgpB/gkQI4ghj04ndooQdONBRgwUOG4KwCRU+fUUkwOASt40GcR4hI9jnOEQvaWuUxSlDJXaX/hVaFUPWpnWzph+odTrFpOGRUsYke2CX7IXdsyv2xN5/rdUIajS97NOstbTcVaLHo5m3f1UWzQLVT9WfngXKWAi8GuTdDZjmLfSWvn5w8pJZTE82ptg5eyb/Z+yR3dEN7PqrfpHi6dM//GjkhV6MGpT43o6fIDc3myCcSsaXlsNWdWMcE5ihfsxjCWvYRJbqH+EC17iRZGlVWpc2WqlSJNSM4EtIhQ9u9Jki
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhVmRYlYBW0s8zAPiCHsrpO4ZF/sbgIx+AOCrSmsFCzED/ADbPwBi3yCWEawsfBmsyAajHeZnTNn7rlzZq5saarjMtYLCBOTU9MzwdnQ3PzCYjiytFxwzKat8LxiaqZdkiWHa6rB867qarxk2VzSZY0X5cbBYL/Y4rajmsaR27Z4RZfqhlpTFcklKqdUxWokxhLMi+goEH0Qgx9pM/KIY5zAhIImdHAYcAlrkODQV4YIBou4CjrE2YRUb5/jHCHSNimLU4ZEbIP+dVqVfdag9aCm46kVOkWjYZMyijh7Yfesz57ZA3tln3/W6ng1Bl7aNMtDLbeq4YvV3Me/Kp1mF6ffqrGeXdSw63lVybvlMYNbKEN966zbz+1l450NdsveyP8N67EnuoHRelfuMjx7PcaPTF7oxahB4u92jILCVkIknNmOpfb9VgWxhnVsUj+SSOEQaeSpfh2XuEJXCAoJYUdIDlOFgK9ZwY8QUl8k/JB+
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmQYlYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqiDXU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKqUSScG4rltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBJxyQfw==
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmPohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYLCGPjE5NTwenQzOzcfDiysFi0jaYl84JsqIZVlkSbq4rOC47iqLxsWlzUJJWXpMa+u19qcctWDP3QaZu8qol1XTlRZNEhKi/XNmuRGEswL6LDIOmDGPzIGJFHHOEYBmQ0oYFDh0NYhQibvgqSYDCJq6JDnEVI8fY5zhEibZOyOGWIxDboX6dVxWd1Wrs1bU8t0ykqDYuUUcTZC7tnffbMHtgr+/yzVser4Xpp0ywNtNyshS+W8x//qjSaHZx+q0Z6dnCCHc+rQt5Nj3FvIQ/0rbNuP7+bi3fW2C17I/83rMee6AZ6612+y/Lc9Qg/EnmhF6MGJX+3YxgUNxJJwtmtWHrPb1UQK1jFOvUjhTQOkEGB6tdxiSt0haCQELaF1CBVCPiaJfwIIf0FKTyQgA==
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmEolYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqiDXU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4VwqltnzWxXEGtaxSf1II4MDZFGk+g1c4go9ISgkhG0hPUwVAr5mBT9CyHwBK1yQgQ==
    AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhUmYohYBW0s8zAPiCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHh3syAajHeZnTNn7rlzZq5kqortMNYPCBOTU9MzwdnQ3PzCYjiytFyyjZYl86JsqIZVkUSbq4rOi47iqLxiWlzUJJWXpea+u19uc8tWDP3Q6Zi8pokNXTlRZNEhqiDXU/VIjCWYF9FRkPRBDH5kjcgjjnAMAzJa0MChwyGsQoRNXxVJMJjE1dAlziKkePsc5wiRtkVZnDJEYpv0b9Cq6rM6rd2atqeW6RSVhkXKKOLshd2zAXtmD+yVff5Zq+vVcL10aJaGWm7WwxerhY9/VRrNDk6/VWM9OzjBjudVIe+mx7i3kIf69llvUNjNx7sb7Ja9kf8b1mdPdAO9/S7f5Xj+eowfibzQi1GDkr/bMQpKW4kk4dx2LLPntyqINaxjk/qRRgYHyKJI9Ru4xBV6QlBICCkhPUwVAr5mBT9CyHwBLXyQgg==
    Model Trees:
    Constant prediction → Linear prediction
    Multivariate Trees:
    Axis-parallel splits → Multivariate linear splits

    View full-size slide

  89. Takeaways for Part 1
    “Of all the well-known learning methods, decision trees come closest to meeting
    the requirements for serving as an off-the-shelf procedure for data mining.”
    https://hastie.su.domains/Papers/ESLII.pdf
    Hastie, Tibshirani, Friedman, The elements of statistical learning (2009)
    1. Natural handling of data of “mixed” type
    2. Handling of missing values
    3. Robustness to outliers in input space
    4. Insensitive to monotone transformations of inputs
    5. Computational scalability
    6. Ability to deal with irrelevant inputs
    7. Ability to extract linear combinations of features
    8. Interpretability
    9. Predictive Power
    "

    View full-size slide

  90. Why we should learn decision tree ensembles?
    • What is a decision tree? (History and basic algorithm)
    • Why did it become so popular once in the past? (Advantages)
    Decision tree ensemble is one of the three key weapons in modern data
    science alongside linear models and deep learning.
    • Despite this, why did it fall out of favor? (Disadvantages)
    • What is ensemble learning? Why can it compensate for the shortcomings?
    • Practice and methods of decision tree ensembles
    [Part 1]
    [Part 2]

    View full-size slide

  91. https://itakigawa.github.io/
    A Modern Introduction to
    Decision Tree Ensembles
    Ichigaku Takigawa
    Institute for Liberal Arts and Sciences, Kyoto University
    Institute for Chemical Reaction Design and Discovery, Hokkaido University
    RIKEN Center for Advanced Intelligence Project
    Part 2

    View full-size slide

  92. Outline
    We will jointly learn decision trees and ensemble learning!
    1. Why we should learn decision trees and ensemble learning?
    2. History of Decision Trees
    3. Basics of Decision Trees
    4. Ensemble Learning: Additive ensembles and decision forests
    5. Random forest and Regularized gradient boosted trees
    6. Practical data analysis with decision tree ensembles
    Part 1
    Part 2

    View full-size slide

  93. Recap: Takeaways for Part 1
    “Of all the well-known learning methods, decision trees come closest to meeting
    the requirements for serving as an off-the-shelf procedure for data mining.”
    https://hastie.su.domains/Papers/ESLII.pdf
    Hastie, Tibshirani, Friedman, The elements of statistical learning (2009)
    1. Natural handling of data of “mixed” type
    2. Handling of missing values
    3. Robustness to outliers in input space
    4. Insensitive to monotone transformations of inputs
    5. Computational scalability
    6. Ability to deal with irrelevant inputs
    7. Ability to extract linear combinations of features
    8. Interpretability
    9. Predictive Power

    View full-size slide

  94. Why we should learn decision tree ensembles?
    • What is a decision tree? (History and basic algorithm)
    • Why did it become so popular once in the past? (Advantages)
    Decision tree ensemble is one of the three key weapons in modern data
    science alongside linear models and deep learning.
    • Despite this, why did it fall out of favor? (Disadvantages)
    • What is ensemble learning? Why can it compensate for the shortcomings?
    • Practice and methods of decision tree ensembles
    [Part 1]
    [Part 2]

    View full-size slide

  95. Drawback (1): Non-smoothness
    The function surface of a decision tree is not smooth. (a histogram
    approximation.) Because of this, the prediction capability is bounded.
    Decision
    Trees
    Random
    Forest
    Extra
    Trees
    Nearest
    Neighbors
    Logistic
    Regression
    SVM
    Gaussian
    Process
    Neural
    Network
    Non-smooth models Smooth models

    View full-size slide

  96. Drawback (1): Non-smoothness
    In high-dimensional real data, the assumption of smoothness isn’t that useful.
    Thus, this very conservative local constant approximation proves to be very
    effective.
    This drawback (non-smoothness) can be compensated by ensemble learning!
    For example
    It's very poor at capturing additive structures... (we get a unnecessarily deep tree)

    View full-size slide

  97. Drawback (2): Instability
    The structure of the tree (partitions) changes significantly with just a slight
    change in the data. Especially for the cases of many variables.
    A decision tree trained with a random 90% of the data on the left.

    View full-size slide

  98. Drawback (2): Instability
    Instability of decision trees
    High bias
    Low variance
    High bias
    High variance
    Low bias
    High variance
    Low bias
    Low variance
    Estimates
    Bias-variance tradeoff (a central problem in supervised learning)
    One cannot reduce both bias and variance simultaneously in predictions
    • Decision trees are a 'low bias, high variance' method, with the variance being especially large.
    • With just a slight difference in the data, you can get substantially different partitions, making
    the interpretation of decision trees somewhat unreliable.
    This drawback (instability) can be also compensated by ensemble learning!
    Truth

    View full-size slide

  99. Drawback (3) & (4) : Fragmentation & Bias
    • Data fragmentation
    There is a bias in variable selection, such as variables with a large number of
    categories being more likely to be chosen for splitting
    • Bias in splitter selection
    • Opting for multi-way splits instead of binary splits can easily lead to fragmented regions
    • When the number of samples in each region becomes too small, it can lead to overfitting
    • One can set a lower limit on the number of samples in each region (min_samples_leaf)
    as a safeguard.
    The more recursive splits are performed, each region contains fewer samples…
    (So we cannot do any complicated statistical processing at the leaf nodes)

    View full-size slide

  100. Model Trees: Constant prediction in leaves → Linear prediction
    Due to these downsides, easy extensions are impractical…
    Regression Trees M5 MARS
    Classification Trees OC1 Multivariate CART
    a broken line model that
    recursively bends straight lines
    Multivariate Trees: Axis-parallel splits → Multivariate linear splits
    Data (Regression)
    Data (Classification)
    ■ OC1/Oblique Tree (Murthy+ 1994)
    ■ Perceptron Tree (Utgoff , 1988)
    ■ Large Margin Tree (Wu+ 1999;
    Bennett+ 2000)
    ■ Margin Tree (Tibshirani & Hastie
    2007)
    ■ Geometric Decision Tree (Manwani
    & Sastry, 2012)
    ■ HHCART (Wickramarachchi+, 2016)
    ■ M5/M5’ (Quinlan, 1992; Wang &
    Witten 1997)
    ■ RETIS (Karalic & Cestnik 1991)
    ■ SECRET (Dobra & Gehrke, 2002)
    ■ SMOTI (Malerba+ 2004)
    ■ MAUVE (Vens & Blockeel, 2006)
    ■ LLRT (Vogel, Asparouhov, Scheffer,
    2007)
    ■ Cubist (Quinlan, 2011)

    View full-size slide

  101. Outline
    We will jointly learn decision trees and ensemble learning!
    1. Why we should learn decision trees and ensemble learning?
    2. History of Decision Trees
    3. Basics of Decision Trees
    4. Ensemble Learning: Additive ensembles and decision forests
    5. Random forest and Regularized gradient boosted trees
    6. Practical data analysis with decision tree ensembles
    Part 1
    Part 2

    View full-size slide

  102. Ensemble learning: Two heads are better than one
    Predicting by multiple different models rather than a single model
    “Ensemble” = A group of separate things that contribute to a coordinated whole

    Consensus
    building
    Single-model Prediction
    Multiple-model Prediction
    (Ensemble Learning)
    Model
    Model 1
    Model 2
    Model K

    View full-size slide

  103. A brief showcase of ensemble learning examples
    1. Voting/Averaging of different models
    For example, using the same data, we can train with Neural Net, SVM, and Random Forest,
    then use Voting or Averaging to produce the final prediction.
    2. Voting/Averaging of different instances of the same model
    For example, we can train five Neural Nets with the same structure but different random initial
    values, then use Voting or Averaging to produce the final prediction (’Deep Ensemble')
    3. Snapshot ensembling (for deep learning)
    After the Neural Network training has converged, the learning rate is altered to induce
    oscillations around a local solution, and snapshots of that process are taken for ensemble.
    e.g. torch.optim.swa_utils implements Stochastic Weight Averaging (SWA).
    4. Stacking/Blending
    Predictions from base models (like decision trees, SVMs, etc.) are used as input for a meta-
    model that makes the final prediction.

    View full-size slide

  104. sklearn.ensemble (Scikit-learn 1.3.0)
    https://scikit-learn.org/stable/modules/classes.html#module-sklearn.ensemble
    Methods we will cover hereafter
    A small note on Adaboost
    • Historically and theoretically
    important algorithm
    • Less hyperparameters but
    works well in practice
    • We will skip this because
    gradient boosted trees is a
    more flexible alternative to it.

    View full-size slide

  105. Outline
    We will jointly learn decision trees and ensemble learning!
    1. Why we should learn decision trees and ensemble learning?
    2. History of Decision Trees
    3. Basics of Decision Trees
    4. Ensemble Learning: Additive ensembles and decision forests
    5. Random forest and Regularized gradient boosted trees
    6. Practical data analysis with decision tree ensembles
    Part 1
    Part 2

    View full-size slide

  106. Decision Tree Ensemble: Learning from multiple decision trees
    We need multiple decision trees from a single dataset
    Dataset
    How do we generate
    multiple trees?
    • “Randomize” (Bagging, Random Forest)
    Introduce some randomness into the learning of each tree to diversify and generate a set of trees.
    • Repeat 'random resampling from data → tree generation'.
    • Repeat 'using different random numbers and data each time to generate a tree'.
    Sequentially update the ensemble by adding a decision tree one by one to improves the prediction accuracy.
    • “Optimize” (Boosting)

    View full-size slide

  107. Overcoming non-smoothness and instability
    DecisionTreeClassifier
    (max_depth=None,
    min_samples_leaf=1)
    DecisionTreeClassifier
    (max_depth=1)
    DecisionTreeClassifier
    (max_depth=5,
    min_samples_leaf=3)
    ExtraTreesClassifier
    (n_estimators=100,
    max_depth=None,
    min_samples_leaf=1)
    RandomForestClassifier
    (n_estimators=100,
    max_depth=None,
    min_samples_leaf=1)
    A single decision tree An ensemble of 100 decision trees
    XGBClassifier
    (n_estimators=100,
    max_depth=3)

    View full-size slide

  108. Basic concept for ensembling: Additive ensemble
    An additive model that makes predictions by 'adding' multiple decision trees
    T1
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuKfUE7lEyaaUOTzJBkhDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31ve/vMLa+sbmVnG7tLO7t39QPjxqmTjVlDVpLGLdCYlhgivWtNwK1kk0IzIUrB2Ob2d++5Fpw2PVsJOEBZIMFY84JdZJD40+7pcrftWfA60SnJMK5Kj3y9+9QUxTyZSlghjTxX5ig4xoy6lg01IvNSwhdEyGrOuoIpKZIJu/OkVnThmgKNZulEVz9fdFRqQxExm6TUnsyCx7M/FfL5RLyTa6DjKuktQyRRfBUSqQjdGsBzTgmlErJo4Qqrn7HdER0YRa11bJlYKXK1glrYsq9qv4/rJSu8nrKcIJnMI5YLiCGtxBHZpAYQjP8AKv3pP35r17H4vVgpffHMMfeJ8/E+CUIg==
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuKfUE7lEyaaUOTzJBkhDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31ve/vMLa+sbmVnG7tLO7t39QPjxqmTjVlDVpLGLdCYlhgivWtNwK1kk0IzIUrB2Ob2d++5Fpw2PVsJOEBZIMFY84JdZJD40+7pcrftWfA60SnJMK5Kj3y9+9QUxTyZSlghjTxX5ig4xoy6lg01IvNSwhdEyGrOuoIpKZIJu/OkVnThmgKNZulEVz9fdFRqQxExm6TUnsyCx7M/FfL5RLyTa6DjKuktQyRRfBUSqQjdGsBzTgmlErJo4Qqrn7HdER0YRa11bJlYKXK1glrYsq9qv4/rJSu8nrKcIJnMI5YLiCGtxBHZpAYQjP8AKv3pP35r17H4vVgpffHMMfeJ8/E+CUIg==
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuKfUE7lEyaaUOTzJBkhDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31ve/vMLa+sbmVnG7tLO7t39QPjxqmTjVlDVpLGLdCYlhgivWtNwK1kk0IzIUrB2Ob2d++5Fpw2PVsJOEBZIMFY84JdZJD40+7pcrftWfA60SnJMK5Kj3y9+9QUxTyZSlghjTxX5ig4xoy6lg01IvNSwhdEyGrOuoIpKZIJu/OkVnThmgKNZulEVz9fdFRqQxExm6TUnsyCx7M/FfL5RLyTa6DjKuktQyRRfBUSqQjdGsBzTgmlErJo4Qqrn7HdER0YRa11bJlYKXK1glrYsq9qv4/rJSu8nrKcIJnMI5YLiCGtxBHZpAYQjP8AKv3pP35r17H4vVgpffHMMfeJ8/E+CUIg==
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GXRjcuKfUE7lEyaaUOTzJBkhDL2E9zq3p249Wfc+iWm7Sy09cCFwzn3ci4nTAQ31ve/vMLa+sbmVnG7tLO7t39QPjxqmTjVlDVpLGLdCYlhgivWtNwK1kk0IzIUrB2Ob2d++5Fpw2PVsJOEBZIMFY84JdZJD40+7pcrftWfA60SnJMK5Kj3y9+9QUxTyZSlghjTxX5ig4xoy6lg01IvNSwhdEyGrOuoIpKZIJu/OkVnThmgKNZulEVz9fdFRqQxExm6TUnsyCx7M/FfL5RLyTa6DjKuktQyRRfBUSqQjdGsBzTgmlErJo4Qqrn7HdER0YRa11bJlYKXK1glrYsq9qv4/rJSu8nrKcIJnMI5YLiCGtxBHZpAYQjP8AKv3pP35r17H4vVgpffHMMfeJ8/E+CUIg==
    T2
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047JiX9AOJZNm2tAkMyQZoYz9BLe6dydu/Rm3folpOwttPXDhcM69nMsJE8GN9bwvtLa+sbm1Xdgp7u7tHxyWjo5bJk41ZU0ai1h3QmKY4Io1LbeCdRLNiAwFa4fj25nffmTa8Fg17CRhgSRDxSNOiXXSQ6Nf7ZfKXsWbA68SPydlyFHvl757g5imkilLBTGm63uJDTKiLaeCTYu91LCE0DEZsq6jikhmgmz+6hSfO2WAo1i7URbP1d8XGZHGTGToNiWxI7PszcR/vVAuJdvoOsi4SlLLFF0ER6nANsazHvCAa0atmDhCqObud0xHRBNqXVtFV4q/XMEqaVUrvlfx7y/LtZu8ngKcwhlcgA9XUIM7qEMTKAzhGV7gFT2hN/SOPharayi/OYE/QJ8/FXSUIw==
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047JiX9AOJZNm2tAkMyQZoYz9BLe6dydu/Rm3folpOwttPXDhcM69nMsJE8GN9bwvtLa+sbm1Xdgp7u7tHxyWjo5bJk41ZU0ai1h3QmKY4Io1LbeCdRLNiAwFa4fj25nffmTa8Fg17CRhgSRDxSNOiXXSQ6Nf7ZfKXsWbA68SPydlyFHvl757g5imkilLBTGm63uJDTKiLaeCTYu91LCE0DEZsq6jikhmgmz+6hSfO2WAo1i7URbP1d8XGZHGTGToNiWxI7PszcR/vVAuJdvoOsi4SlLLFF0ER6nANsazHvCAa0atmDhCqObud0xHRBNqXVtFV4q/XMEqaVUrvlfx7y/LtZu8ngKcwhlcgA9XUIM7qEMTKAzhGV7gFT2hN/SOPharayi/OYE/QJ8/FXSUIw==
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047JiX9AOJZNm2tAkMyQZoYz9BLe6dydu/Rm3folpOwttPXDhcM69nMsJE8GN9bwvtLa+sbm1Xdgp7u7tHxyWjo5bJk41ZU0ai1h3QmKY4Io1LbeCdRLNiAwFa4fj25nffmTa8Fg17CRhgSRDxSNOiXXSQ6Nf7ZfKXsWbA68SPydlyFHvl757g5imkilLBTGm63uJDTKiLaeCTYu91LCE0DEZsq6jikhmgmz+6hSfO2WAo1i7URbP1d8XGZHGTGToNiWxI7PszcR/vVAuJdvoOsi4SlLLFF0ER6nANsazHvCAa0atmDhCqObud0xHRBNqXVtFV4q/XMEqaVUrvlfx7y/LtZu8ngKcwhlcgA9XUIM7qEMTKAzhGV7gFT2hN/SOPharayi/OYE/QJ8/FXSUIw==
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFl047JiX9AOJZNm2tAkMyQZoYz9BLe6dydu/Rm3folpOwttPXDhcM69nMsJE8GN9bwvtLa+sbm1Xdgp7u7tHxyWjo5bJk41ZU0ai1h3QmKY4Io1LbeCdRLNiAwFa4fj25nffmTa8Fg17CRhgSRDxSNOiXXSQ6Nf7ZfKXsWbA68SPydlyFHvl757g5imkilLBTGm63uJDTKiLaeCTYu91LCE0DEZsq6jikhmgmz+6hSfO2WAo1i7URbP1d8XGZHGTGToNiWxI7PszcR/vVAuJdvoOsi4SlLLFF0ER6nANsazHvCAa0atmDhCqObud0xHRBNqXVtFV4q/XMEqaVUrvlfx7y/LtZu8ngKcwhlcgA9XUIM7qEMTKAzhGV7gFT2hN/SOPharayi/OYE/QJ8/FXSUIw==
    Preliminary: What does 'adding decision trees (classification or regression trees)' mean?
    AAACJnicbVDLSsNAFJ3UV62vqks3g0WoCCGRom4KRTduhAp9QRPDZDpth84kYWYilpC/8CP8Bre6difiTv/E6UOwrQcuHM65l3vv8SNGpbKsTyOztLyyupZdz21sbm3v5Hf3GjKMBSZ1HLJQtHwkCaMBqSuqGGlFgiDuM9L0B1cjv3lPhKRhUFPDiLgc9QLapRgpLXl5swwdxKI+8ix4Ah0Zcy+hZTu9u/nVKax5tOj4PHlIj718wTKtMeAisaekAKaoevlvpxPimJNAYYakbNtWpNwECUUxI2nOiSWJEB6gHmlrGiBOpJuM/0rhkVY6sBsKXYGCY/XvRIK4lEPu606OVF/OeyPxP68dq+6Fm9AgihUJ8GRRN2ZQhXAUEuxQQbBiQ00QFlTfCnEfCYSVjnJmi89TnYk9n8AiaZya9plZui0VKpfTdLLgAByCIrDBOaiAa1AFdYDBI3gGL+DVeDLejHfjY9KaMaYz+2AGxtcP5lSkyw==
    = ↵0 +
    M
    X
    i=1
    ↵iTi(x)
    AAACZXicbVFBSxtBGJ3d2Gq1tqlKLx46NJZakLBbxPYSCO3Fi2AhUSGbLt9OvpjBmd115ts2Ydlf6cljT/0NvXUSIxr1wcDjve/xfbxJciUtBcG159eWnj1fXnmxuvZy/dXr+puNE5sVRmBXZCozZwlYVDLFLklSeJYbBJ0oPE0uvk/9019orMzSDk1y7Gs4T+VQCiAnxXXd4pEtdFzKVlD9POKdj7HcjRJdjqtPezy6LGAQEY6p/D1Cg3zH+cGtz10WVD6CONhbCN4Zknfu5J0qrjeCZjADf0zCOWmwOY7j+p9okIlCY0pCgbW9MMipX4IhKRRWq1FhMQdxAefYczQFjbZfzmqp+AenDPgwM+6lxGfq/UQJ2tqJTtykBhrZh95UfMrrFTT82i9lmheEqbhZNCwUp4xPO+YDaVCQmjgCwkh3KxcjMCDI/cTClkRPOwkfNvCYnHxuhgfN/R/7jfa3eTsrbJu9Z7ssZF9Ymx2yY9Zlgl2xf57v1by//rq/5b+9GfW9eWaTLcB/9x8im7Zk
    =
    M
    X
    i=0
    T0
    i
    (x), where T0
    0
    (x) = ↵0, T0
    i
    (x) = ↵iTi(x)

    View full-size slide

  109. Adding decision trees → For classification trees, add using class probabilities
    Additive ensemble of classification trees
    = + +
    + + +
    RandomForestClassifier
    (n_estimators=6,
    max_leaf_nodes=4) 6 × DecisionTreeClassifier(max_leaf_nodes=4)
    Set the number of regions to 4
    After ensemble, the number
    of regions increases
    significantly more than 4 due
    to combinations!

    View full-size slide

  110. Adding decision trees → For regression trees, add them directly.
    Additive ensemble of regression trees
    RandomForestRegressor
    (n_estimators=6,
    max_leaf_nodes=8) 6 × DecisionTreeRegressor(max_leaf_nodes=8)
    = + +
    + + +
    Set the number of regions to 8
    After ensemble, the number
    of regions increases
    significantly more than 8 due
    to combinations!

    View full-size slide

  111. A high-level glance: “Randomize” type
    RandomForestRegressor
    (n_estimators=10, max_leaf_nodes=12)
    10 x
    DecisionTreeRegressor(max_leaf_nodes=12)
    Take the average Independently generate N decision trees
    Independently generate N decision trees
    ExtraTreesRegressor
    (n_estimators=10, max_leaf_nodes=32)
    10 x
    ExtraTreeRegressor(max_leaf_nodes=32)
    Take the average
    The 'Randomize' type (such as Bagging and Random Forest) leads to 'smoothing'.

    View full-size slide

  112. A high-level glance:“Optimize” type
    The 'Optimize' type (like Boosting) aims for 'optimization'
    Learning rate
    (Step size)

    View full-size slide

  113. Wrap up for now: “Randomize” vs “Optimize”
    ‣ RandomForestRegressor
    max_depth=None, max_leaf_nodes=None
    ‣ ExtraTreesRegressor
    max_depth=None, max_leaf_nodes=None
    ‣ GradientBoostingRegressor
    max_depth=3, max_leaf_nodes=None
    ‣ HistGradientBoostingRegressor
    max_depth=None, max_leaf_nodes=31
    ‣ XGBRegressor
    max_depth=3
    ‣ LGBMRegressor
    num_leaves=31, max_depth=-1
    • Use a large base tree (e.g. Fully-grown tree)
    • Each decision tree is trained to overfit.
    • Use a small base tree (Size-bounded tree)
    • Each decision tree is trained to underfit.
    “Weak" learner
    Scikit default values (Examples of Regressor)
    • Random Projection Ensemble (2017)
    • Random Composite Forest (2016)
    • Random Rotation Ensemble (2016)
    • Oblique Random Forests (2011)
    • Rotation Forests (2006)
    Other
    examples
    • Globally Induced Forest (ICML2017)
    • Regularized Greedy Forest (TPAMI 2014)
    • CatBoost (NeurIPS2018)
    • “Randomize” (Bagging, Random Forest)
    • “Optimize” (Boosting)
    Other
    examples

    View full-size slide

  114. Integrating “Randomize” and “Optimize” strategies
    Modern implementations of the
    “Optimize” type can also use
    the “Randomize” type
    (Stochastic Gradient Boosting)
    • Friedman’s Stochastic Gradient Boosting (Comput Stat Data Anal, 2002)
    • Breiman’s Iterated Bagging (Mach Learn, 2001)
    • Ensemble on Random Patches (Louppe & Geurts, ECML PKDD 2012)
    The basic idea is to introduce 'Randomize' into the 'Optimize' process.

    View full-size slide

  115. Decision Tree Ensemble: Learning from multiple decision trees
    We need multiple decision trees from a single dataset
    • “Randomize” (Bagging, Random Forest)
    Dataset
    How do we generate
    multiple trees?
    Introduce some randomness into the learning of each tree to diversify and generate a set of trees.
    • Repeat 'random resampling from data → tree generation'.
    • Repeat 'using different random numbers and data each time to generate a tree'.
    Sequentially update the ensemble by adding a decision tree one by one to improves the prediction accuracy.
    • “Optimize” (Boosting)

    View full-size slide

  116. Preparation: Bootstrap and variance reduction
    Bootstrap = Resampling from the empirical distribution
    AAACt3ichVHLTttAFD2YPniUJi0bJDajRlRBasM1ReWxQrBhySuAhGlkuxMY4YyNPYkIET/AD1Cpq1ZigfoB3XXTBfwACz4BsaRSN130xrFUtQh6LXvOnHvP9Zm5XhSoxBBddlndDx4+etzT29f/ZOBpLv/s+VoS1mNflv0wCOMNz01koLQsG2UCuRHF0q15gVz3dufb+fWGjBMV6lXTjORWzd3Wqqp81zBVyY8X7df2mB59p4VjQmGPSeG4URSH+4JKb95OCeeVKHZyjtJV0xwVlXyBSpSGuA3sDBSQxWKY/woH7xHCRx01SGgYxgFcJPxswgYhYm4LLeZiRirNSxyij7V1rpJc4TK7y99t3m1mrOZ9u2eSqn3+S8BvzEqBEbqgU7qhc/pCV/Trzl6ttEfbS5NXr6OVUSV3NLTy87+qGq8GO39U93o2qGIq9arYe5Qy7VP4HX3j4PhmZWZ5pPWSPtM1+/9El/SdT6AbP/yTJbn88R4/HnvhG+MB2f+O4zZYGy/ZjJcmCrNz2ah6MIwXKPI8JjGLBSyizP0/4BvOcG5NWxWrau10Sq2uTDOIv8La+w2awaAA
    AAACqXichVHLSsNQED3GV62vqhvBjVgqglAnIiiuim5cWrW1+ECS69UG0yQkaaEWf8CF2wquFFyIn+GmP+DCTxCXFdy4cJIGREt1QnLPPTNn7rkZ3TENzyd66VK6e3r7+mMD8cGh4ZHRxNh43rPLrpA5YZu2W9A1T5qGJXO+4Zuy4LhSK+mm3NXP1oP8bkW6nmFbO37VkYcl7dQyTgyh+QGlLqh0lEhSmsKYbgdqBJKIYtNONHCAY9gQKKMECQs+YxMaPH72oYLgMHeIGnMuIyPMS1wgztoyV0mu0Jg94+8p7/Yj1uJ90NML1YJPMfl1WTmNFD3TAzWpQY/0Sp8de9XCHoGXKq96Syudo9HLye2Pf1UlXn0Uv1V/evZxgpXQq8HenZAJbiFa+sp5vbm9upWqzdIdvbH/W3qhJ76BVXkX91m5dfOHH529dP5jQT6q4BGqvwfWDvKLaZVxdimZWYuGGcMUZjDHE1tGBhvYRI5PKOIKdVwr80pWKSh7rVKlK9JM4Eco4gv9upe5
    AAACI3icbZDLSgMxFIYz9VbrbdSlm2ARBKHMSFE3haIblxXsBTrjkEkzbdpkZkgy0jLMQ/gQPoNbXbsTNy5c+Caml4Vt/SHw5z/ncJLPjxmVyrK+jNzK6tr6Rn6zsLW9s7tn7h80ZJQITOo4YpFo+UgSRkNSV1Qx0ooFQdxnpOkPbsb15iMRkkbhvRrFxOWoG9KAYqR05Jlnjo9EOsw8CivQCQTCqZ2ltpU5MuFe2q/Y2cP4Code3zOLVsmaCC4be2aKYKaaZ/44nQgnnIQKMyRl27Zi5aZIKIoZyQpOIkmM8AB1SVvbEHEi3XTyqQye6KQDg0joEyo4Sf9OpIhLOeK+7uRI9eRibRz+V2snKrhyUxrGiSIhni4KEgZVBMeEYIcKghUbaYOwoPqtEPeQJqM0x7ktPs80E3uRwLJpnJfsi1L5rlysXs/o5MEROAanwAaXoApuQQ3UAQZP4AW8gjfj2Xg3PozPaWvOmM0cgjkZ3790HKSw
    ¯
    xi =
    1
    10
    10
    X
    j=1
    xj
    In the case of a normal distribution
    Resampling and taking the average leads to a reduction in variance
    • Sample N items from the N data points, allowing for duplicates (sampling with replacement)
    • Some are duplicated and some are not selected, resulting in a random subset
    • The probability that a sample is never selected:
    AAACD3icbVC7TsMwFHXKq5RXgIGBxaJCYqiqpOI1VrAwFom2SG0UOa7TWrWdyHZQq6gfwTewwsyGWPkERv4Ep81AW45k6eice3SvTxAzqrTjfFuFldW19Y3iZmlre2d3z94/aKkokZg0ccQi+RggRRgVpKmpZuQxlgTxgJF2MLzN/PYTkYpG4kGPY+Jx1Bc0pBhpI/n20ch3K3Dk1yqw24u0ynjqOhPfLjtVZwq4TNyclEGOhm//mDxOOBEaM6RUx3Vi7aVIaooZmZS6iSIxwkPUJx1DBeJEeen0AxN4apQeDCNpntBwqv5NpIgrNeaBmeRID9Sil4n/eZ1Eh9deSkWcaCLwbFGYMKgjmLUBe1QSrNnYEIQlNbdCPEASYW06m9sS8KwTd7GBZdKqVd3L6sX9ebl+k7dTBMfgBJwBF1yBOrgDDdAEGEzAC3gFb9az9W59WJ+z0YKVZw7BHKyvXypLm2w=
    x1, x2, . . . , x10 AAAB/XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PQi8cI5gHJEmYns8mQmdllZlYMS/AbvOrZm3j1Wzz6J84mezCJBQ1FVTfdXUHMmTau++0UVlbX1jeKm6Wt7Z3dvfL+QVNHiSK0QSIeqXaANeVM0oZhhtN2rCgWAaetYHSb+a1HqjSL5IMZx9QXeCBZyAg2Vmp1A6zSp0mvXHGr7hRomXg5qUCOeq/80+1HJBFUGsKx1h3PjY2fYmUY4XRS6iaaxpiM8IB2LJVYUO2n03Mn6MQqfRRGypY0aKr+nUix0HosAtspsBnqRS8T//M6iQmv/ZTJODFUktmiMOHIRCj7HfWZosTwsSWYKGZvRWSIFSbGJjS3JRBZJt5iAsukeVb1LqsX9+eV2k2eThGO4BhOwYMrqMEd1KEBBEbwAq/w5jw7786H8zlrLTj5zCHMwfn6BXKplmE=
    ¯
    x
    AAACC3icbVDLSsNAFJ3UV62vaJduBovgqiTia1l047KCfUATymQ6aYfOTMLMRAghn+A3uNW1O3HrR7j0T5y0WdjWAxcO59zLuZwgZlRpx/m2KmvrG5tb1e3azu7e/oF9eNRVUSIx6eCIRbIfIEUYFaSjqWakH0uCeMBIL5jeFX7viUhFI/Go05j4HI0FDSlG2khDu+5pyomCXigRztw8c518aDecpjMDXCVuSRqgRHto/3ijCCecCI0ZUmrgOrH2MyQ1xYzkNS9RJEZ4isZkYKhAJtHPZs/n8NQoIxhG0ozQcKb+vcgQVyrlgdnkSE/UsleI/3mDRIc3fkZFnGgi8DwoTBjUESyagCMqCdYsNQRhSc2vEE+QqUGbvhZSAl504i43sEq65033qnn5cNFo3ZbtVMExOAFnwAXXoAXuQRt0AAYpeAGv4M16tt6tD+tzvlqxyps6WID19QtS/psh

    1
    10
    reduced to
    one tenth

    View full-size slide

  117. Bagging (Bootstrap Aggregating)
    Method 1: Bagging (Bootstrap Aggregating)
    Decision trees have a high prediction variance = If the samples change slightly, the tree structure
    (or partitions) also changes significantly. Bagging is particularly effective for such a “have high
    variance and low bias” methods.
    tree =
    DecisionTreeClassifier()
    BaggingClassifier
    (tree, n_estimators=5)
    BaggingClassifier
    (tree, n_estimators=10)
    BaggingClassifier
    (tree, n_estimators=100)
    BaggingClassifier
    (tree, n_estimators=500)
    • Generate multiple bootstrap samples and average the models trained on each one.
    (When using a random subset without duplicates, it's called 'Pasting').
    • A simple and powerful general method to reduce the prediction variance.

    View full-size slide

  118. Problems with decision tree bagging
    The importance of diversity: The effect of bagging is highest when each weak learner is
    independent. On the other hand, when decision trees are simply bagged, individual trees tend to
    be highly correlated with each other.
    Basic Strategy of “Randomize”-type Ensembles
    AAACpnichVE9S8NQFD2NX7V+tOoiuBRLpVN5FUFxKnVxkn7YD6ilJvFVQ9MkJGmhFv+AroqDk4KD+DNc+gcc+hPEsYKLgzdpQLS03pC88869577zciVDVSybsZ5PmJicmp7xzwbm5hcWg6Gl5YKlN02Z52Vd1c2SJFpcVTSetxVb5SXD5GJDUnlRqu85+WKLm5aia4d22+CVhniqKTVFFm2iMqlqKMLizI3wMEh4IAIv0nqoiyOcQIeMJhrg0GATViHCoqeMBBgM4iroEGcSUtw8xwUCpG1SFacKkdg6fU9pV/ZYjfZOT8tVy3SKSq9JyjCi7JU9sT7rsmf2xr5G9uq4PRwvbVqlgZYb1eDlau7zX1WDVhtnP6qxnm3UsON6Vci74TLOLeSBvnV+28/tZqOdDfbA3sn/PeuxF7qB1vqQHzM8ezfGj0ReRv8xJ+9V0AgTfwc2DAqb8QThzFYkmfKG6cca1hGjiW0jiX2kkacTOK5wjRshJhwIeaE4KBV8nmYFv0I4/gZs35cc
    AAACrXichVHLTsJAFL3UF+ID1I2JGyLBuMIpMdG4IrpxycMCCa+0dcAJfaUdSJD4Ay7duMCNJi6Mn+GGH3DBJxiXmLhx4W1pYpSAt2nnzLn33J6Zq1gaczghg4AwMzs3vxBcDC0tr6yGI2vrecds2SqVVFMz7aIiO1RjBpU44xotWjaVdUWjBaV54uYLbWo7zDTOeMeiFV1uGKzOVJkjVSw7rKHL1WQtEiMJ4kV0HIg+iIEfaTPShzKcgwkqtEAHCgZwxBrI4OBTAhEIWMhVoIucjYh5eQpXEEJtC6soVsjINvHbwF3JZw3cuz0dT63iXzR8bVRGIU5eyRMZkj55Jm/ka2KvrtfD9dLBVRlpqVULX2/mPv9V6bhyuPhRTfXMoQ6HnleG3i2PcU+hjvTty9th7igb7+6QB/KO/u/JgLzgCYz2h/qYodneFD8Kepl8Y27er8ARin8HNg7yyYSIOLMfSx37wwzCFmzDLk7sAFJwCmmQvEndQA/uhD1BEspCdVQqBHzNBvwKofENqlyaHQ==
    AAACqXichVHLSsNAFD2Nr1ofrboR3BRLRRDKVATFVdGNyz7sA9tSkjhtQ9MkJGmhFn/AhdsKrhRciJ/hpj/gop8gLiu4ceFNGhAt1huSOXPuPXfO5EqGqlg2YwOfMDU9Mzvnnw8sLC4tB0MrqzlLb5kyz8q6qpsFSbS4qmg8ayu2yguGycWmpPK81Dh28vk2Ny1F107tjsHLTbGmKVVFFm2HKpl1vRKKsBhzIzwO4h6IwIukHuqjhHPokNFCExwabMIqRFj0FBEHg0FcGV3iTEKKm+e4RIC0LariVCES26BvjXZFj9Vo7/S0XLVMp6j0mqQMI8pe2CMbsj57Yq/s889eXbeH46VDqzTScqMSvFrPfPyratJqo/6tmujZRhUHrleFvBsu49xCHunbF71h5jAd7W6xe/ZG/u/YgD3TDbT2u/yQ4unbCX4k8vL3H3PyXgWNMP57YOMgtxuLE07tRRJH3jD92MAmtmli+0jgBElk6YQ6rtHDjbAjpISCcDYqFXyeZg0/QpC/AAHtmJ0=
    1. We need to make the trees somehow dissimilar as possible.
    2. Besides, each tree needs to have a small training error.
    (We can reduce the training error by increasing the depth of a tree)
    The effect of bagging:
    The variance of the average of random variables, each with a variance of
    (independent) (correlated at )

    View full-size slide

  119. Method 2: Random Subspace and Random Patches
    • Limit the set of available variables to a random subset for each decision tree training.
    • Also known as feature bagging or attribute bagging.
    • Ensemble method applicable beyond just decision trees.
    Random Subspace Method (Ho, 1998)
    Random Patches Method (Louppe & Geurts, 2012)
    Another idea: Instead of taking random subsets of data for each decision tree learning,
    one can take random subsets of features (variables)
    → Row sampling (data instances) vs. Column sampling (features/variables)
    • Perform bagging and feature bagging simultaneously.
    • Random patch = random subsets of both samples and features.
    • Ensemble method applicable beyond just decision trees.

    View full-size slide

  120. Method 3: Random Forest
    • At each best split search, select a random subset of input variables.
    • This reduces tree correlation and enhances the effects of bagging.
    Random Forest (Breiman, 2001)
    2. Search for the best split using
    only the selected variables
    3. Execute the best split
    … and recursively apply
    to its children L & R.
    At every call of best split search, do the following:
    2
    4
    1
    3
    1. Select a random subset
    of variables.

    View full-size slide

  121. • Even if we randomly constraint the variable set during each split search and we sometimes
    miss the relevant variables, deeper trees can achieve low training errors anyway.
    • Inherently effective with multivariate methods (univariate results similar to bagging).
    • Random variable selection uses sampling without replacement (Fisher–Yates Shuffle).
    Random Forest Essentials
    •sqrt(n_features)
    •log2(n_features)
    •n_features
    ‣ Number of trees (n_estimators)
    ‣ Number of features to select (max_features)
    Classification tree common settings (Breiman's original)
    Plain decision tree bagging (without variable subset selection)
    Note: In scikit-learn, RandomForestClassifier defaults to 'sqrt' for max_features, while
    RandomForestRegressor defaults to 'n_features,' so please be aware of this distinction
    Two key hyperparameters
    bagging = sampling with replacement

    View full-size slide

  122. Random Forest is similar to Nearest Neighbors
    KNeighborsClassifier
    KNeighborsRegressor
    RandomForestClassifier
    RandomForestRegressor
    Lin and Jeon, Random Forests and Adaptive Nearest Neighbors, Journal of the American Statistical
    Association, 101(474): 578-590, 2006.

    View full-size slide

  123. Method 4: ExtraTrees
    2
    4
    1
    3
    • A Random Forest variant
    • replaces the best split search on selected variables with a random split
    ExtraTrees (Geurts, Ernst, Wehenkel, 2006)
    1. Select a random subset
    of variables.
    2. Taking a random split point for
    each selected variable (uniformly
    distributed within the data range)
    3. Execute the best split
    … and recursively apply
    to its children L & R.
    At every call of best split search, do the following:

    View full-size slide

  124. • Essentially shares properties with Random Forest, being effective primarily with multivariate
    data, with the only difference being the substitution of complete random splits for the exhaustive
    search for the best split.
    • Since the exhaustive search for the best split is the most time-consuming part of decision tree
    learning, even with large or high-dimensional data, it's extremely fast.
    • Decision trees can reduce training errors even with this extremely randomized setting, as long as
    the trees are made deep, and since the split points are random, the trees are dissimilar.
    • Predictive accuracy is equivalent to or even better than Random Forest, especially in regression.
    • Note that the default setting in Scikit-learn is bootstrap=False, which means it doesn't perform
    bootstrapping (bagging), making it more prone to overfitting, so be cautious.
    ExtraTrees Essentials
    While Random Forest without bagging doesn't make much sense, Extra Trees without
    bagging is an extremely promising and interesting method based on empirical evidence

    View full-size slide

  125. • Random forests usually assume bagging as a major premise, and since they are trained on a
    random subset of the training data, they are almost impossible to overfit in principle.
    • Conversely, if you want the training error to be zero anyway in special situations, they are
    impossible to overfit.
    Noteworthy Property 1
    • Extra Trees without bagging (Scikit's default setting) generates many completely different
    decision trees with training error 0 because the split points are random
    • Very interestingly, when taking this ensemble of many overfitting trees, there are many cases
    where the training error is (nearly) zero and the test error is also small (the well-known "benign
    overfitting" phenomenon).
    → This is why bagging is off by default in Scikit
    • Extra Trees with bagging on should always be tried, but interestingly, experience has shown
    that the test error is often smaller with "bagging off”.
    The "benign overfitting (or harmless overfitting)“ phenomenon

    View full-size slide

  126. Random Forest is the same as decision tree bagging for univariate (1-dimensional) data
    because there is only one variable to choose from, but this is not the case with Extra Trees!
    Noteworthy Property 2
    ExtraTreesRegressor(n_estimators=10)
    RandomForestRegressor(n_estimators=10)
    Since the "Random Split Points" make the number of split point candidates less than finite, the
    ensemble has the characteristics of a natural continuous interpolation between sample points,
    and is recommended over Random Forest, especially for "regression"!
    The “natural conservative interpolation” phenomenon

    View full-size slide

  127. Noteworthy Property 2
    KernelRidge(kernel='rbf', alpha=0.05, gamma=0.1)
    KernelRidge(kernel='rbf', alpha=1e-4, gamma=0.1)
    KernelRidge(kernel='rbf', alpha=1e-4, gamma=2.0)
    ExtraTreesRegressor(n_estimators=50)
    DecisionTreeRegressor()
    conservative
    conservative
    ʁ
    The “natural convervative interpolation” phenomenon
    groundless
    groundless
    groundless

    View full-size slide

  128. Noteworthy Property 2
    Text text
    Geurts, Ernst, Wehenkel, Extremely randomized trees. Mach Learn 63, 3–42 (2006).
    https://doi.org/10.1007/s10994-006-6226-1

    View full-size slide

  129. • Random Forest is the same as decision tree bagging for univariate (1D)
    • ExtraTrees is more non-trivial and interesting in higher dimensions! (If it were just a random
    partition, the approximation would be too inefficient in higher dimensions.)
    Bagging vs ExtraTrees without bagging
    DecisionTreeRegressor
    RandomForestRegressor

    (n_estimators=50)
    ExtraTreesRegressor

    (n_estimators=50)
    depth n_leaves depth n_leaves
    KernelRidge

    View full-size slide

  130. Input Space paritions of
    tree ensembles
    Refinement of partitions and randomness
    Random forests #tree=10
    Extra trees #tree=10
    Diversify the trees by adding some randomness
    Random subsamples, Random subspaces, Random thresholds,
    Random rotations, Random projections, ...

    View full-size slide

  131. Decision Tree Ensemble: Learning from multiple decision trees
    We need multiple decision trees from a single dataset
    • “Randomize” (Bagging, Random Forest)
    Dataset
    How do we generate
    multiple trees?
    Introduce some randomness into the learning of each tree to diversify and generate a set of trees.
    • Repeat 'random resampling from data → tree generation'.
    • Repeat 'using different random numbers and data each time to generate a tree'.
    Sequentially update the ensemble by adding a decision tree one by one to improves the prediction accuracy.
    • “Optimize” (Boosting)

    View full-size slide

  132. A high-level glance: “Randomize” vs “Optimize”
    The 'Optimize' type (like Boosting) aims for 'optimization'
    Learning rate
    (Step size)

    View full-size slide

  133. Preparation: supervised learning as optimization
    Change the parameter so that the loss becomes smaller
    AAACp3ichVG7SgNBFD2u73fURrBZDIqChLtBUQRBtLEzPhIDRsLuOtHFfbE7CWiwF3/AwkrBQgRb7W38AQs/QSwVbCy82SyIinqX2Tlz5p47Z+Yavm2FkuixQWlsam5pbWvv6Ozq7ulN9PXnQq8cmCJrerYX5A09FLbliqy0pC3yfiB0x7DFhrG3WNvfqIggtDx3Xe77YsvRd1yrZJm6ZKqYGC4YTrUgd4XUD9U5dawOi9qEGqP0eDGRpBRFof4EWgySiCPjJW5QwDY8mCjDgYALydiGjpC/TWgg+MxtocpcwMiK9gUO0cHaMmcJztCZ3eP/Dq82Y9blda1mGKlNPsXmEbBSxQg90CW90D1d0RO9/1qrGtWoednn2ahrhV/sPR5ce/tX5fAssfup+tOzRAkzkVeLvfsRU7uFWddXDk5e1mZXR6qjdE7P7P+MHumOb+BWXs2LFbF6+ocfg73wi3GDtO/t+Aly6ZQ2laKVyeT8QtyqNgxhGGPcj2nMYwkZZLn+Ea5xg1tlXFlWckq+nqo0xJoBfAlF/wDm+Zyq
    ✓ = (✓1, ✓2)
    AAACi3ichVHLTsJAFL3UF6II6sbEDZFgXJFbH9EQF0Rj4pKHPBIgpK0DNJS2aQcSJP6ASzcucKOJC+MH+AFu/AEXfIJxiYkbF15KE6NEvM10zpy5586ZubKpqTZH7HmEicmp6RnvrG9u3r8QCC4uZW2jaSksoxiaYeVlyWaaqrMMV7nG8qbFpIassZxcPxzs51rMslVDP+Ftk5UaUlVXK6oicaLyRV5jXCqL5WAYo+hEaBSILgiDGwkj+AhFOAUDFGhCAxjowAlrIIFNXwFEQDCJK0GHOIuQ6uwzOAcfaZuUxShDIrZO/yqtCi6r03pQ03bUCp2i0bBIGYIIvuA99vEZH/AVP/+s1XFqDLy0aZaHWmaWAxcr6Y9/VQ2aOdS+VWM9c6jAnuNVJe+mwwxuoQz1rbOrfjqWinTW8RbfyP8N9vCJbqC33pW7JEt1x/iRyQu9GDVI/N2OUZDdjIo7UUxuh+MHbqu8sAprsEH92IU4HEMCMk4fLqEL14Jf2BJiwv4wVfC4mmX4EcLRF0xPksM=
    ✓1
    AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaUUgLKYKW/uQPqMjMdNPB+WPmKpj0Ai3btLBNQYvoAXqANr1ACx8hWhq0adFxHIiS7Ax37ne/e75zv3uPZKqKzRF7HmFicmp6xjvrm5v3LwSCi0t522haMsvJhmpYRUm0maroLMcVrrKiaTFRk1RWkBoHg/1Ci1m2YujHvG2yiibWdOVUkUVOVLHM64yL1Xg1GMYoOhEaBTEXhMGNlBF8hDKcgAEyNEEDBjpwwiqIYNNXghggmMRVoEOcRUhx9hmcg4+0TcpilCES26B/jVYll9VpPahpO2qZTlFpWKQMQQRf8B77+IwP+Iqff9bqODUGXto0S0MtM6uBi5Xsx78qjWYO9W/VWM8cTmHX8aqQd9NhBreQh/rW2VU/m8hEOut4i2/k/wZ7+EQ30Fvv8l2aZbpj/EjkhV6MGhT73Y5RkI9HY9tRTG+Fk/tuq7ywCmuwQf3YgSQcQQpyTh8uoQvXgl/YFBLC3jBV8LiaZfgRwuEXTm+SxA==
    ✓2
    Change so that the output values are close to the desired values for all samples
    AAACAXicdVDLSsNAFJ3UV62vqks3g0VwFRIpVXdFNy4r2AckoUymk3boTBJmboQSuvIb3Oranbj1S1z6J07aCj4PXDiccy/33hOmgmtwnDertLS8srpWXq9sbG5t71R39zo6yRRlbZqIRPVCopngMWsDB8F6qWJEhoJ1w/Fl4XdvmdI8iW9gkrJAkmHMI04JGMnzQ5n7MGJApv1qzbHdc9dtOPg3cW1nhhpaoNWvvvuDhGaSxUAF0dpznRSCnCjgVLBpxc80SwkdkyHzDI2JZDrIZydP8ZFRBjhKlKkY8Ez9OpETqfVEhqZTEhjpn14h/uV5GURnQc7jNAMW0/miKBMYElz8jwdcMQpiYgihiptbMR0RRSiYlL5tCWWRyefj+H/SObHdhl2/rteaF4t0yugAHaJj5KJT1ERXqIXaiKIE3aMH9GjdWU/Ws/Uyby1Zi5l99A3W6wfh55hS

    View full-size slide

  134. • Initialize with random values
    AAACi3ichVHLTsJAFL3UF6II6sbEDZFgXJFbH9EQF0Rj4pKHPBIgpK0DNJS2aQcSJP6ASzcucKOJC+MH+AFu/AEXfIJxiYkbF15KE6NEvM10zpy5586ZubKpqTZH7HmEicmp6RnvrG9u3r8QCC4uZW2jaSksoxiaYeVlyWaaqrMMV7nG8qbFpIassZxcPxzs51rMslVDP+Ftk5UaUlVXK6oicaLyRV5jXCqL5WAYo+hEaBSILgiDGwkj+AhFOAUDFGhCAxjowAlrIIFNXwFEQDCJK0GHOIuQ6uwzOAcfaZuUxShDIrZO/yqtCi6r03pQ03bUCp2i0bBIGYIIvuA99vEZH/AVP/+s1XFqDLy0aZaHWmaWAxcr6Y9/VQ2aOdS+VWM9c6jAnuNVJe+mwwxuoQz1rbOrfjqWinTW8RbfyP8N9vCJbqC33pW7JEt1x/iRyQu9GDVI/N2OUZDdjIo7UUxuh+MHbqu8sAprsEH92IU4HEMCMk4fLqEL14Jf2BJiwv4wVfC4mmX4EcLRF0xPksM=
    ✓1
    AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaUUgLKYKW/uQPqMjMdNPB+WPmKpj0Ai3btLBNQYvoAXqANr1ACx8hWhq0adFxHIiS7Ax37ne/e75zv3uPZKqKzRF7HmFicmp6xjvrm5v3LwSCi0t522haMsvJhmpYRUm0maroLMcVrrKiaTFRk1RWkBoHg/1Ci1m2YujHvG2yiibWdOVUkUVOVLHM64yL1Xg1GMYoOhEaBTEXhMGNlBF8hDKcgAEyNEEDBjpwwiqIYNNXghggmMRVoEOcRUhx9hmcg4+0TcpilCES26B/jVYll9VpPahpO2qZTlFpWKQMQQRf8B77+IwP+Iqff9bqODUGXto0S0MtM6uBi5Xsx78qjWYO9W/VWM8cTmHX8aqQd9NhBreQh/rW2VU/m8hEOut4i2/k/wZ7+EQ30Fvv8l2aZbpj/EjkhV6MGhT73Y5RkI9HY9tRTG+Fk/tuq7ywCmuwQf3YgSQcQQpyTh8uoQvXgl/YFBLC3jBV8LiaZfgRwuEXTm+SxA==
    ✓2
    The initial value of
    Parameter optimization by iterating “Move a little bit”

    View full-size slide

  135. Parameter optimization by iterating “Move a little bit”
    • Initialize with random values
    Iterate
    • Move a little bit towards the direction that reduces the value of
    (Learning rate)
    AAACh3ichVHLSsNQED3Gd3206kZwUyyKqzoRX7jysXHZh62CFknibRuaJiFJC1r8ARduFVwpuBA/wA9w4w+48BPEZQU3LpykAdFinZDMuefOmZx7R7UN3fWIXrqk7p7evv6BwcjQ8MhoNDY2nnetmqOJnGYZlrOnKq4wdFPkPN0zxJ7tCKWqGmJXrWz5+7t14bi6Ze54x7YoVJWSqRd1TfF8ipIkH8YSnIKItwM5BAmEkbJiDzjAESxoqKEKARMeYwMKXH72IYNgM1dAgzmHkR7sC5wiwtoaVwmuUJit8LfEq/2QNXnt93QDtcZ/Mfh1WBnHDD3THTXpie7plT7/7NUIevhejjmrLa2wD6Nnk9mPf1VVzh7K36qOnj0UsRp41dm7HTD+KbSWvn5y0cyuZWYas3RDb+z/ml7okU9g1t+127TIXHXwo7IXvjEekPx7HO0gv5CUl5KUXkysb4ajGsAUpjHH81jBOraRQo77l3GOC1xKg9K8tCyttkqlrlAzgR8hbXwB3TiQWA==
    0.01 Step size
    AAACi3ichVHLTsJAFL3UF6II6sbEDZFgXJFbH9EQF0Rj4pKHPBIgpK0DNJS2aQcSJP6ASzcucKOJC+MH+AFu/AEXfIJxiYkbF15KE6NEvM10zpy5586ZubKpqTZH7HmEicmp6RnvrG9u3r8QCC4uZW2jaSksoxiaYeVlyWaaqrMMV7nG8qbFpIassZxcPxzs51rMslVDP+Ftk5UaUlVXK6oicaLyRV5jXCqL5WAYo+hEaBSILgiDGwkj+AhFOAUDFGhCAxjowAlrIIFNXwFEQDCJK0GHOIuQ6uwzOAcfaZuUxShDIrZO/yqtCi6r03pQ03bUCp2i0bBIGYIIvuA99vEZH/AVP/+s1XFqDLy0aZaHWmaWAxcr6Y9/VQ2aOdS+VWM9c6jAnuNVJe+mwwxuoQz1rbOrfjqWinTW8RbfyP8N9vCJbqC33pW7JEt1x/iRyQu9GDVI/N2OUZDdjIo7UUxuh+MHbqu8sAprsEH92IU4HEMCMk4fLqEL14Jf2BJiwv4wVfC4mmX4EcLRF0xPksM=
    ✓1
    AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaUUgLKYKW/uQPqMjMdNPB+WPmKpj0Ai3btLBNQYvoAXqANr1ACx8hWhq0adFxHIiS7Ax37ne/e75zv3uPZKqKzRF7HmFicmp6xjvrm5v3LwSCi0t522haMsvJhmpYRUm0maroLMcVrrKiaTFRk1RWkBoHg/1Ci1m2YujHvG2yiibWdOVUkUVOVLHM64yL1Xg1GMYoOhEaBTEXhMGNlBF8hDKcgAEyNEEDBjpwwiqIYNNXghggmMRVoEOcRUhx9hmcg4+0TcpilCES26B/jVYll9VpPahpO2qZTlFpWKQMQQRf8B77+IwP+Iqff9bqODUGXto0S0MtM6uBi5Xsx78qjWYO9W/VWM8cTmHX8aqQd9NhBreQh/rW2VU/m8hEOut4i2/k/wZ7+EQ30Fvv8l2aZbpj/EjkhV6MGhT73Y5RkI9HY9tRTG+Fk/tuq7ywCmuwQf3YgSQcQQpyTh8uoQvXgl/YFBLC3jBV8LiaZfgRwuEXTm+SxA==
    ✓2
    The initial value of

    View full-size slide

  136. How to Move? → In the Direction of the ‘Gradient'
    By repeatedly moving a little bit in the opposite direction of
    the gradient vector, you approach the nearest local minimum!
    How should we choose this direction?
    AAADrHiclVE9bxNBEJ3L8RGOjzjQINFEWIlCY/aiIBBVBBSU+XISyWdZu+e1vcre7Wp3bRFO/gNI1BRUIFEgSgoaqGj4AxT5CYgySDQUzJ1PENsiKHO629k389693WFaCusIOfRm/DNnz52fvRBcvHT5ylxl/uqOVX0T83qspDJ7jFouRcrrTjjJ97ThNGGS77L9h3l9d8CNFSrddgeaNxPaTUVHxNQh1Jr3liLGuyLNWEKdEU+GQfSIS0cj1+OOtsIoGgdWcmDQVs5OVnQQ8bR9TEcbpZ0KpvQ1NU5QGSHQM0kmlbXD5Ygl2UhoeOv2n5ZjJk7NGnd6avrEcVqVKqmRIhamk7BMqlDGuqp8gAjaoCCGPiTAIQWHuQQKFp8GhEBAI9aEDDGDmSjqHIYQILePXRw7KKL7+O3irlGiKe5zTVuwY/yLxNcgcwEWyVfylhyRL+Qd+UZ+/VMrKzRyLwe4shGX69bcs+tbP//LSnB10PvLOtGzgw7cK7wK9K4LJD9FPOIPnr442rq/uZgtkdfkO/p/RQ7JZzxBOvgRv9ngmy9P8MPQC94YDiicHMd0srNSC+/UyMZqde1BOapZuAE3YRnncRfW4DGsQx1i77n33vvoffJr/rbf8Juj1hmv5FyDsfA7vwF/sgLy
    2
    6
    6
    6
    4
    ✓1
    ✓2
    .
    .
    .
    ✓p
    3
    7
    7
    7
    5
    /
    2
    6
    6
    6
    4
    @loss(✓)/@✓1
    @loss(✓)/@✓2
    .
    .
    .
    @loss(✓)/@✓p
    3
    7
    7
    7
    5
    The gradient vector

    View full-size slide

  137. Gradient
    AAAClHichVHLSsNAFL2Nr1ofjQoiuAkWRUHKTVEUUSgWwZW01aqgJSRxWkPzIklLa/EH/AEXXSl2IX6AH+DGH3DRTxCXCm5ceJsGRMV6w2TOnLnnzpm5iq1rrofYCnE9vX39A+HByNDwyGiUHxvfd62yo7KcaumWc6jILtM1k+U8zdPZoe0w2VB0dqCUUu39gwpzXM0y97yazfKGXDS1gqbKHlESH60JG0JhviqJi0JVSixIfAzj6IfwG4gBiEEQaYu/h2M4AQtUKIMBDEzwCOsgg0vfEYiAYBOXhzpxDiHN32dwDhHSlimLUYZMbIn+RVodBaxJ63ZN11erdIpOwyGlALP4hLf4io94h8/48Wetul+j7aVGs9LRMluKXkztvv+rMmj24PRL1dWzBwVY9b1q5N32mfYt1I6+cnb5uruWna3P4TW+kP8rbOED3cCsvKnNDMs2uvhRyAu9GDVI/NmO32A/EReX45hZiiU3g1aFYRpmYJ76sQJJ2IY05PyeNeAGmtwkt86luK1OKhcKNBPwLbidTyDslEI=
    y = f(x1, x2)
    AAAChnichVG7SgNBFD2ur/hM1EawEUPEKtwEQ8QqaGOZh9FADGF3HeOSfbG7CWrwBwRbU1gpWIgf4AfY+AMWfoJYKthYeLNZEA3Gu8zOmTP33DkzV7F1zfWIngekwaHhkdHQ2PjE5NR0ODIzu+NaDUcVRdXSLaekyK7QNVMUPc3TRcl2hGwouthV6pud/d2mcFzNMre9Y1tUDLlmageaKntMFY6qyWokSnHyY7EXJAIQRRBZK3KPPezDgooGDAiY8BjrkOHyV0YCBJu5ClrMOYw0f1/gFOOsbXCW4AyZ2Tr/a7wqB6zJ605N11erfIrOw2HlImL0RLf0Ro90Ry/0+Wetll+j4+WYZ6WrFXY1fDZf+PhXZfDs4fBb1dezhwOs+V419m77TOcWalffPGm/FdbzsdYyXdMr+7+iZ3rgG5jNd/UmJ/KXffwo7IVfjBuU+N2OXrCTjCdSccqtRjMbQatCWMASVrgfaWSwhSyKXL+Gc1ygLYWkuJSS0t1UaSDQzOFHSJkvVTaQmA==
    x2
    AAAChnichVHLSsNAFD2Nr/quuhHciKXiqtyIUnFVdONSW/uAWkoSpzU0TUKSFmvxBwS3duFKwYX4AX6AG3/AhZ8gLhXcuPA2DYgW6w2TOXPmnjtn5qq2obse0XNIGhgcGh4Jj46NT0xOTUdmZrOuVXc0kdEsw3LyquIKQzdFxtM9Q+RtRyg11RA5tbrd2c81hOPqlrnvNW1RrCkVUy/rmuIxlT4uyaVIlOLkx2IvkAMQRRC7VuQeBziEBQ111CBgwmNsQIHLXwEyCDZzRbSYcxjp/r7AKcZYW+cswRkKs1X+V3hVCFiT152arq/W+BSDh8PKRcToiW7pjR7pjl7o889aLb9Gx0uTZ7WrFXZp+mw+/fGvqsazh6NvVV/PHsrY8L3q7N32mc4ttK6+cdJ+S2+mYq1luqZX9n9Fz/TANzAb79rNnkhd9vGjshd+MW6Q/LsdvSC7GpfX47S3Fk1uBa0KYwFLWOF+JJDEDnaR4foVnOMCbSksxaV1KdFNlUKBZg4/Qkp+AVMWkJc=
    x1
    AAACCHicdVDLSgMxFM34rPU16tJNsAh1M8yUqW13RTcuK9gHtKVk0kwbmmSGJFMsQ3/Ab3Cra3fi1r9w6Z+YPgQreuDC4Zx7ufeeIGZUadf9sNbWNza3tjM72d29/YND++i4oaJEYlLHEYtkK0CKMCpIXVPNSCuWBPGAkWYwup75zTGRikbiTk9i0uVoIGhIMdJG6tl2R6CAIRjmOwFP76cXPTvnOu4c0HUKlXKx7BtSKlZ8Q7yllQNL1Hr2Z6cf4YQToTFDSrU9N9bdFElNMSPTbCdRJEZ4hAakbahAnKhuOr98Cs+N0odhJE0JDefqz4kUcaUmPDCdHOmh+u3NxL+8dqLDcjelIk40EXixKEwY1BGcxQD7VBKs2cQQhCU1t0I8RBJhbcJa2RLwqcnk+3H4P2kUHO/S8W/9XPVqmU4GnIIzkAceKIEquAE1UAcYjMEjeALP1oP1Yr1ab4vWNWs5cwJWYL1/AS7mmf8=
    rf(x)
    AAACCXicdVDLTgIxFO3gC/EFunTTSExwIRkGBNwR3bjERB4JENIpHWhoO5O2o5IJX+A3uNW1O+PWr3Dpn9gBTMToSW5ycs69ufceN2BUadv+sBIrq2vrG8nN1Nb2zu5eOrPfVH4oMWlgn/my7SJFGBWkoalmpB1IgrjLSMsdX8Z+65ZIRX1xoycB6XE0FNSjGGkj9dOZ065ALkPQy3VdHt1PT/rprJ0/r5adMwfaeduuOMVyTJxKySnCglFiZMEC9X76szvwcciJ0JghpToFO9C9CElNMSPTVDdUJEB4jIakY6hAnKheNDt9Co+NMoCeL00JDWfqz4kIcaUm3DWdHOmR+u3F4l9eJ9RetRdREYSaCDxf5IUMah/GOcABlQRrNjEEYUnNrRCPkERYm7SWtrh8ajL5fhz+T5pOvlDOl65L2drFIp0kOARHIAcKoAJq4ArUQQNgcAcewRN4th6sF+vVepu3JqzFzAFYgvX+BZf+mjE=
    rf(x)
    Gradient
    The gradient vector consists of partial derivatives with respect to each variable.
    The change in the function value when each variable is altered a little bit.
    = The direction of the normal vector to the isosurface.
    AAAC+XiclVJNb9MwGHbCx0b5WAdHLtYqpE6UKum6Mg5I09hhxyHRbVITRY7rdNbsJLLfoFWWfwJXOHNDXPk1O/JP5qRFsJXLXsnSo+f9eF4/dloKriEIrjz/3v0HD9fWH7UeP3n6bKO9+fxEF5WibEwLUaizlGgmeM7GwEGws1IxIlPBTtOLD3X+9DNTmhf5J5iXLJZklvOMUwKOSjY9FGWKUBOVRAEnAmfdKJXm0m7bv9xlEtrWexwJLhNjokZ1omZpbIL+u71gZzfsBf1BuDMaDR0ImrDRIRNALI6gwIFtLWSyrpv1+o4jem6BQQ9H0wJ0jafb+A1uJq1m7F33s0m784fDqyBcgg5axnHS/u30aCVZDlQQrSdhUEJsaq+oYO6mlWYloRdkxiYO5kQyHZtmKYtfOWaKs0K5kwNu2H87DJFaz2XqKiWBc307V5P/y00qyPZiw/OyApbThVBWCezcr98dT7liFMTcAUIVd7tiek7cm4D7HTdUUll7Et52YBWcDPrhqD/8OOzsHyzdWUcv0RbqohC9RfvoCB2jMaLezPviffW++cb/7v/wfy5KfW/Z8wLdCP/XNSpG4vM=
    @f(x)
    @x1
    = lim
    !0
    f(x1 + , x2, . . . , xd) f(x1, x2, . . . , xd)
    AAAChnichVG7SgNBFD2ur/hM1EawEUPEKtwEQ8QqaGOZh9FADGF3HeOSfbG7CWrwBwRbU1gpWIgf4AfY+AMWfoJYKthYeLNZEA3Gu8zOmTP33DkzV7F1zfWIngekwaHhkdHQ2PjE5NR0ODIzu+NaDUcVRdXSLaekyK7QNVMUPc3TRcl2hGwouthV6pud/d2mcFzNMre9Y1tUDLlmageaKntMFY6qyWokSnHyY7EXJAIQRRBZK3KPPezDgooGDAiY8BjrkOHyV0YCBJu5ClrMOYw0f1/gFOOsbXCW4AyZ2Tr/a7wqB6zJ605N11erfIrOw2HlImL0RLf0Ro90Ry/0+Wetll+j4+WYZ6WrFXY1fDZf+PhXZfDs4fBb1dezhwOs+V419m77TOcWalffPGm/FdbzsdYyXdMr+7+iZ3rgG5jNd/UmJ/KXffwo7IVfjBuU+N2OXrCTjCdSccqtRjMbQatCWMASVrgfaWSwhSyKXL+Gc1ygLYWkuJSS0t1UaSDQzOFHSJkvVTaQmA==
    x2
    AAAChnichVHLSsNAFD2Nr/quuhHciKXiqtyIUnFVdONSW/uAWkoSpzU0TUKSFmvxBwS3duFKwYX4AX6AG3/AhZ8gLhXcuPA2DYgW6w2TOXPmnjtn5qq2obse0XNIGhgcGh4Jj46NT0xOTUdmZrOuVXc0kdEsw3LyquIKQzdFxtM9Q+RtRyg11RA5tbrd2c81hOPqlrnvNW1RrCkVUy/rmuIxlT4uyaVIlOLkx2IvkAMQRRC7VuQeBziEBQ111CBgwmNsQIHLXwEyCDZzRbSYcxjp/r7AKcZYW+cswRkKs1X+V3hVCFiT152arq/W+BSDh8PKRcToiW7pjR7pjl7o889aLb9Gx0uTZ7WrFXZp+mw+/fGvqsazh6NvVV/PHsrY8L3q7N32mc4ttK6+cdJ+S2+mYq1luqZX9n9Fz/TANzAb79rNnkhd9vGjshd+MW6Q/LsdvSC7GpfX47S3Fk1uBa0KYwFLWOF+JJDEDnaR4foVnOMCbSksxaV1KdFNlUKBZg4/Qkp+AVMWkJc=
    x1
    AAACpXichVFNLyxBFD2a52M8DDYSm8mb+FhN7gghVsLGRuJrhsRIp7rVUNHT3emumWAyWwt/wMLqvcRC2OIH2PgDFn6CvCWJjYU7PZ0IgtuprlOn7rl1qq7lOyrURPdNRnPLr9a29o5E5++u7p5kb18+9MqBLXO253jBuiVC6ShX5rTSjlz3AylKliPXrN25+v5aRQah8txVve/LzZLYdlVR2UIzZSZThWIg7GrBF4FWwkkVa294zxyrJcxkmjIUReozyMYgjTgWveQ1CtiCBxtllCDhQjN2IBDyt4EsCD5zm6gyFzBS0b5EDQnWljlLcoZgdpf/27zaiFmX1/WaYaS2+RSHR8DKFIbojs7okW7pnB7o5cta1ahG3cs+z1ZDK32z52hg5flHVYlnjZ031beeNYqYirwq9u5HTP0WdkNfOTh+XJleHqoO0z/6z/7/0j3d8A3cypN9uiSXT77xY7EXfjFuUPZjOz6D/FgmO5GhpfH0zGzcqnYM4g9GuR+TmME8FpHj+oe4wCWujBFjwVg18o1UoynW9ONdGOYrpNKdFQ==
    @f
    @x2
    AAACpXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsoxKQVJSZXxxQkFpVkJuYopNUi2BXxhrVc8QLKBnoGYKCAyTCEMpQZoCAgX2A7QwxDCkM+QzJDKUMuQypDHkMJkJ3DkMhQDITRDIYMBgwFQLFYhmqgWBGQlQmWT2WoZeAC6i0FqkoFqkgEimYDyXQgLxoqmgfkg8wsButOBtqSA8RFQJ0KDKoGVw1WGnw2OGGw2uClwR+cZlWDzQC5pRJIJ0H0phbE83dJBH8nqCsXSJcwZCB04XVzCUMagwXYrZlAtxeARUC+SIboL6ua/jnYKki1Ws1gkcFroPsXGtw0OAz0QV7Zl+SlgalBs/G4JwnoFmCIASPIED06MBlhRnqGpnoGgSbKDk7QqOJgkGZQYtAAxoc5gwODB0MAQyjQ/DaGNQxbGbYxqTP5MoUwhUGUMjFC9QgzoACmeACisJ0U
    @f
    @x1
    AAAC2HicjVFNa9wwEJXdtPno1yY95iKyFFIoi704zeYQCC2UHFPIJqErs0iyvBGRZCPJyy7C0FvpNf8ut/afRHa2aT56yIDg8eaNZuYNKQU3NoqugvDZ0vMXyyuray9fvX7ztrO+cWKKSlM2pIUo9BnBhgmu2NByK9hZqRmWRLBTcvGlyZ9OmTa8UMd2XrJU4oniOafYemrc+e1Q+8lIT0jqol7Uxseo198b7AwSD3Z39pJBUiOFicAw30ZEuln9oYb7EBE24coRia3ms3oN5RpTh0qsLcfijvYfNxvHNUJPU/Zb5TQrrHlqSdaWMJXdDjXudP9uBR+DeAG6YBFH484flBW0kkxZKrAxozgqbeqaFlQwv2RlWInpBZ6wkYcKS2ZS15pYw/eeyWBeaP+UhS17t8JhacxcEq/0852bh7mG/F9uVNl8kDquysoyRW8a5ZWAtoDNWWHGNaNWzD3AVHM/K6Tn2Dtm/fHvdSGy8SR+6MBjcNLvxZ96ybeke/B54c4K2ARbYBvEYBccgENwBIaABl8DEVTBNPwe/gh/hr9upGGwqHkH7kV4eQ2Md+O1
    rf(x) =
    2
    6
    6
    6
    6
    4
    @f(x)
    @x1
    @f(x)
    @x2
    .
    .
    .
    @f(x)
    @xd
    3
    7
    7
    7
    7
    5

    View full-size slide

  138. Gradient Descent
    When minimizing, you should move in
    the opposite direction of the gradient
    LR (learning rate) = Step size: How far to move in that direction
    (typically set as a small value initially to avoid overshooting)
    Gradient descent

    View full-size slide

  139. Tuning the learning rate (step size)
    What's a good value for the learning rate (LR)?
    LR medium
    LR small LR large
    It certainly heads towards the nearest local
    minimum, but requires many iterations
    It can advance significantly with
    few iterations, but it's too coarse
    LR

    View full-size slide

  140. A high-level glance: “Randomize” vs “Optimize”
    The 'Optimize' type (like Boosting) aims for 'optimization'
    Learning rate
    (Step size)
    We usually set a small value,
    say 0.1 for now.
    AAACAHicbVC7SgNBFJ31GeMramkzGASrZVd8NULQxjKCeWCyhNnJ3WTIzOwyMyuEJY3fYKu1ndj6J5b+iZNkC5N44MLhnHu5954w4Uwbz/t2lpZXVtfWCxvFza3tnd3S3n5dx6miUKMxj1UzJBo4k1AzzHBoJgqICDk0wsHt2G88gdIslg9mmEAgSE+yiFFirPTYBkPwNfZcv1Mqe643AV4kfk7KKEe1U/ppd2OaCpCGcqJ1y/cSE2REGUY5jIrtVENC6ID0oGWpJAJ0kE0uHuFjq3RxFCtb0uCJ+nciI0LroQhtpyCmr+e9sfif10pNdBVkTCapAUmni6KUYxPj8fu4yxRQw4eWEKqYvRXTPlGEGhvSzJZQjGwm/nwCi6R+6voX7vn9Wblyk6dTQIfoCJ0gH12iCrpDVVRDFEn0gl7Rm/PsvDsfzue0dcnJZw7QDJyvX1IdliA=
    ⌘ = 0.1

    View full-size slide

  141. The basic idea of Gradient Boosting
    Start with a constant prediction and iteratively add a decision tree to the current
    ensemble to minimize the loss function.
    AAACAXicbVDLSsNAFJ3UV62vqks3g0Wom5KIr2VREJcVrC2koUymk3boPMLMRCwhK7/Bra7diVu/xKV/4rTNwrYeuHA4517uvSeMGdXGdb+dwtLyyupacb20sbm1vVPe3XvQMlGYNLFkUrVDpAmjgjQNNYy0Y0UQDxlphcPrsd96JEpTKe7NKCYBR31BI4qRsZJ/03WrnZCnT9lxt1xxa+4EcJF4OamAHI1u+afTkzjhRBjMkNa+58YmSJEyFDOSlTqJJjHCQ9QnvqUCcaKDdHJyBo+s0oORVLaEgRP170SKuNYjHtpOjsxAz3tj8T/PT0x0GaRUxIkhAk8XRQmDRsLx/7BHFcGGjSxBWFF7K8QDpBA2NqWZLSHPbCbefAKL5OGk5p3Xzu5OK/WrPJ0iOACHoAo8cAHq4BY0QBNgIMELeAVvzrPz7nw4n9PWgpPP7IMZOF+/PgSXSQ==
    F0(x)
    AAACAXicbVDLSsNAFJ3UV62vqks3g0Wom5KIr2VREJcVrC2koUymk3boPMLMRCwhK7/Bra7diVu/xKV/4rTNwrYeuHA4517uvSeMGdXGdb+dwtLyyupacb20sbm1vVPe3XvQMlGYNLFkUrVDpAmjgjQNNYy0Y0UQDxlphcPrsd96JEpTKe7NKCYBR31BI4qRsZJ/0/WqnZCnT9lxt1xxa+4EcJF4OamAHI1u+afTkzjhRBjMkNa+58YmSJEyFDOSlTqJJjHCQ9QnvqUCcaKDdHJyBo+s0oORVLaEgRP170SKuNYjHtpOjsxAz3tj8T/PT0x0GaRUxIkhAk8XRQmDRsLx/7BHFcGGjSxBWFF7K8QDpBA2NqWZLSHPbCbefAKL5OGk5p3Xzu5OK/WrPJ0iOACHoAo8cAHq4BY0QBNgIMELeAVvzrPz7nw4n9PWgpPP7IMZOF+/P5+XSg==
    F1(x)
    AAACAXicbVC7SgNBFJ2NrxhfUUubwSDEJuwGX2VQEMsI5gGbJcxOJsmQeSwzs2JYtvIbbLW2E1u/xNI/cZJsYRIPXDiccy/33hNGjGrjut9ObmV1bX0jv1nY2t7Z3SvuHzS1jBUmDSyZVO0QacKoIA1DDSPtSBHEQ0Za4ehm4rceidJUigczjkjA0UDQPsXIWMm/7VbLnZAnT+lpt1hyK+4UcJl4GSmBDPVu8afTkzjmRBjMkNa+50YmSJAyFDOSFjqxJhHCIzQgvqUCcaKDZHpyCk+s0oN9qWwJA6fq34kEca3HPLSdHJmhXvQm4n+eH5v+VZBQEcWGCDxb1I8ZNBJO/oc9qgg2bGwJworaWyEeIoWwsSnNbQl5ajPxFhNYJs1qxbuonN+flWrXWTp5cASOQRl44BLUwB2ogwbAQIIX8ArenGfn3flwPmetOSebOQRzcL5+AUE6l0s=
    F2(x)
    AAACAXicbVDJSgNBEO2JW4xb1KOXxiDES5hxPwYF8RjBLDAZQk+nJ2nSy9DdI4ZhTn6DVz17E69+iUf/xM5yMIkPCh7vVVFVL4wZ1cZ1v53c0vLK6lp+vbCxubW9U9zda2iZKEzqWDKpWiHShFFB6oYaRlqxIoiHjDTDwc3Ibz4SpakUD2YYk4CjnqARxchYyb/tnJbbIU+fsuNOseRW3DHgIvGmpASmqHWKP+2uxAknwmCGtPY9NzZBipShmJGs0E40iREeoB7xLRWIEx2k45MzeGSVLoyksiUMHKt/J1LEtR7y0HZyZPp63huJ/3l+YqKrIKUiTgwReLIoShg0Eo7+h12qCDZsaAnCitpbIe4jhbCxKc1sCXlmM/HmE1gkjZOKd1E5vz8rVa+n6eTBATgEZeCBS1AFd6AG6gADCV7AK3hznp1358P5nLTmnOnMPpiB8/ULQtWXTA==
    F3(x)
    AAACAXicbVC7TgJBFJ3FF+ILtbSZSEywIbvGV0m0scQEkGTZkNlhFibMYzMzaySbrfwGW63tjK1fYumfOMAWAp7kJifn3Jt77wljRrVx3W+nsLK6tr5R3Cxtbe/s7pX3D9paJgqTFpZMqk6INGFUkJahhpFOrAjiISMP4eh24j88EqWpFE0zjknA0UDQiGJkrOQ3e261G/L0KTvtlStuzZ0CLhMvJxWQo9Er/3T7EiecCIMZ0tr33NgEKVKGYkayUjfRJEZ4hAbEt1QgTnSQTk/O4IlV+jCSypYwcKr+nUgR13rMQ9vJkRnqRW8i/uf5iYmug5SKODFE4NmiKGHQSDj5H/apItiwsSUIK2pvhXiIFMLGpjS3JeSZzcRbTGCZtM9q3mXt4v68Ur/J0ymCI3AMqsADV6AO7kADtAAGEryAV/DmPDvvzofzOWstOPnMIZiD8/ULVJqXVw==
    T0(x)
    AAACAXicbVC7TgJBFJ3FF+ILtbSZSEywIbvGV0m0scQEkGTZkNlhFibMYzMzaySbrfwGW63tjK1fYumfOMAWAp7kJifn3Jt77wljRrVx3W+nsLK6tr5R3Cxtbe/s7pX3D9paJgqTFpZMqk6INGFUkJahhpFOrAjiISMP4eh24j88EqWpFE0zjknA0UDQiGJkrOQ3e161G/L0KTvtlStuzZ0CLhMvJxWQo9Er/3T7EiecCIMZ0tr33NgEKVKGYkayUjfRJEZ4hAbEt1QgTnSQTk/O4IlV+jCSypYwcKr+nUgR13rMQ9vJkRnqRW8i/uf5iYmug5SKODFE4NmiKGHQSDj5H/apItiwsSUIK2pvhXiIFMLGpjS3JeSZzcRbTGCZtM9q3mXt4v68Ur/J0ymCI3AMqsADV6AO7kADtAAGEryAV/DmPDvvzofzOWstOPnMIZiD8/ULVjWXWA==
    T1(x)
    AAACAXicbVC7SgNBFJ2NrxhfUUubwSDEJuwGX2XQxjJCXpAsYXYymwyZxzIzK4ZlK7/BVms7sfVLLP0TJ8kWJvHAhcM593LvPUHEqDau++3k1tY3Nrfy24Wd3b39g+LhUUvLWGHSxJJJ1QmQJowK0jTUMNKJFEE8YKQdjO+mfvuRKE2laJhJRHyOhoKGFCNjpW6jXy33Ap48pef9YsmtuDPAVeJlpAQy1PvFn95A4pgTYTBDWnc9NzJ+gpShmJG00Is1iRAeoyHpWioQJ9pPZien8MwqAxhKZUsYOFP/TiSIaz3hge3kyIz0sjcV//O6sQlv/ISKKDZE4PmiMGbQSDj9Hw6oItiwiSUIK2pvhXiEFMLGprSwJeCpzcRbTmCVtKoV76py+XBRqt1m6eTBCTgFZeCBa1AD96AOmgADCV7AK3hznp1358P5nLfmnGzmGCzA+foFV9CXWQ==
    T2(x)
    AAACAXicbVC7SgNBFJ2NrxhfUUubwSDEJuz6LoM2lhHygs0SZieTZMg8lplZMSxb+Q22WtuJrV9i6Z84SbYwiQcuHM65l3vvCSNGtXHdbye3srq2vpHfLGxt7+zuFfcPmlrGCpMGlkyqdog0YVSQhqGGkXakCOIhI61wdDfxW49EaSpF3YwjEnA0ELRPMTJW8uvd83In5MlTetotltyKOwVcJl5GSiBDrVv86fQkjjkRBjOkte+5kQkSpAzFjKSFTqxJhPAIDYhvqUCc6CCZnpzCE6v0YF8qW8LAqfp3IkFc6zEPbSdHZqgXvYn4n+fHpn8TJFREsSECzxb1YwaNhJP/YY8qgg0bW4KwovZWiIdIIWxsSnNbQp7aTLzFBJZJ86ziXVUuHy5K1dssnTw4AsegDDxwDargHtRAA2AgwQt4BW/Os/PufDifs9ack80cgjk4X79Za5da
    T3(x)
    AAACDnicbVDLSsNAFJ3UV62vquDGzWARKkJJxNey6MZlhb6gCWUynbZDZ5IwcyOW2H/wG9zq2p249Rdc+idO2yxs64ELh3Pu5VyOHwmuwba/rczS8srqWnY9t7G5tb2T392r6zBWlNVoKELV9IlmggesBhwEa0aKEekL1vAHt2O/8cCU5mFQhWHEPEl6Ae9ySsBI7fzBqcuAuLQTAq627aLry+RxdNLOF+ySPQFeJE5KCihFpZ3/cTshjSULgAqidcuxI/ASooBTwUY5N9YsInRAeqxlaEAk014y+X+Ej43Swd1QmQkAT9S/FwmRWg+lbzYlgb6e98bif14rhu61l/AgioEFdBrUjQWGEI/LwB2uGAUxNIRQxc2vmPaJIhRMZTMpvhyZTpz5BhZJ/azkXJYu7s8L5Zu0nSw6REeoiBx0hcroDlVQDVH0hF7QK3qznq1368P6nK5mrPRmH83A+voFyhib3Q==
    +⌘ · T0(x)
    AAACDnicbVDLSsNAFJ3UV62vquDGzWARKkJJxNey6MZlhb6gCWUynbZDZ5IwcyOW2H/wG9zq2p249Rdc+idO2yxs64ELh3Pu5VyOHwmuwba/rczS8srqWnY9t7G5tb2T392r6zBWlNVoKELV9IlmggesBhwEa0aKEekL1vAHt2O/8cCU5mFQhWHEPEl6Ae9ySsBI7fzBqcuAuLQTAq62naLry+RxdNLOF+ySPQFeJE5KCihFpZ3/cTshjSULgAqidcuxI/ASooBTwUY5N9YsInRAeqxlaEAk014y+X+Ej43Swd1QmQkAT9S/FwmRWg+lbzYlgb6e98bif14rhu61l/AgioEFdBrUjQWGEI/LwB2uGAUxNIRQxc2vmPaJIhRMZTMpvhyZTpz5BhZJ/azkXJYu7s8L5Zu0nSw6REeoiBx0hcroDlVQDVH0hF7QK3qznq1368P6nK5mrPRmH83A+voFy7Ob3g==
    +⌘ · T1(x)
    AAACDnicbVDLSsNAFJ3UV62vquDGzWARKkJJiq9l0Y3LCn1BE8JkOm2HziRh5kYssf/gN7jVtTtx6y+49E+cPha2euDC4Zx7OZcTxIJrsO0vK7O0vLK6ll3PbWxube/kd/caOkoUZXUaiUi1AqKZ4CGrAwfBWrFiRAaCNYPBzdhv3jOleRTWYBgzT5JeyLucEjCSnz84dRkQl3YiwDW/XHQDmT6MTvx8wS7ZE+C/xJmRApqh6ue/3U5EE8lCoIJo3XbsGLyUKOBUsFHOTTSLCR2QHmsbGhLJtJdO/h/hY6N0cDdSZkLAE/X3RUqk1kMZmE1JoK8XvbH4n9dOoHvlpTyME2AhnQZ1E4EhwuMycIcrRkEMDSFUcfMrpn2iCAVT2VxKIEemE2exgb+kUS45F6Xzu7NC5XrWThYdoiNURA66RBV0i6qojih6RM/oBb1aT9ab9W59TFcz1uxmH83B+vwBzU6b3w==
    +⌘ · T2(x)

    View full-size slide

  142. The basic idea of Gradient Boosting
    Update
    Gradient values
    of the loss func.
    at training points AAACOHicbZDLSgMxFIYz9VbrbdSlm2ARK2iZEW8boejGRRcV7AU6pWTSTBuauZBkhGGY1/EhfAa3Cu50JW59AjPtiLb1QODn+8/JSX47YFRIw3jVcnPzC4tL+eXCyura+oa+udUQfsgxqWOf+bxlI0EY9UhdUslIK+AEuTYjTXt4nfrNe8IF9b07GQWk46K+Rx2KkVSoq1eq+6Xo0BogGUfJAbyER5bDEY6tAHFJEYNVZcMfP/nlGUq6etEoG6OCs8LMRBFkVevq71bPx6FLPIkZEqJtGoHsxOm1mJGkYIWCBAgPUZ+0lfSQS0QnHv00gXuK9KDjc3U8CUf070SMXCEi11adLpIDMe2l8D+vHUrnohNTLwgl8fB4kRMyKH2YxgZ7lBMsWaQEwpyqt0I8QCopqcKd2GK7aSbmdAKzonFcNs/Kp7cnxcpVlk4e7IBdUAImOAcVcANqoA4weABP4Bm8aI/am/ahfY5bc1o2sw0mSvv6BtzZrRg=
    L0(y, ˆ
    y) =
    @L(y, ˆ
    y)

    y
    AAACN3icbVDLShxBFK3W+IyPMS6zKTIEFGHoFh/ZRRSGLCcwD2F6aKprbmsxVV1N1W3j0PTn+BF+g1tduArZSLb5g9SMgzrqgQuHc+7l3nviTAqLvn/vzcx+mJtfWFxa/riyurZe2fjUtjo3HFpcS21OY2ZBihRaKFDCaWaAqVhCJx6cjPzOBRgrdNrEYQY9xc5SkQjO0ElR5Xs9KnAnKLfCWBWX5TYNJSTIjNG/aD3CJ3mHhoCMhryvkTafjahS9Wv+GPQtCSakSiZoRJU/YV/zXEGKXDJru4GfYa9gBgWXUC6HuYWM8QE7g66jKVNge8X40ZJ+dUqfJtq4SpGO1ZcTBVPWDlXsOhXDc/vaG4nved0ck2+9QqRZjpDyx0VJLilqOkqN9oUBjnLoCONGuFspP2eGcXTZTm2JVekyCV4n8Ja0d2vBQW3/51716HiSziL5TL6QLRKQQ3JEfpAGaRFOrsgNuSV33rX323vw/j62zniTmU0yBe/ff+pRq+s=
    Ft+1(x) Ft(x) + ⌘ · Tt(x)
    AAACA3icbVDLSsNAFL2pr1pfVZduBotQNyURX8uiIC4r2Ae0oUymk3boTBJnJmIJWfoNbnXtTtz6IS79E6dtFrb1wIXDOfdyLseLOFPatr+t3NLyyupafr2wsbm1vVPc3WuoMJaE1knIQ9nysKKcBbSumea0FUmKhcdp0xtej/3mI5WKhcG9HkXUFbgfMJ8RrI3k3nQTnZY7nkie0uNusWRX7AnQInEyUoIMtW7xp9MLSSxooAnHSrUdO9JugqVmhNO00IkVjTAZ4j5tGxpgQZWbTJ5O0ZFResgPpZlAo4n69yLBQqmR8MymwHqg5r2x+J/XjrV/6SYsiGJNAzIN8mOOdIjGDaAek5RoPjIEE8nMr4gMsMREm55mUjyRmk6c+QYWSeOk4pxXzu5OS9WrrJ08HMAhlMGBC6jCLdSgDgQe4AVe4c16tt6tD+tzupqzspt9mIH19QuGOpiZ
    Ft(x)
    AAACB3icbVDLSsNAFJ3UV62PRl26GSxCRSiJ+FoWBXFZwdpCG8JkOmmHzkzCzEQsIR/gN7jVtTtx62e49E+ctlnY1gMXDufcy7mcIGZUacf5tgpLyyura8X10sbm1nbZ3tl9UFEiMWniiEWyHSBFGBWkqalmpB1LgnjASCsYXo/91iORikbiXo9i4nHUFzSkGGkj+Xb5xk/1sZtVuwFPn7Ij3644NWcCuEjcnFRAjoZv/3R7EU44ERozpFTHdWLtpUhqihnJSt1EkRjhIeqTjqECcaK8dPJ4Bg+N0oNhJM0IDSfq34sUcaVGPDCbHOmBmvfG4n9eJ9HhpZdSESeaCDwNChMGdQTHLcAelQRrNjIEYUnNrxAPkERYm65mUgKemU7c+QYWycNJzT2vnd2dVupXeTtFsA8OQBW44ALUwS1ogCbAIAEv4BW8Wc/Wu/VhfU5XC1Z+swdmYH39AuFKmTo=
    Ft+1(x)
    AAACEHicbVDLSgNBEJyNrxhfUS+Cl8EgRISwK76OQS8eI+QF2SXMTibJkJndZaZXDEv8CL/Bq569iVf/wKN/4iTZg0ksaCiquunu8iPBNdj2t5VZWl5ZXcuu5zY2t7Z38rt7dR3GirIaDUWomj7RTPCA1YCDYM1IMSJ9wRr+4HbsNx6Y0jwMqjCMmCdJL+BdTgkYqZ0/OMUuA4Jd2gkBV9tQdH2ZPI5O2vmCXbInwIvESUkBpai08z9uJ6SxZAFQQbRuOXYEXkIUcCrYKOfGmkWEDkiPtQwNiGTaSyYfjPCxUTq4GypTAeCJ+nciIVLrofRNpyTQ1/PeWPzPa8XQvfYSHkQxsIBOF3VjgSHE4zhwhytGQQwNIVRxcyumfaIIBRPazBZfjkwmznwCi6R+VnIuSxf354XyTZpOFh2iI1REDrpCZXSHKqiGKHpCL+gVvVnP1rv1YX1OWzNWOrOPZmB9/QLtWZx1
    +⌘ · Tt(x)
    A regression tree
    AAACAXicbVC7TgJBFJ3FF+ILtbSZSEywIbvGV0m0scQEkGTZkNlhFibMYzMzaySbrfwGW63tjK1fYumfOMAWAp7kJifn3Jt77wljRrVx3W+nsLK6tr5R3Cxtbe/s7pX3D9paJgqTFpZMqk6INGFUkJahhpFOrAjiISMP4eh24j88EqWpFE0zjknA0UDQiGJkrOQ3e6baDXn6lJ32yhW35k4Bl4mXkwrI0eiVf7p9iRNOhMEMae17bmyCFClDMSNZqZtoEiM8QgPiWyoQJzpIpydn8MQqfRhJZUsYOFX/TqSIaz3moe3kyAz1ojcR//P8xETXQUpFnBgi8GxRlDBoJJz8D/tUEWzY2BKEFbW3QjxECmFjU5rbEvLMZuItJrBM2mc177J2cX9eqd/k6RTBETgGVeCBK1AHd6ABWgADCV7AK3hznp1358P5nLUWnHzmEMzB+foFwcaXmw==
    Tt(x)
    AAACJXicbVDLSgMxFM3UV62vqks3wSK0oGVGfC2LblxWsA/olJJkMm1oMjMkGbEM8xV+hN/gVtfuRHAl/omZtoitHggczjmXe3NwxJnStv1h5RYWl5ZX8quFtfWNza3i9k5ThbEktEFCHso2RopyFtCGZprTdiQpEpjTFh5eZX7rjkrFwuBWjyLaFagfMJ8RpI3UKx65mPV52cUiuU977BC6AumBFElfIi/90StZTFZ6xZJdtceAf4kzJSUwRb1X/HK9kMSCBppwpFTHsSPdTZDUjHCaFtxY0QiRIerTjqEBElR1k/G3UnhgFA/6oTQv0HCs/p5IkFBqJLBJZkereS8T//M6sfYvugkLoljTgEwW+TGHOoRZR9BjkhLNR4YgIpm5FZIBkoho0+TMFixS04kz38Bf0jyuOmfV05uTUu1y2k4e7IF9UAYOOAc1cA3qoAEIeABP4Bm8WI/Wq/VmvU+iOWs6swtmYH1+AzBLpjY=
    xi, grad(xi)
    AAACDnicbVDLSsNAFJ3UV62vqODGTbCILUhJxNdGKAriwkUF+4A2hMl00g6dScLMRAwx/+A3uNW1O3HrL7j0T5y2WdjWAxcO59zLuRw3pERI0/zWcnPzC4tL+eXCyura+oa+udUQQcQRrqOABrzlQoEp8XFdEklxK+QYMpfipju4GvrNB8wFCfx7GYfYZrDnE48gKJXk6DsXtwel2CGH144sdVyWPKYOKZcdvWhWzBGMWWJlpAgy1Bz9p9MNUMSwLxGFQrQtM5R2ArkkiOK00IkEDiEawB5uK+pDhoWdjP5PjX2ldA0v4Gp8aYzUvxcJZELEzFWbDMq+mPaG4n9eO5LeuZ0QP4wk9tE4yIuoIQNjWIbRJRwjSWNFIOJE/WqgPuQQSVXZRIrLUtWJNd3ALGkcVazTysndcbF6mbWTB7tgD5SABc5AFdyAGqgDBJ7AC3gFb9qz9q59aJ/j1ZyW3WyDCWhfvwCem2I=
    = L0(yi, Ft(xi))

    View full-size slide

  143. The basic idea of Gradient Boosting
    Update
    Gradient values
    of the loss func.
    at training points AAACOHicbZDLSgMxFIYz9VbrbdSlm2ARK2iZEW8boejGRRcV7AU6pWTSTBuauZBkhGGY1/EhfAa3Cu50JW59AjPtiLb1QODn+8/JSX47YFRIw3jVcnPzC4tL+eXCyura+oa+udUQfsgxqWOf+bxlI0EY9UhdUslIK+AEuTYjTXt4nfrNe8IF9b07GQWk46K+Rx2KkVSoq1eq+6Xo0BogGUfJAbyER5bDEY6tAHFJEYNVZcMfP/nlGUq6etEoG6OCs8LMRBFkVevq71bPx6FLPIkZEqJtGoHsxOm1mJGkYIWCBAgPUZ+0lfSQS0QnHv00gXuK9KDjc3U8CUf070SMXCEi11adLpIDMe2l8D+vHUrnohNTLwgl8fB4kRMyKH2YxgZ7lBMsWaQEwpyqt0I8QCopqcKd2GK7aSbmdAKzonFcNs/Kp7cnxcpVlk4e7IBdUAImOAcVcANqoA4weABP4Bm8aI/am/ahfY5bc1o2sw0mSvv6BtzZrRg=
    L0(y, ˆ
    y) =
    @L(y, ˆ
    y)

    y
    AAACN3icbVDLShxBFK3W+IyPMS6zKTIEFGHoFh/ZRRSGLCcwD2F6aKprbmsxVV1N1W3j0PTn+BF+g1tduArZSLb5g9SMgzrqgQuHc+7l3nviTAqLvn/vzcx+mJtfWFxa/riyurZe2fjUtjo3HFpcS21OY2ZBihRaKFDCaWaAqVhCJx6cjPzOBRgrdNrEYQY9xc5SkQjO0ElR5Xs9KnAnKLfCWBWX5TYNJSTIjNG/aD3CJ3mHhoCMhryvkTafjahS9Wv+GPQtCSakSiZoRJU/YV/zXEGKXDJru4GfYa9gBgWXUC6HuYWM8QE7g66jKVNge8X40ZJ+dUqfJtq4SpGO1ZcTBVPWDlXsOhXDc/vaG4nved0ck2+9QqRZjpDyx0VJLilqOkqN9oUBjnLoCONGuFspP2eGcXTZTm2JVekyCV4n8Ja0d2vBQW3/51716HiSziL5TL6QLRKQQ3JEfpAGaRFOrsgNuSV33rX323vw/j62zniTmU0yBe/ff+pRq+s=
    Ft+1(x) Ft(x) + ⌘ · Tt(x)
    AAACA3icbVDLSsNAFL2pr1pfVZduBotQNyURX8uiIC4r2Ae0oUymk3boTBJnJmIJWfoNbnXtTtz6IS79E6dtFrb1wIXDOfdyLseLOFPatr+t3NLyyupafr2wsbm1vVPc3WuoMJaE1knIQ9nysKKcBbSumea0FUmKhcdp0xtej/3mI5WKhcG9HkXUFbgfMJ8RrI3k3nQTnZY7nkie0uNusWRX7AnQInEyUoIMtW7xp9MLSSxooAnHSrUdO9JugqVmhNO00IkVjTAZ4j5tGxpgQZWbTJ5O0ZFResgPpZlAo4n69yLBQqmR8MymwHqg5r2x+J/XjrV/6SYsiGJNAzIN8mOOdIjGDaAek5RoPjIEE8nMr4gMsMREm55mUjyRmk6c+QYWSeOk4pxXzu5OS9WrrJ08HMAhlMGBC6jCLdSgDgQe4AVe4c16tt6tD+tzupqzspt9mIH19QuGOpiZ
    Ft(x)
    AAACB3icbVDLSsNAFJ3UV62PRl26GSxCRSiJ+FoWBXFZwdpCG8JkOmmHzkzCzEQsIR/gN7jVtTtx62e49E+ctlnY1gMXDufcy7mcIGZUacf5tgpLyyura8X10sbm1nbZ3tl9UFEiMWniiEWyHSBFGBWkqalmpB1LgnjASCsYXo/91iORikbiXo9i4nHUFzSkGGkj+Xb5xk/1sZtVuwFPn7Ij3644NWcCuEjcnFRAjoZv/3R7EU44ERozpFTHdWLtpUhqihnJSt1EkRjhIeqTjqECcaK8dPJ4Bg+N0oNhJM0IDSfq34sUcaVGPDCbHOmBmvfG4n9eJ9HhpZdSESeaCDwNChMGdQTHLcAelQRrNjIEYUnNrxAPkERYm65mUgKemU7c+QYWycNJzT2vnd2dVupXeTtFsA8OQBW44ALUwS1ogCbAIAEv4BW8Wc/Wu/VhfU5XC1Z+swdmYH39AuFKmTo=
    Ft+1(x)
    AAACEHicbVDLSgNBEJyNrxhfUS+Cl8EgRISwK76OQS8eI+QF2SXMTibJkJndZaZXDEv8CL/Bq569iVf/wKN/4iTZg0ksaCiquunu8iPBNdj2t5VZWl5ZXcuu5zY2t7Z38rt7dR3GirIaDUWomj7RTPCA1YCDYM1IMSJ9wRr+4HbsNx6Y0jwMqjCMmCdJL+BdTgkYqZ0/OMUuA4Jd2gkBV9tQdH2ZPI5O2vmCXbInwIvESUkBpai08z9uJ6SxZAFQQbRuOXYEXkIUcCrYKOfGmkWEDkiPtQwNiGTaSyYfjPCxUTq4GypTAeCJ+nciIVLrofRNpyTQ1/PeWPzPa8XQvfYSHkQxsIBOF3VjgSHE4zhwhytGQQwNIVRxcyumfaIIBRPazBZfjkwmznwCi6R+VnIuSxf354XyTZpOFh2iI1REDrpCZXSHKqiGKHpCL+gVvVnP1rv1YX1OWzNWOrOPZmB9/QLtWZx1
    +⌘ · Tt(x)
    A regression tree
    AAACAXicbVC7TgJBFJ3FF+ILtbSZSEywIbvGV0m0scQEkGTZkNlhFibMYzMzaySbrfwGW63tjK1fYumfOMAWAp7kJifn3Jt77wljRrVx3W+nsLK6tr5R3Cxtbe/s7pX3D9paJgqTFpZMqk6INGFUkJahhpFOrAjiISMP4eh24j88EqWpFE0zjknA0UDQiGJkrOQ3e6baDXn6lJ32yhW35k4Bl4mXkwrI0eiVf7p9iRNOhMEMae17bmyCFClDMSNZqZtoEiM8QgPiWyoQJzpIpydn8MQqfRhJZUsYOFX/TqSIaz3moe3kyAz1ojcR//P8xETXQUpFnBgi8GxRlDBoJJz8D/tUEWzY2BKEFbW3QjxECmFjU5rbEvLMZuItJrBM2mc177J2cX9eqd/k6RTBETgGVeCBK1AHd6ABWgADCV7AK3hznp1358P5nLUWnHzmEMzB+foFwcaXmw==
    Tt(x)
    AAACJXicbVDLSgMxFM3UV62vqks3wSK0oGVGfC2LblxWsA/olJJkMm1oMjMkGbEM8xV+hN/gVtfuRHAl/omZtoitHggczjmXe3NwxJnStv1h5RYWl5ZX8quFtfWNza3i9k5ThbEktEFCHso2RopyFtCGZprTdiQpEpjTFh5eZX7rjkrFwuBWjyLaFagfMJ8RpI3UKx65mPV52cUiuU977BC6AumBFElfIi/90StZTFZ6xZJdtceAf4kzJSUwRb1X/HK9kMSCBppwpFTHsSPdTZDUjHCaFtxY0QiRIerTjqEBElR1k/G3UnhgFA/6oTQv0HCs/p5IkFBqJLBJZkereS8T//M6sfYvugkLoljTgEwW+TGHOoRZR9BjkhLNR4YgIpm5FZIBkoho0+TMFixS04kz38Bf0jyuOmfV05uTUu1y2k4e7IF9UAYOOAc1cA3qoAEIeABP4Bm8WI/Wq/VmvU+iOWs6swtmYH1+AzBLpjY=
    xi, grad(xi)
    AAACDnicbVDLSsNAFJ3UV62vqODGTbCILUhJxNdGKAriwkUF+4A2hMl00g6dScLMRAwx/+A3uNW1O3HrL7j0T5y2WdjWAxcO59zLuRw3pERI0/zWcnPzC4tL+eXCyura+oa+udUQQcQRrqOABrzlQoEp8XFdEklxK+QYMpfipju4GvrNB8wFCfx7GYfYZrDnE48gKJXk6DsXtwel2CGH144sdVyWPKYOKZcdvWhWzBGMWWJlpAgy1Bz9p9MNUMSwLxGFQrQtM5R2ArkkiOK00IkEDiEawB5uK+pDhoWdjP5PjX2ldA0v4Gp8aYzUvxcJZELEzFWbDMq+mPaG4n9eO5LeuZ0QP4wk9tE4yIuoIQNjWIbRJRwjSWNFIOJE/WqgPuQQSVXZRIrLUtWJNd3ALGkcVazTysndcbF6mbWTB7tgD5SABc5AFdyAGqgDBJ7AC3gFb9qz9q59aJ/j1ZyW3WyDCWhfvwCem2I=
    = L0(yi, Ft(xi))
    AAACH3icbZDLSsNAFIYn9VbrLerSzWARK2hJireNUHTjwkUFe4Emlsl00g6dXJiZCCHkEXwIn8Gtrt2J2y59E6dtBNt6YODn/8/hnPmckFEhDWOo5RYWl5ZX8quFtfWNzS19e6chgohjUscBC3jLQYIw6pO6pJKRVsgJ8hxGms7gZpQ3nwgXNPAfZBwS20M9n7oUI6msjn54V4qPrT6SSZwewStouRzhxEyTSlqKT36Dx0pHLxplY1xwXpiZKIKsah392+oGOPKILzFDQrRNI5R2grikmJG0YEWChAgPUI+0lfSRR4SdjD+UwgPldKEbcPV8Ccfu34kEeULEnqM6PST7YjYbmf9l7Ui6l3ZC/TCSxMeTRW7EoAzgiA7sUk6wZLESCHOqboW4jxQSqRhObXG8VDExZwnMi0albJ6Xz+5Pi9XrjE4e7IF9UAImuABVcAtqoA4weAav4A28ay/ah/apfU1ac1o2swumShv+AEqronY=
    L(y, ˆ
    y) =
    1
    2
    (y ˆ
    y)2
    Example) Least squares
    AAACNXicbVDJSgNBEO1x34169NIYBD0YZsTtIoiCePCgYFTIhKGn05k06WXorlHDMF/jR/gNXvXkwYMgXv0FOzHg+qDg8V4VVfXiVHALvv/kDQwODY+Mjo1PTE5Nz8yW5ubPrc4MZVWqhTaXMbFMcMWqwEGwy9QwImPBLuL2Qde/uGLGcq3OoJOyuiSJ4k1OCTgpKu2Gx1olhictIMboaxxKAi0j88SQRrESxjK/KSK+indxJ+J4DR9G8KVGpbJf8XvAf0nQJ2XUx0lUegkbmmaSKaCCWFsL/BTqOTHAqWDFRJhZlhLaJgmrOaqIZLae994s8LJTGripjSsFuKd+n8iJtLYjY9fZfcL+9rrif14tg+ZOPecqzYAp+rmomQkMGnczww1uGAXRcYRQw92tmLaIIRRcsj+2xLJwmQS/E/hLztcrwVZl83SjvLffT2cMLaIltIICtI320BE6QVVE0S26Rw/o0bvznr1X7+2zdcDrzyygH/DePwAvYqwp
    =) grad(xi) = yi Ft(xi)

    View full-size slide

  144. Method 5: Gradient Boosted Trees
    Additive tree ensemble with iterations to add regression trees that better approximate the
    gradient sample values
    • Gradient Tree Boosting
    • GBM (Gradient Boosting Machines)
    • GBDT (Gradient Boosted Decision Trees)
    • GBRT (Gradient Boosted Regression Trees)
    • MART (Multiple Additive Regression Trees)
    • TreeNet
    • AnyBoost
    • Regression trees are used even for classification, but arbitrary loss functions can be learned
    • Can be viewed as approximate gradient descent in function space (Mason+ NIPS'99)
    • Various names for historical reasons
    Forest Forest Tree
    AAACTHicbZDPSxwxFMcza2vVVru1x16CS2EFXWakv46ilx56UNhVYbMMb7KZ3WAyE5I3xWE6/5h/hHdv9ljP3qTQ7DpQfz0IfPl838tLvolR0mEYXgathRcvF18tLa+8frO69rb9bv3I5YXlYsBzlduTBJxQMhMDlKjEibECdKLEcXK6P/OPfwrrZJ71sTRipGGSyVRyQI/idr8fY5clujqrY7lJGRhj8zPKlEixR7dZaoFXzIBFCYr+6JZbbApYlfVm/R83qKbMyskUf8XtTtgL50WfiqgRHdLUQdz+zcY5L7TIkCtwbhiFBkfV7HquRL3CCicM8FOYiKGXGWjhRtX89zX96MmYprn1J0M6p/cnKtDOlTrxnRpw6h57M/icNyww/TaqZGYKFBm/W5QWimJOZ1HSsbSCoyq9AG6lfyvlU/CBoQ/8wZZE1z6T6HECT8XRTi/60vt8+Kmzu9eks0Q+kA3SJRH5SnbJd3JABoSTc3JF/pDr4CK4CW6Dv3etraCZeU8eVGvxH19AtU0=
    Tt(xi) ⇡
    @L(y, ˆ
    y)

    y AAAB+3icbVBNS8NAEJ3Ur1q/qh69LBbBU0nEr4tQ9OKxgrGFNpTNdtMu3d2E3Y0QSn6DVz17E6/+GI/+E7dtDrb1wcDjvRlm5oUJZ9q47rdTWlldW98ob1a2tnd296r7B086ThWhPol5rNoh1pQzSX3DDKftRFEsQk5b4ehu4reeqdIslo8mS2gg8ECyiBFsrORnN1mP9ao1t+5OgZaJV5AaFGj2qj/dfkxSQaUhHGvd8dzEBGOsDCOc5pVuqmmCyQgPaMdSiQXVwXh6bI5OrNJHUaxsSYOm6t+JMRZaZyK0nQKboV70JuJ/Xic10XUwZjJJDZVktihKOTIxmnyO+kxRYnhmCSaK2VsRGWKFibH5zG0JRW4z8RYTWCZPZ3Xvsn7xcF5r3BbplOEIjuEUPLiCBtxDE3wgwOAFXuHNyZ1358P5nLWWnGLmEObgfP0CgWeVQw==
    y = yi
    AAACDXicbVDLSsNAFJ34rPVVFVduBotQNyURXxuhKIjLCvYBbQmT6aQdOpOEmRuxhHyD3+BW1+7Erd/g0j9x2mZhWw9cOJxzL+dyvEhwDbb9bS0sLi2vrObW8usbm1vbhZ3dug5jRVmNhiJUTY9oJnjAasBBsGakGJGeYA1vcDPyG49MaR4GDzCMWEeSXsB9TgkYyS3st/sEkmF6detCqe3J5Cl1+bFbKNpleww8T5yMFFGGqlv4aXdDGksWABVE65ZjR9BJiAJOBUvz7ViziNAB6bGWoQGRTHeS8fspPjJKF/uhMhMAHqt/LxIitR5Kz2xKAn09643E/7xWDP5lJ+FBFAML6CTIjwWGEI+6wF2uGAUxNIRQxc2vmPaJIhRMY1MpnkxNJ85sA/OkflJ2zstn96fFynXWTg4doENUQg66QBV0h6qohihK0At6RW/Ws/VufVifk9UFK7vZQ1Owvn4BOzecMQ==
    ˆ
    y = Ft(xi)

    View full-size slide

  145. 10.11 Right-Sized Trees for Boosting 361
    Algorithm 10.3 Gradient Tree Boosting Algorithm.
    1. Initialize f0(x) = arg minγ
    N
    i=1
    L(yi, γ).
    2. For m = 1 to M:
    (a) For i = 1, 2, . . . , N compute
    rim = −
    ∂L(yi, f(xi))
    ∂f(xi)
    f=fm−1
    .
    (b) Fit a regression tree to the targets rim giving terminal regions
    Rjm, j = 1, 2, . . . , Jm.
    (c) For j = 1, 2, . . . , Jm compute
    γjm = arg min
    γ
    xi
    ∈Rjm
    L (yi, fm−1(xi) + γ) .
    (d) Update fm(x) = fm−1(x) + Jm
    j=1
    γjmI(x ∈ Rjm).
    3. Output ˆ
    f(x) = fM (x).
    The algorithm for classification is similar. Lines 2(a)–(d) are repeated
    K times at each iteration m, once for each class using (10.38). The result
    at line 3 is K different (coupled) tree expansions fkM (x), k = 1, 2, . . . , K.
    These produce probabilities via (10.21) or do classification as in (10.20).
    Friedman’s gradient boosting
    https://hastie.su.domains/Papers/ESLII.pdf
    Hastie, Tibshirani, Friedman, The elements of statistical learning (2009)
    AAACL3icdVDLSgMxFM3UV62vqks3wSK0C8u0VOtGKOrChYta7AM6dcikmTY0kxmSjFCG+RQ/wm9wq2txI7r0L0xfYH0cuHA4517uvccJGJXKNF+NxMLi0vJKcjW1tr6xuZXe3mlIPxSY1LHPfNFykCSMclJXVDHSCgRBnsNI0xmcj/zmHRGS+vxGDQPS8VCPU5dipLRkp8vWBWEKZWUOnkLLFQhH3L6C3K7FEY9h1nKQiIaxlg7hjNdyt0U7nTHzxSNTA/4mhbw5RgZMUbXTH1bXx6FHuMIMSdkumIHqREgoihmJU1YoSYDwAPVIW1OOPCI70fjBGB5opQtdX+jiCo7V7xMR8qQceo7u9JDqy5/eSPzLa4fKPelElAehIhxPFrkhg8qHo7RglwqCFRtqgrCg+laI+0inpHSmc1scL9aZzB6H/5NGMV84zpeuS5nK2TSdJNgD+yALCqAMKuASVEEdYHAPHsETeDYejBfjzXiftCaM6cwumIPx+QUy6aiI
    (s) =
    nLnR
    n

    yL ¯
    yR)2
    AAACNXicdVDLSgMxFM34rO9Rl26CRWgXlmmpj40g6sKFiypWhc5YMmnahiaZIcmIwzBf40f4DW515cKFIG79BTN9gM8DF07OvZebc/yQUaUd59kaG5+YnJrOzczOzS8sLtnLKxcqiCQmdRywQF75SBFGBalrqhm5CiVB3Gfk0u8dZv3LGyIVDcS5jkPicdQRtE0x0kZq2nsuR7oreRLK4DZOC+4RYRoVVLEI96Bonpg6gwXXRzKJU/PchCN+VryuNO28U6psOQbwNymXnD7yYIha0351WwGOOBEaM6RUo+yE2kuQ1BQzks66kSIhwj3UIQ1DBeJEeUnfZgo3jNKC7UCaEhr21a8bCeJKxdw3k5kp9bOXiX/1GpFu73oJFWGkicCDQ+2IQR3ALDPYopJgzWJDEJbU/BXiLpIIa5Pstys+T00mI+Pwf3JRKZW3S9XTan7/YJhODqyBdVAAZbAD9sExqIE6wOAOPIBH8GTdWy/Wm/U+GB2zhjur4Busj0/4Bare
    proxy( (s)) = nLnR(¯
    yL ¯
    yR)2
    1. A special splitting criteria “Friedman’s
    MSE” is used instead of standard MSE.
    2. Instead of for the entire tree , each
    leaf constant is optimized separately.
    AAAB+nicbVA9TwJBEJ3DL8Qv1NJmIzGxInfGr5JoY4mJIAlcyN4yBxt27y67eyYE+Qu2WtsZW/+Mpf/EPbhCwJdM8vLeTGbmBYng2rjut1NYWV1b3yhulra2d3b3yvsHTR2nimGDxSJWrYBqFDzChuFGYCtRSGUg8DEY3mb+4xMqzePowYwS9CXtRzzkjJpM6qCh3XLFrbpTkGXi5aQCOerd8k+nF7NUYmSYoFq3PTcx/pgqw5nASamTakwoG9I+ti2NqETtj6e3TsiJVXokjJWtyJCp+ndiTKXWIxnYTknNQC96mfif105NeO2PeZSkBiM2WxSmgpiYZI+THlfIjBhZQpni9lbCBlRRZmw8c1sCObGZeIsJLJPmWdW7rF7cn1dqN3k6RTiCYzgFD66gBndQhwYwGMALvMKb8+y8Ox/O56y14OQzhzAH5+sXwhKU2A==

    AAAB+nicbVA9TwJBEJ3DL8Qv1NJmIzGxInfGr5JoY4mJIAlcyN4yBxt27y67eyYE+Qu2WtsZW/+Mpf/EPbhCwJdM8vLeTGbmBYng2rjut1NYWV1b3yhulra2d3b3yvsHTR2nimGDxSJWrYBqFDzChuFGYCtRSGUg8DEY3mb+4xMqzePowYwS9CXtRzzkjJpM6qCh3XLFrbpTkGXi5aQCOerd8k+nF7NUYmSYoFq3PTcx/pgqw5nASamTakwoG9I+ti2NqETtj6e3TsiJVXokjJWtyJCp+ndiTKXWIxnYTknNQC96mfif105NeO2PeZSkBiM2WxSmgpiYZI+THlfIjBhZQpni9lbCBlRRZmw8c1sCObGZeIsJLJPmWdW7rF7cn1dqN3k6RTiCYzgFD66gBndQhwYwGMALvMKb8+y8Ox/O56y14OQzhzAH5+sXwhKU2A==
    ⌘ AAAB/nicbVA9SwNBEJ2LXzF+RS1tDoNgFe7ErzJoYxnBxEByhLnNXrJkd+/Y3RPCEfA32GptJ7b+FUv/iZvkCpP4YODx3gwz88KEM20879sprKyurW8UN0tb2zu7e+X9g6aOU0Vog8Q8Vq0QNeVM0oZhhtNWoiiKkNPHcHg78R+fqNIslg9mlNBAYF+yiBE0Vmp1+igEdv1uueJVvSncZeLnpAI56t3yT6cXk1RQaQhHrdu+l5ggQ2UY4XRc6qSaJkiG2KdtSyUKqoNseu/YPbFKz41iZUsad6r+nchQaD0Soe0UaAZ60ZuI/3nt1ETXQcZkkhoqyWxRlHLXxO7kebfHFCWGjyxBopi91SUDVEiMjWhuSyjGNhN/MYFl0jyr+pfVi/vzSu0mT6cIR3AMp+DDFdTgDurQAAIcXuAV3pxn5935cD5nrQUnnzmEOThfv33nllk=
    1
    AAAB/nicbVDLSgNBEOyNrxhfUY9eBoPgKewGX8egF48RzAOSJfROJsmQmd1lZlYIS8Bv8Kpnb+LVX/HonzhJ9mASCxqKqm66u4JYcG1c99vJra1vbG7ltws7u3v7B8XDo4aOEkVZnUYiUq0ANRM8ZHXDjWCtWDGUgWDNYHQ39ZtPTGkehY9mHDNf4iDkfU7RWKnVGaCU2K10iyW37M5AVomXkRJkqHWLP51eRBPJQkMFat323Nj4KSrDqWCTQifRLEY6wgFrWxqiZNpPZ/dOyJlVeqQfKVuhITP170SKUuuxDGynRDPUy95U/M9rJ6Z/46c8jBPDQjpf1E8EMRGZPk96XDFqxNgSpIrbWwkdokJqbEQLWwI5sZl4ywmskkal7F2VLx8uStXbLJ08nMApnIMH11CFe6hBHSgIeIFXeHOenXfnw/mct+acbOYYFuB8/QJ/epZa
    2
    AAAB/nicbVDLSgNBEOyNrxhfUY9eFoPgKez6Pga9eIxgHpAsoXcymwyZmV1mZoUQAn6DVz17E6/+ikf/xEmyB5NY0FBUddPdFSacaeN5305uZXVtfSO/Wdja3tndK+4f1HWcKkJrJOaxaoaoKWeS1gwznDYTRVGEnDbCwd3EbzxRpVksH80woYHAnmQRI2is1Gz3UAjsnHeKJa/sTeEuEz8jJchQ7RR/2t2YpIJKQzhq3fK9xAQjVIYRTseFdqppgmSAPdqyVKKgOhhN7x27J1bpulGsbEnjTtW/EyMUWg9FaDsFmr5e9Cbif14rNdFNMGIySQ2VZLYoSrlrYnfyvNtlihLDh5YgUcze6pI+KiTGRjS3JRRjm4m/mMAyqZ+V/avy5cNFqXKbpZOHIziGU/DhGipwD1WoAQEOL/AKb86z8+58OJ+z1pyTzRzCHJyvX4ENlls=
    3
    AAAB/nicbVDLSgNBEOyNrxhfUY9eBoPgKexKfByDXjxGMA9IltA7mSRDZnaXmVkhLAG/wauevYlXf8Wjf+Ik2YNJLGgoqrrp7gpiwbVx3W8nt7a+sbmV3y7s7O7tHxQPjxo6ShRldRqJSLUC1EzwkNUNN4K1YsVQBoI1g9Hd1G8+MaV5FD6accx8iYOQ9zlFY6VWZ4BSYrfSLZbcsjsDWSVeRkqQodYt/nR6EU0kCw0VqHXbc2Pjp6gMp4JNCp1EsxjpCAesbWmIkmk/nd07IWdW6ZF+pGyFhszUvxMpSq3HMrCdEs1QL3tT8T+vnZj+jZ/yME4MC+l8UT8RxERk+jzpccWoEWNLkCpubyV0iAqpsREtbAnkxGbiLSewShoXZe+qfPlQKVVvs3TycAKncA4eXEMV7qEGdaAg4AVe4c15dt6dD+dz3ppzspljWIDz9QuCoJZc
    4

    View full-size slide

  146. Customize the loss function as you like
    F0
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sRh/eYWV1bX1jeJmaWt7Z3evvH/QNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0cK9cwVU8A1omfk4qkKPeK393+zFNJVOWCmJMx8eJDTKiLaeCTUrd1LCE0BEZsI6jikhmgmz26gSdOKWPoli7URbN1N8XGZHGjGXoNiWxQ7PoTcV/vVAuJNvoMsi4SlLLFJ0HR6lANkbTHlCfa0atGDtCqObud0SHRBNqXVslV4q/WMEyaZ5VfVz1784rtau8niIcwTGcgg8XUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/wJlBM=
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sRh/eYWV1bX1jeJmaWt7Z3evvH/QNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0cK9cwVU8A1omfk4qkKPeK393+zFNJVOWCmJMx8eJDTKiLaeCTUrd1LCE0BEZsI6jikhmgmz26gSdOKWPoli7URbN1N8XGZHGjGXoNiWxQ7PoTcV/vVAuJNvoMsi4SlLLFJ0HR6lANkbTHlCfa0atGDtCqObud0SHRBNqXVslV4q/WMEyaZ5VfVz1784rtau8niIcwTGcgg8XUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/wJlBM=
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sRh/eYWV1bX1jeJmaWt7Z3evvH/QNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0cK9cwVU8A1omfk4qkKPeK393+zFNJVOWCmJMx8eJDTKiLaeCTUrd1LCE0BEZsI6jikhmgmz26gSdOKWPoli7URbN1N8XGZHGjGXoNiWxQ7PoTcV/vVAuJNvoMsi4SlLLFJ0HR6lANkbTHlCfa0atGDtCqObud0SHRBNqXVslV4q/WMEyaZ5VfVz1784rtau8niIcwTGcgg8XUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/wJlBM=
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sRh/eYWV1bX1jeJmaWt7Z3evvH/QNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0cK9cwVU8A1omfk4qkKPeK393+zFNJVOWCmJMx8eJDTKiLaeCTUrd1LCE0BEZsI6jikhmgmz26gSdOKWPoli7URbN1N8XGZHGjGXoNiWxQ7PoTcV/vVAuJNvoMsi4SlLLFJ0HR6lANkbTHlCfa0atGDtCqObud0SHRBNqXVslV4q/WMEyaZ5VfVz1784rtau8niIcwTGcgg8XUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/wJlBM=
    F1
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sb7/5RVWVtfWN4qbpa3tnd298v5B08SppqxBYxHrdkgME1yxhuVWsHaiGZGhYK1wdD31W49MGx6rBztOWCDJQPGIU2KddH/Tw71yxa/6M6BlgnNSgRz1Xvm7249pKpmyVBBjOthPbJARbTkVbFLqpoYlhI7IgHUcVUQyE2SzVyfoxCl9FMXajbJopv6+yIg0ZixDtymJHZpFbyr+64VyIdlGl0HGVZJapug8OEoFsjGa9oD6XDNqxdgRQjV3vyM6JJpQ69oquVLwYgXLpHlWxX4V351Xald5PUU4gmM4BQwXUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/2dlBQ=
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sb7/5RVWVtfWN4qbpa3tnd298v5B08SppqxBYxHrdkgME1yxhuVWsHaiGZGhYK1wdD31W49MGx6rBztOWCDJQPGIU2KddH/Tw71yxa/6M6BlgnNSgRz1Xvm7249pKpmyVBBjOthPbJARbTkVbFLqpoYlhI7IgHUcVUQyE2SzVyfoxCl9FMXajbJopv6+yIg0ZixDtymJHZpFbyr+64VyIdlGl0HGVZJapug8OEoFsjGa9oD6XDNqxdgRQjV3vyM6JJpQ69oquVLwYgXLpHlWxX4V351Xald5PUU4gmM4BQwXUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/2dlBQ=
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sb7/5RVWVtfWN4qbpa3tnd298v5B08SppqxBYxHrdkgME1yxhuVWsHaiGZGhYK1wdD31W49MGx6rBztOWCDJQPGIU2KddH/Tw71yxa/6M6BlgnNSgRz1Xvm7249pKpmyVBBjOthPbJARbTkVbFLqpoYlhI7IgHUcVUQyE2SzVyfoxCl9FMXajbJopv6+yIg0ZixDtymJHZpFbyr+64VyIdlGl0HGVZJapug8OEoFsjGa9oD6XDNqxdgRQjV3vyM6JJpQ69oquVLwYgXLpHlWxX4V351Xald5PUU4gmM4BQwXUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/2dlBQ=
    AAAB+nicbVDLSgMxFL1TX7W+qi7dBIvgqkxE0GVREJcV7QPaoWTSTBuaZIYkI5Sxn+BW9+7ErT/j1i8xbWehrQcuHM65l3M5YSK4sb7/5RVWVtfWN4qbpa3tnd298v5B08SppqxBYxHrdkgME1yxhuVWsHaiGZGhYK1wdD31W49MGx6rBztOWCDJQPGIU2KddH/Tw71yxa/6M6BlgnNSgRz1Xvm7249pKpmyVBBjOthPbJARbTkVbFLqpoYlhI7IgHUcVUQyE2SzVyfoxCl9FMXajbJopv6+yIg0ZixDtymJHZpFbyr+64VyIdlGl0HGVZJapug8OEoFsjGa9oD6XDNqxdgRQjV3vyM6JJpQ69oquVLwYgXLpHlWxX4V351Xald5PUU4gmM4BQwXUINbqEMDKAzgGV7g1Xvy3rx372O+WvDym0P4A+/zB/2dlBQ=
    F2
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFkUxGVF+4B2KJk004YmmSHJCGXsJ7jVvTtx68+49UtM21lo64ELh3Pu5VxOmAhurOd9oZXVtfWNzcJWcXtnd2+/dHDYNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0qr1S2at4M+Bl4uekDDnqvdJ3tx/TVDJlqSDGdHwvsUFGtOVUsEmxmxqWEDoiA9ZxVBHJTJDNXp3gU6f0cRRrN8rimfr7IiPSmLEM3aYkdmgWvan4rxfKhWQbXQYZV0lqmaLz4CgV2MZ42gPuc82oFWNHCNXc/Y7pkGhCrWur6ErxFytYJs1qxfcq/t15uXaV11OAYziBM/DhAmpwC3VoAIUBPMMLvKIn9Ibe0cd8dQXlN0fwB+jzB/8xlBU=
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFkUxGVF+4B2KJk004YmmSHJCGXsJ7jVvTtx68+49UtM21lo64ELh3Pu5VxOmAhurOd9oZXVtfWNzcJWcXtnd2+/dHDYNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0qr1S2at4M+Bl4uekDDnqvdJ3tx/TVDJlqSDGdHwvsUFGtOVUsEmxmxqWEDoiA9ZxVBHJTJDNXp3gU6f0cRRrN8rimfr7IiPSmLEM3aYkdmgWvan4rxfKhWQbXQYZV0lqmaLz4CgV2MZ42gPuc82oFWNHCNXc/Y7pkGhCrWur6ErxFytYJs1qxfcq/t15uXaV11OAYziBM/DhAmpwC3VoAIUBPMMLvKIn9Ibe0cd8dQXlN0fwB+jzB/8xlBU=
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFkUxGVF+4B2KJk004YmmSHJCGXsJ7jVvTtx68+49UtM21lo64ELh3Pu5VxOmAhurOd9oZXVtfWNzcJWcXtnd2+/dHDYNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0qr1S2at4M+Bl4uekDDnqvdJ3tx/TVDJlqSDGdHwvsUFGtOVUsEmxmxqWEDoiA9ZxVBHJTJDNXp3gU6f0cRRrN8rimfr7IiPSmLEM3aYkdmgWvan4rxfKhWQbXQYZV0lqmaLz4CgV2MZ42gPuc82oFWNHCNXc/Y7pkGhCrWur6ErxFytYJs1qxfcq/t15uXaV11OAYziBM/DhAmpwC3VoAIUBPMMLvKIn9Ibe0cd8dQXlN0fwB+jzB/8xlBU=
    AAAB+nicbVDLSgMxFL3xWeur6tJNsAiuykwRdFkUxGVF+4B2KJk004YmmSHJCGXsJ7jVvTtx68+49UtM21lo64ELh3Pu5VxOmAhurOd9oZXVtfWNzcJWcXtnd2+/dHDYNHGqKWvQWMS6HRLDBFesYbkVrJ1oRmQoWCscXU/91iPThsfqwY4TFkgyUDzilFgn3d/0qr1S2at4M+Bl4uekDDnqvdJ3tx/TVDJlqSDGdHwvsUFGtOVUsEmxmxqWEDoiA9ZxVBHJTJDNXp3gU6f0cRRrN8rimfr7IiPSmLEM3aYkdmgWvan4rxfKhWQbXQYZV0lqmaLz4CgV2MZ42gPuc82oFWNHCNXc/Y7pkGhCrWur6ErxFytYJs1qxfcq/t15uXaV11OAYziBM/DhAmpwC3VoAIUBPMMLvKIn9Ibe0cd8dQXlN0fwB+jzB/8xlBU=
    F3
    AAAB+nicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegIB4jmgckS5idzCZD5rHMzAphzSd41bs38erPePVLnCR70GhBQ1HVTTUVJZwZ6/ufXmFpeWV1rbhe2tjc2t4p7+41jUo1oQ2iuNLtCBvKmaQNyyyn7URTLCJOW9Hoauq3Hqg2TMl7O05oKPBAspgRbJ10d9077ZUrftWfAf0lQU4qkKPeK391+4qkgkpLODamE/iJDTOsLSOcTkrd1NAEkxEe0I6jEgtqwmz26gQdOaWPYqXdSItm6s+LDAtjxiJymwLboVn0puK/XiQWkm18EWZMJqmlksyD45Qjq9C0B9RnmhLLx45gopn7HZEh1phY11bJlRIsVvCXNE+qgV8Nbs8qtcu8niIcwCEcQwDnUIMbqEMDCAzgCZ7hxXv0Xr03732+WvDym334Be/jGwDUlBY=
    AAAB+nicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegIB4jmgckS5idzCZD5rHMzAphzSd41bs38erPePVLnCR70GhBQ1HVTTUVJZwZ6/ufXmFpeWV1rbhe2tjc2t4p7+41jUo1oQ2iuNLtCBvKmaQNyyyn7URTLCJOW9Hoauq3Hqg2TMl7O05oKPBAspgRbJ10d9077ZUrftWfAf0lQU4qkKPeK391+4qkgkpLODamE/iJDTOsLSOcTkrd1NAEkxEe0I6jEgtqwmz26gQdOaWPYqXdSItm6s+LDAtjxiJymwLboVn0puK/XiQWkm18EWZMJqmlksyD45Qjq9C0B9RnmhLLx45gopn7HZEh1phY11bJlRIsVvCXNE+qgV8Nbs8qtcu8niIcwCEcQwDnUIMbqEMDCAzgCZ7hxXv0Xr03732+WvDym334Be/jGwDUlBY=
    AAAB+nicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegIB4jmgckS5idzCZD5rHMzAphzSd41bs38erPePVLnCR70GhBQ1HVTTUVJZwZ6/ufXmFpeWV1rbhe2tjc2t4p7+41jUo1oQ2iuNLtCBvKmaQNyyyn7URTLCJOW9Hoauq3Hqg2TMl7O05oKPBAspgRbJ10d9077ZUrftWfAf0lQU4qkKPeK391+4qkgkpLODamE/iJDTOsLSOcTkrd1NAEkxEe0I6jEgtqwmz26gQdOaWPYqXdSItm6s+LDAtjxiJymwLboVn0puK/XiQWkm18EWZMJqmlksyD45Qjq9C0B9RnmhLLx45gopn7HZEh1phY11bJlRIsVvCXNE+qgV8Nbs8qtcu8niIcwCEcQwDnUIMbqEMDCAzgCZ7hxXv0Xr03732+WvDym334Be/jGwDUlBY=
    AAAB+nicbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegIB4jmgckS5idzCZD5rHMzAphzSd41bs38erPePVLnCR70GhBQ1HVTTUVJZwZ6/ufXmFpeWV1rbhe2tjc2t4p7+41jUo1oQ2iuNLtCBvKmaQNyyyn7URTLCJOW9Hoauq3Hqg2TMl7O05oKPBAspgRbJ10d9077ZUrftWfAf0lQU4qkKPeK391+4qkgkpLODamE/iJDTOsLSOcTkrd1NAEkxEe0I6jEgtqwmz26gQdOaWPYqXdSItm6s+LDAtjxiJymwLboVn0puK/XiQWkm18EWZMJqmlksyD45Qjq9C0B9RnmhLLx45gopn7HZEh1phY11bJlRIsVvCXNE+qgV8Nbs8qtcu8niIcwCEcQwDnUIMbqEMDCAzgCZ7hxXv0Xr03732+WvDym334Be/jGwDUlBY=
    F10
    AAAB/XicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7APaoWTSTBubZIYkI5Sh+Atude9O3Potbv0S03YW2nrgwuGcezmXEyaCG4vxl1dYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0ybXis7u04YYEkA8UjTol1UvOml/l40itXcBXPgJaJn5MK5Kj3yt/dfkxTyZSlghjT8XFig4xoy6lgk1I3NSwhdEQGrOOoIpKZIJt9O0EnTumjKNZulEUz9fdFRqQxYxm6TUns0Cx6U/FfL5QLyTa6DDKuktQyRefBUSqQjdG0CtTnmlErxo4Qqrn7HdEh0YRaV1jJleIvVrBMmmdVH1f9u/NK7SqvpwhHcAyn4MMF1OAW6tAACg/wDC/w6j15b9679zFfLXj5zSH8gff5A0PKlVo=
    AAAB/XicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7APaoWTSTBubZIYkI5Sh+Atude9O3Potbv0S03YW2nrgwuGcezmXEyaCG4vxl1dYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0ybXis7u04YYEkA8UjTol1UvOml/l40itXcBXPgJaJn5MK5Kj3yt/dfkxTyZSlghjT8XFig4xoy6lgk1I3NSwhdEQGrOOoIpKZIJt9O0EnTumjKNZulEUz9fdFRqQxYxm6TUns0Cx6U/FfL5QLyTa6DDKuktQyRefBUSqQjdG0CtTnmlErxo4Qqrn7HdEh0YRaV1jJleIvVrBMmmdVH1f9u/NK7SqvpwhHcAyn4MMF1OAW6tAACg/wDC/w6j15b9679zFfLXj5zSH8gff5A0PKlVo=
    AAAB/XicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7APaoWTSTBubZIYkI5Sh+Atude9O3Potbv0S03YW2nrgwuGcezmXEyaCG4vxl1dYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0ybXis7u04YYEkA8UjTol1UvOml/l40itXcBXPgJaJn5MK5Kj3yt/dfkxTyZSlghjT8XFig4xoy6lgk1I3NSwhdEQGrOOoIpKZIJt9O0EnTumjKNZulEUz9fdFRqQxYxm6TUns0Cx6U/FfL5QLyTa6DDKuktQyRefBUSqQjdG0CtTnmlErxo4Qqrn7HdEh0YRaV1jJleIvVrBMmmdVH1f9u/NK7SqvpwhHcAyn4MMF1OAW6tAACg/wDC/w6j15b9679zFfLXj5zSH8gff5A0PKlVo=
    AAAB/XicbVDLSgMxFL1TX7W+qi7dBIvgqmRE0GVREJcV7APaoWTSTBubZIYkI5Sh+Atude9O3Potbv0S03YW2nrgwuGcezmXEyaCG4vxl1dYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0ybXis7u04YYEkA8UjTol1UvOml/l40itXcBXPgJaJn5MK5Kj3yt/dfkxTyZSlghjT8XFig4xoy6lgk1I3NSwhdEQGrOOoIpKZIJt9O0EnTumjKNZulEUz9fdFRqQxYxm6TUns0Cx6U/FfL5QLyTa6DDKuktQyRefBUSqQjdG0CtTnmlErxo4Qqrn7HdEh0YRaV1jJleIvVrBMmmdVH1f9u/NK7SqvpwhHcAyn4MMF1OAW6tAACg/wDC/w6j15b9679zFfLXj5zSH8gff5A0PKlVo=
    http://arogozhnikov.github.io/2016/06/24/gradient_boosting_explained.html
    (softmax)
    Loss Gradient
    Regression (MSE)
    Regression (MAE)
    K-class classi caiton
    (cross entropy)
    MSE case
    AAACBHicbVDLSsNAFJ3UV62vqks3g0UQhJKIr2XRjcsKfUETy2QyaYfOJGHmRiihW7/Bra7diVv/w6V/4rTNwrYeuHA4517O5fiJ4Bps+9sqrKyurW8UN0tb2zu7e+X9g5aOU0VZk8YiVh2faCZ4xJrAQbBOohiRvmBtf3g38dtPTGkeRw0YJcyTpB/xkFMCRno8cxkQlwYx4EbP7pUrdtWeAi8TJycVlKPeK/+4QUxTySKggmjddewEvIwo4FSwcclNNUsIHZI+6xoaEcm0l02/HuMTowQ4jJWZCPBU/XuREan1SPpmUxIY6EVvIv7ndVMIb7yMR0kKLKKzoDAVGGI8qQAHXDEKYmQIoYqbXzEdEEUomKLmUnw5Np04iw0sk9Z51bmqXj5cVGq3eTtFdISO0Sly0DWqoXtUR01EkUIv6BW9Wc/Wu/Vhfc5WC1Z+c4jmYH39Al0vmHA=
    +⌘ · T0
    AAACBHicbVDLSsNAFJ3UV62vqks3g0UQhJKIr2XRjcsKfUETy2QyaYfOJGHmRiihW7/Bra7diVv/w6V/4rTNwrYeuHA4517O5fiJ4Bps+9sqrKyurW8UN0tb2zu7e+X9g5aOU0VZk8YiVh2faCZ4xJrAQbBOohiRvmBtf3g38dtPTGkeRw0YJcyTpB/xkFMCRno8cxkQlwYx4EbP6ZUrdtWeAi8TJycVlKPeK/+4QUxTySKggmjddewEvIwo4FSwcclNNUsIHZI+6xoaEcm0l02/HuMTowQ4jJWZCPBU/XuREan1SPpmUxIY6EVvIv7ndVMIb7yMR0kKLKKzoDAVGGI8qQAHXDEKYmQIoYqbXzEdEEUomKLmUnw5Np04iw0sk9Z51bmqXj5cVGq3eTtFdISO0Sly0DWqoXtUR01EkUIv6BW9Wc/Wu/Vhfc5WC1Z+c4jmYH39Al7CmHE=
    +⌘ · T1 AAACBHicbVDLSsNAFJ34rPVVdelmsAiCUJLia1l047JCX9DEMplM2qGTSZi5EUro1m9wq2t34tb/cOmfOG2zsK0HLhzOuZdzOX4iuAbb/rZWVtfWNzYLW8Xtnd29/dLBYUvHqaKsSWMRq45PNBNcsiZwEKyTKEYiX7C2P7yb+O0npjSPZQNGCfMi0pc85JSAkR7PXQbEpUEMuNGr9kplu2JPgZeJk5MyylHvlX7cIKZpxCRQQbTuOnYCXkYUcCrYuOimmiWEDkmfdQ2VJGLay6Zfj/GpUQIcxsqMBDxV/15kJNJ6FPlmMyIw0IveRPzP66YQ3ngZl0kKTNJZUJgKDDGeVIADrhgFMTKEUMXNr5gOiCIUTFFzKX40Np04iw0sk1a14lxVLh8uyrXbvJ0COkYn6Aw56BrV0D2qoyaiSKEX9IrerGfr3fqwPmerK1Z+c4TmYH39AmBVmHI=
    +⌘ · T2
    AAACBHicbVDLSsNAFJ34rPVVdelmsAiCUBLfy6IblxX6giaWyWTSDp1kwsyNUEK3foNbXbsTt/6HS//EaZuFbT1w4XDOvZzL8RPBNdj2t7W0vLK6tl7YKG5ube/slvb2m1qmirIGlUKqtk80EzxmDeAgWDtRjES+YC1/cDf2W09MaS7jOgwT5kWkF/OQUwJGejx1GRCXBhJwvXveLZXtij0BXiROTsooR61b+nEDSdOIxUAF0brj2Al4GVHAqWCjoptqlhA6ID3WMTQmEdNeNvl6hI+NEuBQKjMx4In69yIjkdbDyDebEYG+nvfG4n9eJ4Xwxst4nKTAYjoNClOBQeJxBTjgilEQQ0MIVdz8immfKELBFDWT4kcj04kz38AiaZ5VnKvK5cNFuXqbt1NAh+gInSAHXaMqukc11EAUKfSCXtGb9Wy9Wx/W53R1ycpvDtAMrK9fYeiYcw==
    +⌘ · T3
    AAACBHicbVDLSsNAFJ34rPVVdelmsAiCUBLxuSu6cVmhL2himUwm7dBJJszcCCV06ze41bU7cet/uPRPnLZZ2NYDFw7n3Mu5HD8RXINtf1tLyyura+uFjeLm1vbObmlvv6llqihrUCmkavtEM8Fj1gAOgrUTxUjkC9byB3djv/XElOYyrsMwYV5EejEPOSVgpMdTlwFxaSAB17s33VLZrtgT4EXi5KSMctS6pR83kDSNWAxUEK07jp2AlxEFnAo2KrqpZgmhA9JjHUNjEjHtZZOvR/jYKAEOpTITA56ofy8yEmk9jHyzGRHo63lvLP7ndVIIr72Mx0kKLKbToDAVGCQeV4ADrhgFMTSEUMXNr5j2iSIUTFEzKX40Mp048w0skuZZxbmsXDycl6u3eTsFdIiO0Aly0BWqontUQw1EkUIv6BW9Wc/Wu/VhfU5Xl6z85gDNwPr6BWtamHk=
    +⌘ · T9

    View full-size slide

  147. What impurity to use in regression of gradient values?
    A regression tree
    Partitions AAACDHicbVDLTgIxFO3gC/E1PnZuGomJC0Jm8LkkunGJRJAEJpNO6UBD25m0HRMk/ILf4FbX7oxb/8Glf2IHZiHgSdqcnHNP7s0JYkaVdpxvK7e0vLK6ll8vbGxube/Yu3tNFSUSkwaOWCRbAVKEUUEammpGWrEkiAeMPASDm9R/eCRS0Ujc62FMPI56goYUI20k3z6o+24J1v1K+p2WYKcbaeXbRafsTAAXiZuRIshQ8+0fk8MJJ0JjhpRqu06svRGSmmJGxoVOokiM8AD1SNtQgThR3mhy/RgeG6ULw0iaJzScqH8TI8SVGvLATHKk+2reS8X/vHaiwytvREWcaCLwdFGYMKgjmFYBu1QSrNnQEIQlNbdC3EcSYW0Km9kS8LHpxJ1vYJE0K2X3onx+d1asXmft5MEhOAInwAWXoApuQQ00AAZP4AW8gjfr2Xq3PqzP6WjOyjL7YAbW1y8bs5m2
    R1, R2, R3, . . .
    Constants AAACDHicbVC7TsMwFHXKq5RXeGwsFhUSQ1Ul5TlWsDAWidJKbRQ5rtNatZ3IdpBK1F/gG1hhZkOs/AMjf4LTZqAtR7J1dM49ulcniBlV2nG+rcLS8srqWnG9tLG5tb1j7+49qCiRmDRxxCLZDpAijArS1FQz0o4lQTxgpBUMbzK/9UikopG416OYeBz1BQ0pRtpIvn2AfbcCsV/LvtMK7PYirXy77FSdCeAicXNSBjkavv1jcjjhRGjMkFId14m1lyKpKWZkXOomisQID1GfdAwViBPlpZPrx/DYKD0YRtI8oeFE/ZtIEVdqxAMzyZEeqHkvE//zOokOr7yUijjRRODpojBhUEcwqwL2qCRYs5EhCEtqboV4gCTC2hQ2syXgY9OJO9/AInmoVd2L6vndWbl+nbdTBIfgCJwAF1yCOrgFDdAEGDyBF/AK3qxn6936sD6nowUrz+yDGVhfv27Gmek=
    c1, c2, c3, . . .
    What we want to minimize:
    AAACJHicbZDLSgMxFIYz9VbrrerSTbAILZYyI942QlEQFy4q9AZtHTJp2oYmmSHJiGWYl/AhfAa3unYnLtwIvonpBbTVHwIf/zmHc/J7AaNK2/aHlZibX1hcSi6nVlbX1jfSm1tV5YcSkwr2mS/rHlKEUUEqmmpG6oEkiHuM1Lz+xbBeuyNSUV+U9SAgLY66gnYoRtpYbjrfVCF3I3rmxLcCXmcHLs3Dy2zT49F97NLcfvmHc246YxfskeBfcCaQAROV3PRXs+3jkBOhMUNKNRw70K0ISU0xI3GqGSoSINxHXdIwKBAnqhWNfhXDPeO0YceX5gkNR+7viQhxpQbcM50c6Z6arQ3N/2qNUHdOWxEVQaiJwONFnZBB7cNhRLBNJcGaDQwgLKm5FeIekghrE+TUFo/HJhNnNoG/UD0oOMeFo5vDTPF8kk4S7IBdkAUOOAFFcAVKoAIweABP4Bm8WI/Wq/VmvY9bE9ZkZhtMyfr8Bqm5pCU=
    n
    X
    i=1
    L(yi, F(xi) + T(xi))
    AAACXHicdZFPS8MwGMaz+t85rQpevASHMA+OdvjvKHrxqOJ0sM6SZmkXl6Q1ScVR+gX9Bl4Ev4FXPZluO7ipLwR+PO/z8iZPgoRRpR3ntWTNzM7NLywuLZdXKqtr9vrGrYpTiUkTxyyWrQApwqggTU01I61EEsQDRu6C/nnRv3siUtFY3OhBQjocRYKGFCNtJN/uehF5VAwJDfe9UCKcuXnWyOGIvYBGrOaplPuGefac+xR6VMBr/yGHkU8Lg9y7b+TZf6aeT3Pfrjp1Z1jwN7hjqIJxXfr2u9eNccqJ0Jghpdquk+hOhqSmmJF82UsVSRDuo4i0DQrEiepkwzRyuGuULgxjaY551lD9OZEhrtSAB8bJke6p6V4h/tVrpzo86WRUJKkmAo8WhSmDOoZFtLBLJcGaDQwgLKm5K8Q9ZHLU5gMmtgS8yMSdTuA33Dbq7lH98Oqgeno2TmcRbIMdUAMuOAan4AJcgibA4AV8gE/wVXqzZq2yVRlZrdJ4ZhNMlLX1DSf6uP4=
    > 1
    2
    P
    xi
    2Rj
    gi
    2
    P
    xi
    2Rj
    hi
    The minimum when we tune AAAB+XicbVC7TsNAEFyHVwivACXNiQiJKrIRrzKChjII8pASKzpfzsmRu7N1d0aKrHwCLdR0iJavoeRPuDguSMJIK41mdrW7E8ScaeO6305hZXVtfaO4Wdra3tndK+8fNHWUKEIbJOKRagdYU84kbRhmOG3HimIRcNoKRrdTv/VMlWaRfDTjmPoCDyQLGcHGSg+k99QrV9yqmwEtEy8nFchR75V/uv2IJIJKQzjWuuO5sfFTrAwjnE5K3UTTGJMRHtCOpRILqv00O3WCTqzSR2GkbEmDMvXvRIqF1mMR2E6BzVAvelPxP6+TmPDaT5mME0MlmS0KE45MhKZ/oz5TlBg+tgQTxeytiAyxwsTYdOa2BGJiM/EWE1gmzbOqd1m9uD+v1G7ydIpwBMdwCh5cQQ3uoA4NIDCAF3iFNyd13p0P53PWWnDymUOYg/P1C/UBlGQ=
    cj
    Use this for the impurity for AAAB+XicdVDLTgJBEOz1ifhCPXqZSEw8bRYCAjeiF4/44JHAhswOszAyO7uZmTUhGz7Bq569Ga9+jUf/xAHWRIxW0kmlqjvdXV7EmdKO82GtrK6tb2xmtrLbO7t7+7mDw5YKY0lok4Q8lB0PK8qZoE3NNKedSFIceJy2vfHlzG8/UKlYKO70JKJugIeC+YxgbaTbm/59P5d3bGcO5NjFWrVcLRlSKddKhhRSKw8pGv3cZ28QkjigQhOOleoWnEi7CZaaEU6n2V6saITJGA9p11CBA6rcZH7qFJ0aZYD8UJoSGs3VnxMJDpSaBJ7pDLAeqd/eTPzL68bar7oJE1GsqSCLRX7MkQ7R7G80YJISzSeGYCKZuRWREZaYaJPO0hYvmJpMvh9H/5NW0S6c2+XrUr5+kaaTgWM4gTMoQAXqcAUNaAKBITzCEzxbifVivVpvi9YVK505giVY719Gz5Sc
    Rj
    AAACMnicbZDLSgMxFIYzXmu9VV26CRahRSkz4g1BKAriwkUVe4FOGTJpqmmTzJBkxDLMu/gQPoNb3equuPUhTNsRvB0IfPz/OZyT3w8ZVdq2X62JyanpmdnMXHZ+YXFpObeyWlNBJDGp4oAFsuEjRRgVpKqpZqQRSoK4z0jd750O/fodkYoG4lr3Q9Li6EbQDsVIG8nLHR1DV0Xci7tJCq7P4/vEo9ClAl55Rr8o9D26Dc8KX1ZxC3vdopfL2yV7VPAvOCnkQVoVLzdw2wGOOBEaM6RU07FD3YqR1BQzkmTdSJEQ4R66IU2DAnGiWvHojwncNEobdgJpntBwpH6fiBFXqs9908mRvlW/vaH4n9eMdOewFVMRRpoIPF7UiRjUARwGBttUEqxZ3wDCkppbIb5FEmFtYv2xxeeJycT5ncBfqO2UnP3S3uVuvnySppMB62ADFIADDkAZnIMKqAIMHsATeAYv1qP1Zg2s93HrhJXOrIEfZX18AkBPqhI=
    =
    X
    j
    X
    xi
    2Rj
    L(yi, F(xi) + cj)
    Separate cases by partition
    Constant
    Quadratic Taylor
    approximation
    AAACEXicbVDLSsNAFJ3UV62vqCtxM1jEFqQk4msjFAVx4aKCfUBbwmQ6bYfOJGFmIoYQ/Ai/wa2u3Ylbv8Clf+K0zcK2HrhwOOde7r3HDRiVyrK+jczc/MLiUnY5t7K6tr5hbm7VpB8KTKrYZ75ouEgSRj1SVVQx0ggEQdxlpO4OroZ+/YEISX3vXkUBaXPU82iXYqS05Jg7PYfCC3h7UIgcenhdaLk8fkwcWiw6Zt4qWSPAWWKnJA9SVBzzp9XxcciJpzBDUjZtK1DtGAlFMSNJrhVKEiA8QD3S1NRDnMh2PHohgfta6cCuL3R5Co7UvxMx4lJG3NWdHKm+nPaG4n9eM1Td83ZMvSBUxMPjRd2QQeXDYR6wQwXBikWaICyovhXiPhIIK53axBaXJzoTezqBWVI7KtmnpZO743z5Mk0nC3bBHigAG5yBMrgBFVAFGDyBF/AK3oxn4934MD7HrRkjndkGEzC+fgF36Zwc
    gi = L0(yi, F(xi))
    AAACEnicbVDLSgMxFM34rPVVdaebYJG2IGVGfG2EoiAuXFSwD2iHIZNm2tAkMyQZsQwFP8JvcKtrd+LWH3Dpn5g+Frb1wIXDOfdy7z1+xKjStv1tzc0vLC4tp1bSq2vrG5uZre2qCmOJSQWHLJR1HynCqCAVTTUj9UgSxH1Gan73auDXHohUNBT3uhcRl6O2oAHFSBvJy+x2PAov4G0ul+959PA63/R58tj3aKHgZbJ20R4CzhJnTLJgjLKX+Wm2QhxzIjRmSKmGY0faTZDUFDPSTzdjRSKEu6hNGoYKxIlyk+EPfXhglBYMQmlKaDhU/04kiCvV477p5Eh31LQ3EP/zGrEOzt2EiijWRODRoiBmUIdwEAhsUUmwZj1DEJbU3ApxB0mEtYltYovP+yYTZzqBWVI9KjqnxZO742zpcpxOCuyBfZAHDjgDJXADyqACMHgCL+AVvFnP1rv1YX2OWues8cwOmID19Qvg8pxO
    hi = L00(yi, F(xi))
    AAACZXicbZFNa9tAEIZXcj8cp2ndJvTSQ5eagk2LkUw/cjQplB56cEucGCxXrNYre+3dldgdFQuhX5lTjjn1N/TWta1C7GRg4eGdd5jh3SgV3IDnXTtu7cHDR4/rB43DJ0dPnzWfv7gwSaYpG9JEJHoUEcMEV2wIHAQbpZoRGQl2GS2/rPuXv5k2PFHnkKdsIslM8ZhTAlYKmzIgaaqTFQ5MJsNiUVYQRLJYlSHHAVf4Z7jWBYuh3fjezkP+Hn9t/3d0OvgdnlknDReWglgTWvhl0SvxfKv+6uFA89kcOmGz5XW9TeG74FfQQlUNwuZNME1oJpkCKogxY99LYVIQDZwKVjaCzLCU0CWZsbFFRSQzk2ITS4nfWmWK40TbpwBv1NsTBZHG5DKyTklgbvZ7a/G+3jiD+HRScJVmwBTdLoozgSHB64zxlGtGQeQWCNXc3orpnNhcwP7EzpZIljYTfz+Bu3DR6/qfuh9/fGj1z6p06ugVeoPayEefUR99QwM0RBRdob+O69ScP+6Re+K+3Fpdp5o5Rjvlvv4H7iy3PA==

    X
    j
    X
    xi
    2Rj

    L(yi, F(xi)) + gicj +
    1
    2
    hic2
    j

    AAACMHicbZDLTgIxFIY7eEO8jbp000gMEBIyQ7ywMSG6cYmJXBIYSad0oKFzSdsxksm8ig/hM7jVta4MW5/CDsxCwJM0/fKf8+e0vx0wKqRhfGmZtfWNza3sdm5nd2//QD88agk/5Jg0sc983rGRIIx6pCmpZKQTcIJcm5G2Pb5N+u0nwgX1vQc5CYjloqFHHYqRVFJfrznF5/KoBK+hghIsQ6eQ3CNFPYcjHJlxVI2dwlx9rCb6wJci19fzRsWYFVwFM4U8SKvR16fKiEOXeBIzJETXNAJpRYhLihmJc71QkADhMRqSrkIPuURY0eyHMTxTygA6PlfHk3Cm/nVEyBVi4tpq0kVyJJZ7ifhfrxtKp2ZF1AtCSTw8X+SEDEofJnHBAeUESzZRgDCn6q0Qj5AKRqpQF7bYbqwyMZcTWIVWtWJeVi7uz/P1mzSdLDgBp6AITHAF6uAONEATYPAC3sA7+NBetU/tW5vORzNa6jkGC6X9/AJlyaUI
    f(x + h) = f(x) + f0(x)h +
    1
    2
    f00(x)h2 + . . .
    Taylor approximation

    View full-size slide

  148. Impurity improvement (search criterion for best split)
    AAAB93icbVDLSgNBEOyNrxhfUY9eFoPgKeyKr2PQi8cEzAOSJcxOepMhM7PLzKywhHyBVz17E69+jkf/xEmyBxMtaCiquunuChPOtPG8L6ewtr6xuVXcLu3s7u0flA+PWjpOFcUmjXmsOiHRyJnEpmGGYydRSETIsR2O72d++wmVZrF8NFmCgSBDySJGibFSQ/fLFa/qzeH+JX5OKpCj3i9/9wYxTQVKQznRuut7iQkmRBlGOU5LvVRjQuiYDLFrqSQCdTCZHzp1z6wycKNY2ZLGnau/JyZEaJ2J0HYKYkZ61ZuJ/3nd1ES3wYTJJDUo6WJRlHLXxO7sa3fAFFLDM0sIVcze6tIRUYQam83SllBMbSb+agJ/Seui6l9XrxqXldpdnk4RTuAUzsGHG6jBA9ShCRQQnuEFXp3MeXPenY9Fa8HJZ45hCc7nD4aqk5c=
    s
    AAADGXicjVJNaxRBEO0ZPxLXr40eBWkMwi6SZWbBJBchJCIePERxN4Gdsenp7Zlt0t0zdNdIlmHO/oQc/A1e9exNvHryqL/Enp09mA9ICpp+vKp6VbzupJDCQhD89vxr12/cXFm91bl95+69+921B2Obl4bxEctlbg4TarkUmo9AgOSHheFUJZIfJEd7Tf7gIzdW5Po9zAseK5ppkQpGwVFkzcPRSy6B9mwfv8AbUWooq8K6GtY4kjyFSaelokRkshfZUhGHVXVcE4EjofG7GmdENGnT/zCsq4tLZkTUeKOV7F0qSd5cQbQpamQ7zy7Xu8qSZLlmJzIim0F/ecekux4MgkXg8yBcgvWdVyeP/+6NT/ZJ9080zVmpuAYmqbWTMCggrqgBwSR3+qXlBWVHNOMTBzVV3MbV4ilr/NQxU5zmxh0NeMH+31FRZe1cJa5SUZjZs7mGvCg3KSHdjiuhixK4Zu2gtJQYctz8CzwVhjOQcwcoM8LtitmMOl/B/Z5TUxJVO0/Csw6cB+PhINwcPH/rzNlFbayiR+gJ6qEQbaEd9BrtoxFi3ifvi/fV++Z/9r/7P/yfbanvLXseolPh//oHxxUA8w==
    (s) =
    1
    2
    " P
    xi
    2R
    gi
    2
    P
    xi
    2R
    hi
    P
    xi
    2RL
    gi
    2
    P
    xi
    2RL
    hi
    +
    P
    xi
    2RR
    gi
    2
    P
    xi
    2RR
    hi
    !#
    The impurity improvement for split s is given by
    We just need to calculate AAAB/nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PQi8cI5gHJEmYns8mQmdllZlYMy4Lf4FXP3sSrv+LRP3GS7MEkFjQUVd10dwUxZ9q47rdTWFldW98obpa2tnd298r7B00dJYrQBol4pNoB1pQzSRuGGU7bsaJYBJy2gtHtxG89UqVZJB/MOKa+wAPJQkawsVK7G4j0KeuxXrniVt0p0DLxclKBHPVe+afbj0giqDSEY607nhsbP8XKMMJpVuommsaYjPCAdiyVWFDtp9N7M3RilT4KI2VLGjRV/06kWGg9FoHtFNgM9aI3Ef/zOokJr/2UyTgxVJLZojDhyERo8jzqM0WJ4WNLMFHM3orIECtMjI1obksgMpuJt5jAMmmeVb3L6sX9eaV2k6dThCM4hlPw4ApqcAd1aAABDi/wCm/Os/PufDifs9aCk88cwhycr181QZbN
    xi
    AAACEXicbVDLSsNAFJ3UV62vqCtxM1jEFqQk4msjFAVx4aKCfUBbwmQ6bYfOJGFmIoYQ/Ai/wa2u3Ylbv8Clf+K0zcK2HrhwOOde7r3HDRiVyrK+jczc/MLiUnY5t7K6tr5hbm7VpB8KTKrYZ75ouEgSRj1SVVQx0ggEQdxlpO4OroZ+/YEISX3vXkUBaXPU82iXYqS05Jg7PYfCC3h7UIgcenhdaLk8fkwcWiw6Zt4qWSPAWWKnJA9SVBzzp9XxcciJpzBDUjZtK1DtGAlFMSNJrhVKEiA8QD3S1NRDnMh2PHohgfta6cCuL3R5Co7UvxMx4lJG3NWdHKm+nPaG4n9eM1Td83ZMvSBUxMPjRd2QQeXDYR6wQwXBikWaICyovhXiPhIIK53axBaXJzoTezqBWVI7KtmnpZO743z5Mk0nC3bBHigAG5yBMrgBFVAFGDyBF/AK3oxn4934MD7HrRkjndkGEzC+fgF36Zwc
    gi = L0(yi, F(xi))
    AAACEnicbVDLSgMxFM34rPVVdaebYJG2IGVGfG2EoiAuXFSwD2iHIZNm2tAkMyQZsQwFP8JvcKtrd+LWH3Dpn5g+Frb1wIXDOfdy7z1+xKjStv1tzc0vLC4tp1bSq2vrG5uZre2qCmOJSQWHLJR1HynCqCAVTTUj9UgSxH1Gan73auDXHohUNBT3uhcRl6O2oAHFSBvJy+x2PAov4G0ul+959PA63/R58tj3aKHgZbJ20R4CzhJnTLJgjLKX+Wm2QhxzIjRmSKmGY0faTZDUFDPSTzdjRSKEu6hNGoYKxIlyk+EPfXhglBYMQmlKaDhU/04kiCvV477p5Eh31LQ3EP/zGrEOzt2EiijWRODRoiBmUIdwEAhsUUmwZj1DEJbU3ApxB0mEtYltYovP+yYTZzqBWVI9KjqnxZO742zpcpxOCuyBfZAHDjgDJXADyqACMHgCL+AVvFnP1rv1YX2OWues8cwOmID19Qvg8pxO
    hi = L00(yi, F(xi))
    for each sample
    Calculation of this score is straightforward.
    Regression tree learning explicitly defined as minimizing the loss function:
    AAACJHicbZDLSgMxFIYz9VbrrerSTbAILZYyI942QlEQFy4q9AZtHTJp2oYmmSHJiGWYl/AhfAa3unYnLtwIvonpBbTVHwIf/zmHc/J7AaNK2/aHlZibX1hcSi6nVlbX1jfSm1tV5YcSkwr2mS/rHlKEUUEqmmpG6oEkiHuM1Lz+xbBeuyNSUV+U9SAgLY66gnYoRtpYbjrfVCF3I3rmxLcCXmcHLs3Dy2zT49F97NLcfvmHc246YxfskeBfcCaQAROV3PRXs+3jkBOhMUNKNRw70K0ISU0xI3GqGSoSINxHXdIwKBAnqhWNfhXDPeO0YceX5gkNR+7viQhxpQbcM50c6Z6arQ3N/2qNUHdOWxEVQaiJwONFnZBB7cNhRLBNJcGaDQwgLKm5FeIekghrE+TUFo/HJhNnNoG/UD0oOMeFo5vDTPF8kk4S7IBdkAUOOAFFcAVKoAIweABP4Bm8WI/Wq/VmvY9bE9ZkZhtMyfr8Bqm5pCU=
    n
    X
    i=1
    L(yi, F(xi) + T(xi)) AAACMnicbZDLSgMxFIYzXmu9VV26CRahRSkz4g1BKAriwkUVe4FOGTJpqmmTzJBkxDLMu/gQPoNb3equuPUhTNsRvB0IfPz/OZyT3w8ZVdq2X62JyanpmdnMXHZ+YXFpObeyWlNBJDGp4oAFsuEjRRgVpKqpZqQRSoK4z0jd750O/fodkYoG4lr3Q9Li6EbQDsVIG8nLHR1DV0Xci7tJCq7P4/vEo9ClAl55Rr8o9D26Dc8KX1ZxC3vdopfL2yV7VPAvOCnkQVoVLzdw2wGOOBEaM6RU07FD3YqR1BQzkmTdSJEQ4R66IU2DAnGiWvHojwncNEobdgJpntBwpH6fiBFXqs9908mRvlW/vaH4n9eMdOewFVMRRpoIPF7UiRjUARwGBttUEqxZ3wDCkppbIb5FEmFtYv2xxeeJycT5ncBfqO2UnP3S3uVuvnySppMB62ADFIADDkAZnIMKqAIMHsATeAYv1qP1Zg2s93HrhJXOrIEfZX18AkBPqhI=
    =
    X
    j
    X
    xi
    2Rj
    L(yi, F(xi) + cj)
    Impurity before split Impurity after split

    View full-size slide

  149. Method 6: XGBoost
    XGBoost (Chen & Guestrin, KDD )
    Very fast due to huge engineering e orts (parallelization, sparsity, cache, )
    Flexible and largely customizable with many detailed options
    The splitting criterion for the base regression-tree learning directly comes from ( )
    Combined use of Randomize type: Bagging-like operation (row- and column- sampling)
    can be used together
    Regression tree learning explicitly defined as minimizing the loss function with regularization
    Regularized gradient boosted trees
    . . . . . ( )
    AAADDXicbZJNb9MwGMed8DbGyzrgxsWiQuq0qXLSNUtuE0iIA4eB6DapKZHjuK07O4liB1G5+Qx8Bq5w5oa47jPsyDfBTcrL1j2S5b+ex8/Pfz9ynHMmFUIXln3j5q3bdzbubt67/+DhVmv70bHMyoLQAcl4VpzGWFLOUjpQTHF6mhcUi5jTk/js5bJ+8pEWkmXpezXP6UjgScrGjGBlUtG29SSUpYj0rIKNCGOhP1URgyFL4bvI5N905hHbg686f0o7uySa7cBdqMPawLCYxCONusjrByjYQ90+cvxaIOT5bs+gJ1gIbHaSZAouQm0AYbW4BmHC8+pOg3CMCALfdQPTys2jkr+McFxgop1KuyvjM2iYH9x1ZOB7aP+gRvq9Xq+xF9SuMM+n/4gNZWEwi6jVbrwgBNeFsxJtsIqjqPUrTDJSCpoqwrGUQwflaqRxoRjhtNoMS0lzTM7whA6NTLGgcqRrnxV8bjIJHGeFWamCdfb/Do2FlHMRm5MCq6m8Wlsmr6sNSzX2R5qlealoSpqLxiWHKoPLvwATVlCi+NwITApmvEIyxWawyvyYS7fEojIzca5OYF0cu13H6/bf7rcPX6ymswGegmegAxxwAA7Ba3AEBoBY2vpifbW+2Z/t7/YP+2dz1LZWPY/BpbDPfwO3o+vs
    X
    j
    X
    xi
    2Rj
    L(yi, F(xi) + cj) + · |{cj
    }| + ·
    1
    2
    X
    j
    c2
    j
    + ↵ ·
    X
    j
    |cj
    |
    AAADO3iclVJNb9NAEF2brxI+msKRy4oK1AoarRvaphKgqnDgWCTSVoqtaL0ZJ6uu19buGEgs/6D+CH4C4kolLogb4to76yQH2nLJSCs9zcx7b3Y0ca6kRcbOPP/a9Rs3by3dbty5e+/+cnPlwaHNCiOgKzKVmeOYW1BSQxclKjjODfA0VnAUn7yp60cfwViZ6Q84ziFK+VDLRAqOLtVf8faTfhlOdXpmGEcla+12ttmLneesxVin3W47sMWC3c12RUOu8hGv1ibr9BUNYxhKXQpnbqvG5NmiKvQpndCXdGNhXhg2JouzarfXdGFa7cYcOUT4jGWGIzCfpIWqEYIezD/fb67WCnXQqyCYg9W99fOvG29/nB70mz/DQSaKFDQKxa3tBSzHqOQGpVC1dmEh5+KED6HnoOYp2KicTl7RJy4zoElm3NNIp9l/GSVPrR2nsetMOY7s5Vqd/F+tV2DSiUqp8wJBi5lRUiiKGa0vhw6kAYFq7AAXRrpZqRhxwwW6+7rgEqf1ToLLG7gKDjdbwXZr671bzj6ZxRJ5RB6TNRKQHbJH3pED0iXCO/W+ed+9M/+L/8v/7f+ZtfrenPOQXAj//C8p+Pmw
    f↵(z) =
    8
    >
    <
    >
    :
    z + ↵ z < ↵
    z ↵ z > ↵
    0 otherwise
    AAAEfnictZNNb9MwGMe9NsAob904DiGLCdRqtDgp7ZID0sQQ4sBhINpNakrkuE5qzXmR7SCqKEc+wz4bRzjzIXCSHlhXTtMsRf7refw8/18ey37KmVQI/dxqNI1bt+9s323du//g4aP2zu5EJpkgdEwSnogzH0vKWUzHiilOz1JBceRzeuqfH5f5029USJbEX9QypbMIhzELGMFKh7ydxg/3HeUKd2QXvoE9NxCY5GaRWwV0OQ3UtFWHAi93K7epCP1ZjvqOPUKvD1+iPkL2YDDQYohMxxroOszTBda7z0LecWUW6Vo/yr8XHoMui+HnAoYeK9Oi+9Uq8rXOSK/RqOps2sjUwnFsy3JKIv1jc1wcbG668FgBezV35ya4vY/XJYcb0cu+JXzr4Eaorz3v/1CvRt5yBQsXqrvaZ/oK1h1GQwc5NbRdCe1oV/QhjiJt4LX3axCE4FVhrsT+0fuLp7+PJxcnXvuXO09IFtFYEY6lnJooVbMcC8UIp5opkzTF5ByHdKpljCMqZ3lFVcDnOjKHQSL0FytYRf+tyHEk5TLy9ckIq4Vcz5XBTblppgJ7lrM4zRSNSW0UZByqBJaPD86ZoETxpRaYCKZZIVlgfeNKP9FLLn5U6JmY6xO4KiZW3xz1h5/0cN6Cem2DPfAMdIAJDsER+ABOwBiQxp/mbnOv+cQAxgujZ7yqjza2VjWPwaVl2H8Bu3Njsw==
    (s) =
    1
    2
    "
    f↵
    P
    xi
    2R
    gi
    2
    +
    P
    xi
    2R
    hi
    f↵
    P
    xi
    2RL
    gi
    2
    +
    P
    xi
    2RL
    hi
    +
    f↵
    P
    xi
    2RR
    gi
    2
    +
    P
    xi
    2RR
    hi
    !#

    View full-size slide

  150. Method 7: LightGBM
    Best-first expansions of regression tree recursion
    The successor to XGBoost that improves the e ciency and scalability when the feature dimension is
    high and data size is large.
    Best- rst expansion + Two novel techniques
    Gradient-based One-Side Sampling (GOSS): Reduce # of samples
    Exclude a signi cant proportion of data instances with small gradients
    Exclusive Feature Bundling (EFB): Reduce # of features
    Bundle mutually exclusive features (i.e., they rarely take nonzero values simultaneously)
    LightGBM (Ke+ NIPS )
    Leaf-wise growth
    Level-wise growth
    c.f. depth-first, width-first
    Now this is widely used in many
    implementations e.g. scikit-learn
    Use max_leaf_nodes instead of
    max_depth for leaf-wise growth

    View full-size slide

  151. XGBoost and LightGBM developed as open source implementations
    Also come with Scikit-learn AIPs
    Widely used methods have good implementations
    https://github.com/dmlc/xgboost https://github.com/Microsoft/LightGBM
    First Release 24 Apr 2017 (guolinke)
    First Release 27 Mar 2014 (tqchen)
    DMLC (Distributed Machine Learning Community)
    http://dmlc.ml
    DMTK (Distributed Machine Learning Toolkit)
    http://www.dmtk.io
    !pip install xgboost
    from xgboost import XGBClassifier
    from xgboost import XGBRegressor
    !pip install lightgbm
    from lightgbm import LGBMClassifier
    from lightgbm import LGBMRegressor

    View full-size slide

  152. Code example
    Text text
    !pip install xgboost lightgbm
    from sklearn.ensemble import GradientBoostingClassifier
    from sklearn.ensemble import HistGradientBoostingClassifier
    from lightgbm import LGBMClassifier
    from xgboost import XGBClassifier
    from sklearn.datasets import load_breast_cancer
    from sklearn.model_selection import cross_validate
    def cv_5fold(clf, X, y, metrics=['accuracy', 'roc_auc']):
    scores = cross_validate(clf, X, y, cv=5,
    scoring=metrics, return_estimator=True)
    print(clf.__class__.__name__, {m: '%0.5f' % scores[f'test_{m}'].mean() for m in metrics},
    'fit_time', '%0.5f' % scores['fit_time'].mean())
    return scores
    X, y = load_breast_cancer(return_X_y=True)
    result = cv_5fold(GradientBoostingClassifier(), X, y)
    result = cv_5fold(HistGradientBoostingClassifier(), X, y)
    result = cv_5fold(XGBClassifier(), X, y)
    result = cv_5fold(LGBMClassifier(), X, y)
    result = cv_5fold(LGBMClassifier(n_estimators=200, learning_rate=0.1,
    min_child_samples=5, reg_lambda=1.0), X, y)
    GradientBoostingClassifier {'accuracy': '0.95962', 'roc_auc': '0.99104'} fit_time 0.43648
    HistGradientBoostingClassifier {'accuracy': '0.96660', 'roc_auc': '0.99268'} fit_time 0.35848
    XGBClassifier {'accuracy': '0.96487', 'roc_auc': '0.99349'} fit_time 0.09564
    LGBMClassifier {'accuracy': '0.96662', 'roc_auc': '0.99243'} fit_time 0.14272
    LGBMClassifier {'accuracy': '0.97014', 'roc_auc': '0.99304'} fit_time 0.18123

    View full-size slide

  153. Code example (with hyperparameter tuning)
    Text text
    from scipy.stats import uniform, randint
    from sklearn.utils.fixes import loguniform
    from sklearn.model_selection import RandomizedSearchCV
    model = RandomizedSearchCV(LGBMClassifier(),
    {"n_estimators": [100, 200, 500],
    "min_child_samples": [1, 5, 10, 50],
    "num_leaves": randint(3, 31),
    "reg_alpha": loguniform(1e-2, 1),
    "reg_lambda": loguniform(1e-2, 1),
    "learning_rate": loguniform(1e-3, 1),
    "subsample": uniform(0, 1),
    "colsample_bytree": uniform(0, 1)},
    n_iter=20)
    result = cv_5fold(model, X, y)
    for e in result['estimator']:
    print(e.best_params_)
    RandomizedSearchCV {'accuracy': '0.97190', 'roc_auc': '0.99408'} fit_time 7.78688
    {'colsample_bytree': 0.8157047506148358, 'learning_rate': 0.10355525231624815, 'min_child_sa
    {'colsample_bytree': 0.9631901921844805, 'learning_rate': 0.4912284442404967, 'min_child_sam
    {'colsample_bytree': 0.37151158305148924, 'learning_rate': 0.2702583980583705, 'min_child_sa
    {'colsample_bytree': 0.3435018330608688, 'learning_rate': 0.05038610774240793, 'min_child_sa
    {'colsample_bytree': 0.12849846893758976, 'learning_rate': 0.165067814006174, 'min_child_sam
    We should increase the number of iterations
    as much as possible for this large search grid
    (20 trials are too small)

    View full-size slide

  154. Takeaways for Part 2: “Randomize” vs “Optimize”
    ‣ RandomForestRegressor
    max_depth=None, max_leaf_nodes=None
    ‣ ExtraTreesRegressor
    max_depth=None, max_leaf_nodes=None
    ‣ GradientBoostingRegressor
    max_depth=3, max_leaf_nodes=None
    ‣ HistGradientBoostingRegressor
    max_depth=None, max_leaf_nodes=31
    ‣ XGBRegressor
    max_depth=3
    ‣ LGBMRegressor
    num_leaves=31, max_depth=-1
    • Use a large base tree (e.g. Fully-grown tree)
    • Each decision tree is trained to overfit.
    • Use a small base tree (Size-bounded tree)
    • Each decision tree is trained to underfit.
    “Weak" learner
    Scikit default values (Examples of Regressor)
    • Random Projection Ensemble (2017)
    • Random Composite Forest (2016)
    • Random Rotation Ensemble (2016)
    • Oblique Random Forests (2011)
    • Rotation Forests (2006)
    Other
    examples
    • Globally Induced Forest (ICML2017)
    • Regularized Greedy Forest (TPAMI 2014)
    • CatBoost (NeurIPS2018)
    • “Randomize” (Bagging, Random Forest)
    • “Optimize” (Boosting)
    Other
    examples

    View full-size slide

  155. Remark 1: Integrating “Randomize” and “Optimize” strategies
    Modern implementations of the
    “Optimize” type can also use
    the “Randomize” type
    (Stochastic Gradient Boosting)
    • Friedman’s Stochastic Gradient Boosting (Comput Stat Data Anal, 2002)
    • Breiman’s Iterated Bagging (Mach Learn, 2001)
    • Ensemble on Random Patches (Louppe & Geurts, ECML PKDD 2012)
    The basic idea is to introduce 'Randomize' into the 'Optimize' process.

    View full-size slide

  156. Remark 2: Control the complexity of the tree ensembles
    It is more common to control the complexity of base trees directly with structural constraints
    (i.e. max_depth, max_leaf_nodes, min_samples_leaf, etc)
    • Pre-pruning (Stopping)
    Learn trees/forests in a way that keeps the ensemble compact.
    RGF(Johnson & Zhang, 2014)ɺFeature-Budgeted Random Forest (Nan+ ICML2015)ɺGIF(Begon+ ICML2017)ɺBonsai(Kumar+
    ICML2017)ɺADAPT-GBRT(Nan+ NIPS2017), etc.
    • Post-pruning (Pruning)
    After initially creating a set of trees, prune them to learn the appropriate trees/forests.
    ISLE (Friedman & Popescu 2003)ɺRuleFit (Friedman & Popescu 2008)ɺForest garrote (Meinshausen+ 2009)ɺL1-based
    compression (Joly+ 2012)ɺGlobal refinement (Ren+ CVPR 2015)ɺDART (Rashmi & Gilad-Bachrach, AISTATS 2015)ɺWavelet
    decomposition (Elisha & Dekel JMLR 2015)ɺBudgetPrune (Nan+ NIPS2016), etc.
    • Re-learning (Distillation)
    After initially creating a set of trees, relearn an equivalent but more compact trees/forests.
    CMM(Domingos, ICML1997)ɺNode Harvest (Meinshausen, 2010), etc.
    But there are also advanced methods for controlling the complexity of the entire forest

    View full-size slide

  157. Outline
    We will jointly learn decision trees and ensemble learning!
    1. Why we should learn decision trees and ensemble learning?
    2. History of Decision Trees
    3. Basics of Decision Trees
    4. Ensemble Learning: Additive ensembles and decision forests
    5. Random forest and Regularized gradient boosted trees
    6. Practical data analysis with decision tree ensembles
    Part 1
    Part 2

    View full-size slide

  158. Practical data analysis using tree ensembles
    In many cases, prediction alone is not sufficient; analysis of the machine learning
    model is necessary.
    Visualization: Plotting the tree structure and accompanying information.
    Model Tuning: Adjusting hyperparameters for decision trees and decision forests.
    Model Interpretation:
    Variable Importance: Which input variables have a strong in uence on model predictions?
    Partial Dependence Plot (PDP): How does the prediction change when variable values are altered?
    SHapley Additive exPlanations (SHAP): Why was a particular prediction made?
    Uncertainty Quanti cation (UQ):
    Randomize type Law of total variance
    Optimize type Quantile Regression

    View full-size slide

  159. dtreeviz: A visualization tool compatible with scikit-learn, XGBoost, and LightGBM.
    Visualization of Decision Trees
    https://github.com/parrt/dtreeviz
    While scikit-learn has built-in decision tree plotting capabilities using graphviz,
    dtreeviz o ers overwhelmingly better and more exible visualizations.
    Check out the Jupyter Notebook examples on GitHub for how to use dtreeviz.
    Parr & Grover, How to visualize decision trees
    https://explained.ai/decision-tree-viz/index.html
    https://github.com/parrt/dtreeviz/tree/master/notebooks
    Must-read: Detailed explanation on decision tree visualization by the authors of dtreeviz

    View full-size slide

  160. Gallery (dtreevis)
    https://github.com/parrt/dtreeviz/blob/master/notebooks/examples.ipynb

    View full-size slide

  161. Determine the number of trees (n_estimators) to create, say 100, 200, and 500 or so, and
    tune the other main hyperparameters through cross-validation.
    Model Tuning (hyperparameter tuning)
    • Decision Tree: max_leaf_nodes (or max_depth) + min_samples_leaf + Pruning by ccp_alpha
    • Decision Forest (Randomize-type): bootstrap + max_features
    • Decision Forest (Optimize-type): learning_rate (Adjust according to n_estimators)
    + with the Randomize type (subsample, colsample)
    + with regularization (1-norm regularization reg_alpha, 2-norm regularization reg_lambda)
    1. Optuna
    2. SMAC
    3. Hyperopt
    4. Scikit-Hyperband
    https://optuna.org/
    https://github.com/automl/SMAC3
    https://github.com/hyperopt/hyperopt
    https://github.com/thuijskens/scikit-hyperband
    Use Scikit's RandomSearchCV or GridSearchCV, as well as the following tuners for more
    systematic exploration. (Depending on the computation time and resources.)

    View full-size slide

  162. Code example
    import optuna
    from lightgbm import LGBMRegressor
    from sklearn.metrics import mean_squared_error
    from sklearn.datasets import load_diabetes
    from sklearn.model_selection import train_test_split
    X, y = load_diabetes(return_X_y=True)
    X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.33)
    def objective(trial):
    param = {
    "num_leaves": trial.suggest_int("num_leaves", 2, 256),
    "learning_rate": trial.suggest_float("learning_rate", 1e-5, 1.0, log=True),
    "min_child_samples": trial.suggest_int("min_child_samples", 1, 100),
    "subsample": trial.suggest_float("subsample", 0.4, 1.0),
    "subsample_freq": trial.suggest_int("subsample_freq", 1, 7),
    "colsample_bytree": trial.suggest_float("colsample_bytree", 0.4, 1.0),
    "reg_alpha": trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True),
    "reg_lambda": trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True),
    }
    model = LGBMRegressor(n_estimators=200, verbose=-1, **param)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_val)
    return mean_squared_error(y_val, y_pred)
    optuna.logging.set_verbosity(optuna.logging.ERROR)
    study = optuna.create_study(direction="minimize")
    study.optimize(objective, n_trials=100)
    trial = study.best_trial
    best_params = trial.params
    model = LGBMRegressor(n_estimators=200,
    verbose=-1, **best_params)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_val)
    mean_squared_error(y_val, y_pred)
    model = LGBMRegressor(n_estimators=200,
    verbose=-1)
    model.fit(X_train, y_train)
    y_pred = model.predict(X_val)
    mean_squared_error(y_val, y_pred)
    Compare

    View full-size slide

  163. Model interpretation
    • The Mythos of Model Interpretability:
    The concept of "interpretability" is ambiguous and subjective, potentially dubious...?
    Three basic techniques for extracting information from decision tree ensembles.
    Variable importance
    Partial Dependence Plot (PDP)
    SHapley Additive exPlanations (SHAP)
    https://doi.org/10.1145/3236386.3241340
    • Rashomon effect: The multiplicity of good ML models
    We can always have multiple different ML models having the same good prediction accuracy.
    • While decision trees are often said to have "high interpretability", they have issues with instability.
    The tree structures can change significantly due to random noise, making it largely unreliable.
    But obviously, decision tree ensembles would be not interpretable…

    View full-size slide

  164. Variable Importance (Feature importance) is a convenient and user-friendly tool to
    understand which of the input variables used are important for predictions.
    Variable importances: MDI vs PFI
    • Feature Importance by Mean Decrease Impurity (MDI)
    Scoring how much each variable contributed to the prediction
    based on the internal information obtained during training
    from decision trees.
    • Permutation Feature Importance (PFI)
    Scoring a variable's importance by measuring the
    deterioration in accuracy when predicting with the variable
    versus predicting without it on any given data.
    In practice, it's preferable to use PFI over MDI.
    However, be aware that there are also drawbacks to consider. (masking by correlated features)
    Parr et al., Beware Default Random Forest Importances.
    https://explained.ai/rf-importance/

    View full-size slide

  165. Nodes where variable is used for splitting
    Feature Importance 1: Mean Decrease Impurity (MDI)
    The weighted average of impurity improvement at nodes where variable is used.
    Since it's a relative scale normalized to a sum of 1 or 100, in actual calculations,
    you can simply add up the values of .
    AAACpnichVE9S8NQFD2NX7V+tOoiuBRLpVN5FUFxKro4ST/sB9RSk/haQ9MkJGmhFv+AroqDk4KD+DNc+gcc+hPEsYKLgzdpQLS03pC88869577zciVDVSybsZ5PmJicmp7xzwbm5hcWg6Gl5bylN02Z52Rd1c2iJFpcVTSesxVb5UXD5GJDUnlBqu87+UKLm5aia0d22+DlhljTlKoiizZRaa0SirA4cyM8DBIeiMCLlB7q4hin0CGjiQY4NNiEVYiw6CkhAQaDuDI6xJmEFDfPcYEAaZtUxalCJLZO3xrtSh6r0d7pablqmU5R6TVJGUaUvbIn1mdd9sze2NfIXh23h+OlTas00HKjErxczX7+q2rQauPsRzXWs40qdlyvCnk3XMa5hTzQt85v+9ndTLSzwR7YO/m/Zz32QjfQWh/yY5pn7sb4kcjL6D/m5L0KGmHi78CGQX4zniCc3ook97xh+rGGdcRoYttI4gAp5OgEjitc40aICYdCTigMSgWfp1nBrxBOvgHQN5dI
    AAACqXichVHLSsNAFD2Nr/ps1Y3gRiyKIshUBMVV0Y3LVq0tVpFknNqhaRKSaaEWf8CF2wquFFyIn+GmP+CinyAuFdy48CYNiIr1hmTOnHvPnTO5hmNKTzHWjmg9vX39A9HBoeGR0bFYfHxi37OrLhdZbpu2mzd0T5jSElkllSnyjiv0imGKnFHe8vO5mnA9aVt7qu6Io4p+asmi5LryKWtBLR7HE2yZBTHzGyRDkEAYaTvewiFOYIOjigoELCjCJnR49BSQBIND3BEaxLmEZJAXOMcQaatUJahCJ7ZM31PaFULWor3f0wvUnE4x6XVJOYM59sTu2StrsQf2zD7+7NUIevhe6rQaHa1wjmMXU7vv/6oqtCqUvlRdPSsUsR54leTdCRj/Fryjr501X3c3duYa8+yWvZD/G9Zmj3QDq/bG7zJi57qLH4O8/P3H/HxYQSNM/hzYb7C/spwknFlNpDbDYUYxjVks0MTWkMI20sjSCSVcookrbUnLaHntoFOqRULNJL6Fxj8BACiYKw==
    AAACpnichVE9S8NQFD2NX7V+tOoiuBRLpVN5FUFxKro4ST/sB9RSk/haQ9MkJGmhFv+AroqDk4KD+DNc+gcc+hPEsYKLgzdpQLS03pC88869577zciVDVSybsZ5PmJicmp7xzwbm5hcWg6Gl5bylN02Z52Rd1c2iJFpcVTSesxVb5UXD5GJDUnlBqu87+UKLm5aia0d22+DlhljTlKoiizZR6VYlFGFx5kZ4GCQ8EIEXKT3UxTFOoUNGEw1waLAJqxBh0VNCAgwGcWV0iDMJKW6e4wIB0japilOFSGydvjXalTxWo73T03LVMp2i0muSMowoe2VPrM+67Jm9sa+RvTpuD8dLm1ZpoOVGJXi5mv38V9Wg1cbZj2qsZxtV7LheFfJuuIxzC3mgb53f9rO7mWhngz2wd/J/z3rshW6gtT7kxzTP3I3xI5GX0X/MyXsVNMLE34ENg/xmPEE4vRVJ7nnD9GMN64jRxLaRxAFSyNEJHFe4xo0QEw6FnFAYlAo+T7OCXyGcfAPiR5dQ
    AAACpnichVE9S8NQFD3G7++qi+AiloqT3IqgOIkuTmLV2EItNYmvNTRfJK+FWvwDuioOTgoO4s9w6R9w8CeIo4KLgzdpQLSoNyTvvHPvue+8XN2zzEASPXUonV3dPb19/QODQ8Mjo4mx8b3ArfqGUA3Xcv2crgXCMh2hSlNaIuf5QrN1S2T1ynqYz9aEH5iusyvrnijYWtkxS6ahSaYysphI0jxFMd0O0jFIIo4tN9HEPg7hwkAVNgQcSMYWNAT85JEGwWOugAZzPiMzygucYIC1Va4SXKExW+FvmXf5mHV4H/YMIrXBp1j8+qycRooe6Y5eqUn39Ewfv/ZqRD1CL3Ve9ZZWeMXR08md939VNq8SR1+qPz1LlLAceTXZuxcx4S2Mlr52fPm6s7KdaszSDb2w/2t6oge+gVN7M24zYvvqDz86e/n9j4X5uIJHmP45sHawtzCfZpxZTK6uxcPswxRmMMcTW8IqNrAFlU8QOMM5LpQ5ZVNRlWyrVOmINRP4FsrBJ93Dl04=
    AAACpnichVE9S8NQFD3G7++qi+AiloqT3IqgOIkuTmLV2EItNYmvNTRfJK+FWvwDuioOTgoO4s9w6R9w8CeIo4KLgzdpQLSoNyTvvHPvue+8XN2zzEASPXUonV3dPb19/QODQ8Mjo4mx8b3ArfqGUA3Xcv2crgXCMh2hSlNaIuf5QrN1S2T1ynqYz9aEH5iusyvrnijYWtkxS6ahSaYysphI0jxFMd0O0jFIIo4tN9HEPg7hwkAVNgQcSMYWNAT85JEGwWOugAZzPiMzygucYIC1Va4SXKExW+FvmXf5mHV4H/YMIrXBp1j8+qycRooe6Y5eqUn39Ewfv/ZqRD1CL3Ve9ZZWeMXR08md939VNq8SR1+qPz1LlLAceTXZuxcx4S2Mlr52fPm6s7KdaszSDb2w/2t6oge+gVN7M24zYvvqDz86e/n9j4X5uIJHmP45sHawtzCfZpxZTK6uxcPswxRmMMcTW8IqNrAFlU8QOMM5LpQ5ZVNRlWyrVOmINRP4FsrBJ93Dl04=
    Impurity improvement at node
    # of all samples
    # of samples at node
    petal.width <= 0.8
    gini = 0.67
    samples = 150 petal.width <= 1.75
    gini = 0.5
    samples = 100
    petal.length <= 4.95
    gini = 0.17
    samples = 54
    petal.length <= 4.85
    gini = 0.04
    samples = 46
    gini = 0.0
    samples = 50
    gini = 0.04
    samples = 48
    gini = 0.44
    samples = 8
    gini = 0.44
    samples = 3
    gini = 0.0
    samples = 43
    150×0.67–
    (50×0.0+100×0.5) 100×0.5-
    (54×0.17+46×0.04)

    View full-size slide

  166. Feature Importance 2: Permutation Feature Importance (PFI)
    The decrease in prediction accuracy when the values of a specific variable are
    shuffled within the data.
    For importance features in predictions,
    the prediction accuracy should decrease
    when the feature values are randomly replaced
    with arbitrary values!
    →Features that cause a significant decline in
    prediction accuracy are considered important
    features.
    a trained model
    a trained model


    Accuracy A Accuracy B
    Feature importance of X3
    = Accuracy A - Accuracy B

    View full-size slide

  167. • PFI doesn't require retraining and can be calculated on either training or test data. However,
    careful interpretation is needed depending on which data the PFI was calculated on.
    • PFI involves averaging results over multiple shuffles due to the randomness of the shuffling,
    providing a more robust estimate.
    Advantages and Disadvantages of PFI
    Advantages
    • PFI assumes that the model has a high predictive accuracy. A poor model might undervalue
    an important feature that would have higher importance in a better model.
    • Masking Phenomenon: If there are multiple variables with high correlation, shuffling one
    variable might have its effects compensated for by the others. Consequently, using PFI can
    make the importance of all these variables appear lower.
    • A common way to handle this is to preprocess the data and represent highly correlated
    features with a single representative feature.
    Disadvantages

    View full-size slide

  168. Example
    https://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance.html

    View full-size slide

  169. Partial Dependence Plot (PDP)
    https://scikit-learn.org/stable/auto_examples/ensemble/plot_partial_dependence.html
    Plot of empirical mean of change in output values when only the value of each variable is varied
    Example)
    Varying x2, while marginalizing the
    other variables (x1 and x3) at their
    empirical averages from the sample
    We can not only analyze the relative importance of each variable, but also see how they
    affect the output.
    AAACDnicbVDLSsNAFJ3UV62vqODGzdAiVISSFHwsi7pwWcE+oClhMp22Q2eSMDMRY8w/uPAL3Oranbj1F7r0T5w+Frb1wIXDOfdyD8cLGZXKsoZGZml5ZXUtu57b2Nza3jF39+oyiAQmNRywQDQ9JAmjPqkpqhhphoIg7jHS8AZXI79xT4SkgX+n4pC0Oer5tEsxUlpyzQOHI9UXPKleV1M3eXDLafHxGLpmwSpZY8BFYk9JoZJ3Tl6Glbjqmj9OJ8ARJ77CDEnZsq1QtRMkFMWMpDknkiREeIB6pKWpjziR7WScP4VHWunAbiD0+AqO1b8XCeJSxtzTm6O0ct4bif95rUh1L9oJ9cNIER9PHnUjBlUAR2XADhUEKxZrgrCgOivEfSQQVrqymS8eT3Un9nwDi6ReLtlnpdNbXc4lmCALDkEeFIENzkEF3IAqqAEMnsAreAPvxrPxYXwaX5PVjDG92QczML5/AaVHnz4=
    PDPx2
    (z)
    AAACMXicbZDLSgMxFIYzXuu96tJNUIQKUmYUb4tC0YUuK1grdOqQSTNtaJIZkoxYxzyLe7c+g1tdd6dufQnTVsTbD4E//zmHk3xhwqjSrttzRkbHxicmc1PTM7Nz8wv5xaVzFacSkyqOWSwvQqQIo4JUNdWMXCSSIB4yUgs7R/167YpIRWNxprsJaXDUEjSiGGkbBfmDEvQjiXDmmUwY6KuUBxkteeZSwKjghzy7NjbwzCa82YRf922zEeTX3KI7EPxrvE+zVj6G935w26oE+Ve/GeOUE6ExQ0rVPTfRjQxJTTEjZtpPFUkQ7qAWqVsrECeqkQ2+aOC6TZowiqU9QsNB+n0iQ1ypLg9tJ0e6rX7X+uF/tXqqo/1GRkWSaiLwcFGUMqhj2OcFm1QSrFnXGoQltW+FuI0sMW2p/tgScmOZeL8J/DXnW0Vvt7hzauEcgqFyYAWsggLwwB4ogxNQAVWAwR14BE/g2Xlwes6L8zZsHXE+Z5bBDznvH2iurSI=
    =
    1
    n
    n
    X
    i=1
    f(xi1, z, xi3)
    2-variable PDP
    1-variable PDPs

    View full-size slide

  170. SHAP (SHapley Additive exPlanations)
    A model explanation method that decomposes the difference between a given
    prediction value and the dataset average into the "sum of contributions for each
    feature (SHAP values)”. (based on Cooperative game theory)
    a given prediction value
    the dataset average
    https://github.com/slundberg/shap

    View full-size slide

  171. Example
    The dataset average is an annual income of 5 million yen, yet this individual was predicted to
    have an annual income of 10 million yen.
    5 million
    0 10 million
    dataset average Master's
    degree CS
    Specialty
    Position:
    Section Chief
    Don’t speak
    English
    +2
    +2
    +3
    -2
    Negative factors are
    also taken into account.
    Where does the difference of 5 million yen come from?

    View full-size slide

  172. SHAP value
    The SHAP value of variable i for sample x = A weighted average of all possible differences
    Prediction when
    we use variable i
    Prediction when
    we don’t use variable i
    The total number of
    possible combinations
    AAAB93icbVDLSgNBEJyNrxhfUY9eBoPgKeyKr2NQEI8JmAckS5id9CZDZmaXmVlhWfIFXvXsTbz6OR79EyfJHkxiQUNR1U13VxBzpo3rfjuFtfWNza3idmlnd2//oHx41NJRoig0acQj1QmIBs4kNA0zHDqxAiICDu1gfD/128+gNIvkk0lj8AUZShYySoyVGg/9csWtujPgVeLlpIJy1Pvln94gookAaSgnWnc9NzZ+RpRhlMOk1Es0xISOyRC6lkoiQPvZ7NAJPrPKAIeRsiUNnql/JzIitE5FYDsFMSO97E3F/7xuYsJbP2MyTgxIOl8UJhybCE+/xgOmgBqeWkKoYvZWTEdEEWpsNgtbAjGxmXjLCayS1kXVu65eNS4rtbs8nSI6QafoHHnoBtXQI6qjJqII0At6RW9O6rw7H87nvLXg5DPHaAHO1y8/05Nq
    F
    AAAB93icbVDLSgNBEJyNrxhfUY9eBoPgKeyKr2PQi8cEzQOSJcxOepMhM7PLzKywLPkCr3r2Jl79HI/+iZNkDyaxoKGo6qa7K4g508Z1v53C2vrG5lZxu7Szu7d/UD48aukoURSaNOKR6gREA2cSmoYZDp1YAREBh3Ywvp/67WdQmkXyyaQx+IIMJQsZJcZKjcd+ueJW3RnwKvFyUkE56v3yT28Q0USANJQTrbueGxs/I8owymFS6iUaYkLHZAhdSyURoP1sdugEn1llgMNI2ZIGz9S/ExkRWqcisJ2CmJFe9qbif143MeGtnzEZJwYknS8KE45NhKdf4wFTQA1PLSFUMXsrpiOiCDU2m4UtgZjYTLzlBFZJ66LqXVevGpeV2l2eThGdoFN0jjx0g2roAdVRE1EE6AW9ojcndd6dD+dz3lpw8pljtADn6xdUSpN3
    S
    The set of all features
    feature subset
    AAAB/nicbVDLSsNAFL2pr1pfVZdugkWom5KIr2XRjctK7QPaECbTSTt0ZhJmJmIJBb/Bra7diVt/xaV/4rTNwrYeuHA4517uvSeIGVXacb6t3Mrq2vpGfrOwtb2zu1fcP2iqKJGYNHDEItkOkCKMCtLQVDPSjiVBPGCkFQxvJ37rkUhFI/GgRzHxOOoLGlKMtJHaoV8vP/n1U79YcirOFPYycTNSggw1v/jT7UU44URozJBSHdeJtZciqSlmZFzoJorECA9Rn3QMFYgT5aXTe8f2iVF6dhhJU0LbU/XvRIq4UiMemE6O9EAtehPxP6+T6PDaS6mIE00Eni0KE2bryJ48b/eoJFizkSEIS2putfEASYS1iWhuS8DHJhN3MYFl0jyruJeVi/vzUvUmSycPR3AMZXDhCqpwBzVoAAYGL/AKb9az9W59WJ+z1pyVzRzCHKyvX+vRlf0=
    fS(xS) The prediction for feature values in set S that are marginalized
    over features that are not included in set S
    AAACa3icbVFdb9MwFHXCV1e+CnsDHiyqSdsDVYL4ekGaQJp4HCrdJtVR5Lg3rTXbCfYNonLzQ/fID+A3gNPlgW1cyfLROffcax0XtZIOk+Qiim/dvnP33mBneP/Bw0ePR0+enriqsQJmolKVPSu4AyUNzFCigrPaAteFgtPi/HOnn/4A62RlvuG6hkzzpZGlFBwDlY8cq1cyl/QjZa7RuZ92d+EA4Ts9ChhQS9M4yrxkbUtZabnwaetZIU2l/eZo0/rNdNNJCkqcD8v9n2EKE03dew7oK9qR04Mhs3K5wiwfjZNJsi16E6Q9GJO+jvPRL7aoRKPBoFDcuXma1Jh5blEKBe2QNQ5qLs75EuYBGq7BZX4bTkv3ArOgZWXDMUi37L8Oz7Vza12ETs1x5a5rHfk/bd5g+SHz0tQNghGXi8pGUaxolzRdSAsC1ToALqwMb6VixUN+GP7jypZCtyGT9HoCN8HJ60n6bvL265vx4ac+nQF5Tl6SfZKS9+SQfCHHZEYEuSB/okG0E/2Od+Nn8YvL1jjqPbvkSsV7fwFqF7wb
    i =
    X
    S✓F \{i}
    1
    |F |
    |S|

    f(xS[{i}
    ) f(xS)
    ⇤ (in predictions with and without variable i)

    View full-size slide

  173. treeSHAP: SHAP for trees is exact & practical!
    https://www.nature.com/articles/s42256-019-0138-9
    https://arxiv.org/abs/1905.04610
    For general ML models, exact computation is NP-Hard (infeasibly time-consuming)
    but for decision trees, it can be computed efficiently in polynomial time!

    View full-size slide

  174. Visualizing SHAP Values (Local Explanation)
    Local Explanation (Decomposition of feature contributions for a specific prediction)
    Force Plot
    Waterfall Plot Local Bar Plot

    View full-size slide

  175. Visualizing SHAP Values (Global Explanation)
    Beeswarm Plot Bar Plot
    Feature importance by SHAP values
    Dependence Plot
    Global Explanation (calculated by aggregating SHAP values for each data point)
    Partial dependency plot by SHAP values

    View full-size slide

  176. Uncertainty Quantification (UQ)
    For the use for actual decision-making, not only the predicted values but also quantifying their
    reliability (uncertainty) are very important.
    A. The predicted stock price of Brand X for tomorrow is , yen (± . yen)!
    B. The predicted stock price of Brand X for tomorrow is , yen (± . yen)!
    Here we focus on how to get the prediction variances because useful quantities such as
    confidence intervals, prediction intervals and expected improvement are calculated from them
    95% prediction interval
    (by normal approximation)

    View full-size slide

  177. An Easy Solution: Use scikit-optimize (skopt)
    https://scikit-optimize.github.io/ We can easily obtain the prediction variances of tree
    ensembles by sklearn wrappers from the skopt library!

    View full-size slide

  178. • A regression tree naturally possesses predictive variance
    → The "variance" of y in the training samples where x falls.
    • Since base trees in Randomize type are independent, the predictive
    variances from each regression tree can be combined to compute the
    predictive variance for the ensemble.
    Prediction variances for regression (Randomize type)
    Law of total variance:
    Total variance = Within-group variance
    + Between-group variance
    Hutter et al, Algorithm Runtime Prediction: Methods & Evaluation (2014) https://arxiv.org/abs/1211.0906

    View full-size slide

  179. https://scikit-optimize.github.io/stable/_modules/skopt/learning/forest.html
    def _return_std(X, trees, predictions, min_variance):
    # This derives std(y | x) as described in 4.3.2 of arXiv:1211.0906
    std = np.zeros(len(X))
    for tree in trees:
    var_tree = tree.tree_.impurity[tree.apply(X)]
    # This rounding off is done in accordance with the
    # adjustment done in section 4.3.3
    # of http://arxiv.org/pdf/1211.0906v2.pdf to account
    # for cases such as leaves with 1 sample in which there
    # is zero variance.
    var_tree[var_tree < min_variance] = min_variance
    mean_tree = tree.predict(X)
    std += var_tree + mean_tree ** 2
    std /= len(trees)
    std -= predictions ** 2.0
    std[std < 0.0] = 0.0
    std = std ** 0.5
    return std
    Hutter et al, Algorithm Runtime Prediction: Methods & Evaluation (2014) https://arxiv.org/abs/1211.0906
    Prediction variances for regression (Randomize type)

    View full-size slide

  180. • Since base trees in Optimize type are NOT independent, the predictive variances are
    calculated through quantile regression.
    For a normal distribution, the quantiles corresponding to the standard deviation are
    . (lower side) and . (upper side). Therefore, by regressing on these quantiles,
    we can obtain the standard deviation (or the variance).
    GradientBoostingRegressor(loss='quantile', alpha=α)
    Quantile Regression: Regression that targets
    a specific tail (q-quantile) of the data, rather
    than the center (mean) of the data.
    Prediction variances for regression (Optimize type)

    View full-size slide

  181. • Use the Pinball loss as the loss function for optimization and perform quantile
    regression for τ=0.16, 0.5, 0.84 respectively.
    GradientBoostingRegressor
    (n_estimators=30, learning_rate=0.1)
    LGBMRegressor
    (n_estimators=30, learning_rate=0.1,
    max_leaf_nodes=8, min_child_samples=5)
    • The 0.5 quantile represents the mean, and the range from 0.16 to 0.84 can
    approximate one standard deviation. (assuming normal distribution)
    Prediction variances for regression (Optimize type)
    Use 1.96 σ
    for 95% PI

    View full-size slide

  182. For your information: Toward Conformal Prediction
    https://github.com/scikit-learn-contrib/MAPIE
    regressor = ExtraTreesRegressor(max_leaf_nodes=32, bootstrap=True)
    MapieRegressor(regressor,
    method="plus", cv=-1)
    MapieRegressor(regressor,
    method="plus",
    cv=Subsample(n_resamplin
    gs=50))
    MapieRegressor(regressor,
    method="plus", cv=-1)
    MapieRegressor(regress
    or, method="plus",
    cv=Subsample(n_resamp
    lings=50))
    Jacknife+
    95% prediction intervals
    Jacknife+ after bootstrap
    95% prediction intervals
    Jacknife+
    90% prediction intervals
    Jacknife+ after bootstrap
    90% prediction intervals
    In recent years, Uncertainty Quantification (UQ) has been actively investigated
    as a method of Conformal Prediction (CP).

    View full-size slide

  183. Summary
    We will jointly learn decision trees and ensemble learning!
    1. Why we should learn decision trees and ensemble learning?
    2. History of Decision Trees
    3. Basics of Decision Trees
    4. Ensemble Learning: Additive ensembles and decision forests
    5. Random forest and Regularized gradient boosted trees
    6. Practical data analysis with decision tree ensembles
    Part 1
    Part 2

    View full-size slide

  184. Outline
    We will jointly learn decision trees and ensemble learning!
    1. Why we should learn decision trees and ensemble learning?
    2. History of Decision Trees
    3. Basics of Decision Trees
    4. Ensemble Learning: Additive ensembles and decision forests
    5. Random forest and Regularized gradient boosted trees
    6. Practical data analysis with decision tree ensembles
    Part 1
    Part 2

    View full-size slide