Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Interpreting Machine Learning Models: Why and How!

OmaymaS
April 06, 2019

Interpreting Machine Learning Models: Why and How!

Invited talk at SatRday Johannesburg #satRdayJoburg
https://joburg2019.satrdays.org/

OmaymaS

April 06, 2019
Tweet

More Decks by OmaymaS

Other Decks in Technology

Transcript

  1. INTERPRETING MACHINE
    LEARNING MODELS:
    WHY AND HOW!
    OMAYMA SAID

    View Slide

  2. View Slide

  3. HOW WOULD YOUR MENTAL MODEL
    LABEL THIS IMAGE?
    CAT → 0
    DOG → 1
    OR

    View Slide

  4. HOW WOULD YOUR MENTAL MODEL
    LABEL THIS IMAGE?
    CAT → 0
    DOG → 1
    OR

    View Slide

  5. HOW WOULD YOUR MENTAL MODEL
    LABEL THIS IMAGE?
    DUCK → 0
    RABBIT → 1
    OR

    View Slide

  6. WHY ?
    HOW WOULD YOUR MENTAL MODEL
    LABEL THIS IMAGE?

    View Slide

  7. https:/
    /github.com/minimaxir/optillusion-animation
    Max Woolf

    View Slide

  8. INTERPRETABILITY
    TRADEOFF
    COMPLEXITY

    View Slide

  9. AI-POWERED [----]
    ML-ENABLED [----]
    MORE AND MORE

    View Slide

  10. View Slide

  11. TAMMY DOBBS, Arkansas
    CEREBRAL PALSY
    REDUCED HOURS
    OF HOME CARE VISITS PER WEEK

    View Slide

  12. TAMMY DOBBS, Arkansas
    CEREBRAL PALSY
    REDUCED HOURS
    OF HOME CARE VISITS PER WEEK
    WHY?

    View Slide



  13. [In Idaho], the state declined to
    disclose the formula it was using,
    saying that its math qualified as a
    TRADE SECRET.

    View Slide

  14. View Slide

  15. “AUTOMATED
    REDACTION, TRANSCRIPTION, REPORTING”

    View Slide

  16. WHAT ELSE
    ?
    I'm In to Connect and Serve*
    “AUTOMATED
    REDACTION, TRANSCRIPTION, REPORTING”

    View Slide

  17. View Slide

  18. Amazon’s system TAUGHT ITSELF that
    male candidates were preferable. It penalized
    resumes that included the word “women’s,” as in “women’s
    chess club captain.” And it downgraded graduates of two
    all-women’s colleges, according to people familiar with the
    matter. They did not specify the names of the schools.

    View Slide

  19. Amazon’s system TAUGHT ITSELF that
    male candidates were preferable. It penalized
    resumes that included the word “women’s,” as in “women’s
    chess club captain.” And it downgraded graduates of two
    all-women’s colleges, according to people familiar with the
    matter. They did not specify the names of the schools.
    “ LEARNED FROM HUMANS

    View Slide

  20. COLLECT/LABEL
    DATA
    IT IS HUMANS WHO
    WRITE
    ALGORITHMS
    DEFINE
    METRICS

    View Slide

  21. COLLECT/LABEL
    DATA
    IT IS HUMANS WHO
    BIAS IN:
    - REPRESENTATION
    - DISTRIBUTION
    - LABELS
    AND MORE…..
    WRITE
    ALGORITHMS
    DEFINE
    METRICS

    View Slide

  22. IT IS HUMANS WHO
    DEFINE
    METRICS
    WRITE
    ALGORITHMS
    COLLECT/LABEL
    DATA
    - TRAIN/TEST SPLIT
    - FEATURES/PROXIES
    - BLACK-BOX MODELS
    AND MORE…..

    View Slide

  23. IT IS HUMANS WHO
    COLLECT/LABEL
    DATA
    DEFINE
    METRICS
    WRITE
    ALGORITHMS
    - WHAT IS THE IMPACT OF
    DIFFERENT ERROR TYPES ON
    DIFFERENT GROUPS?
    - WHAT DO YOU OPTIMIZE FOR?

    View Slide

  24. Practitioners consistently:
    - overestimate their model’s accuracy.
    - propagate feedback loops.
    - fail to notice data leaks.


    “Why Should I Trust You?” Explaining the Predictions of Any Classifier
    https:/
    /arxiv.org/pdf/1602.04938.pdf

    View Slide

  25. ETHICAL IMPLICATIONS
    INTERPRETABLE MACHINE LEARNING: WHY?

    View Slide

  26. TRANSPARENCY
    INTERPRETABLE MACHINE LEARNING: WHY?

    View Slide

  27. DEBUGGING MODELS
    INTERPRETABLE MACHINE LEARNING: WHY?

    View Slide

  28. SECURITY
    (e.g. Dealing with adversarial attacks)
    INTERPRETABLE MACHINE LEARNING: WHY?

    View Slide

  29. 1 LIME
    https:/
    /github.com/thomasp85/lime
    Local Interpretable Model-agnostic Explanations
    iml https:/
    /github.com/christophM/iml
    INTERPRETABLE MACHINE LEARNING: HOW?

    View Slide

  30. LIME
    Explain individual predictions
    of black-box models using a
    local interpretable model.


    View Slide

  31. this documentary was boring and quite stupid…..
    I-Tabular Data
    II-Images
    III-Text
    LIME

    View Slide

  32. 1- Select a point to explain (red).
    Based on an example in “Interpretable Machine Learning” Book by Christoph Molnar
    LIME (Tabular Data)

    View Slide

  33. 2- Sample data points.
    LIME (Tabular Data)
    Based on an example in “Interpretable Machine Learning” Book by Christoph Molnar

    View Slide

  34. 3- Weight points according to their
    proximity to the selected point.
    LIME (Tabular Data)
    Based on an example in “Interpretable Machine Learning” Book by Christoph Molnar

    View Slide

  35. 4- Train a weighted, interpretable local model.
    LIME (Tabular Data)
    Based on an example in “Interpretable Machine Learning” Book by Christoph Molnar

    View Slide

  36. 5- Explain the black-box model prediction
    using the local model.
    LIME (Tabular Data)
    Based on an example in “Interpretable Machine Learning” Book by Christoph Molnar

    View Slide

  37. LIME (Tabular Data)
    https:/
    /github.com/OmaymaS/lime_explainer

    View Slide

  38. LIME (Tabular Data)
    GUESS THE
    DATASET
    ?

    View Slide

  39. LIME (Tabular Data)
    Iris
    (Classification)

    View Slide

  40. set.seed(5658)
    ## load libraries
    library(caret)
    library(lime)
    ## partition the data
    intrain <- createDataPartition(y = iris$Species,p = 0.8, list = F)
    ## create train and test data
    train_data <- iris[intrain, ]
    test_data <- iris[-intrain, ]
    ## train Random Forest model on train_data
    model <- train(x = train_data[, 1:4], y = train_data[, 5],
    method = 'rf')
    TRAIN

    View Slide

  41. set.seed(5658)
    ## load libraries
    library(caret)
    library(lime)
    ## partition the data
    intrain <- createDataPartition(y = iris$Species, p = 0.8, list = F)
    ## create train and test data
    train_data <- iris[intrain, ]
    test_data <- iris[-intrain, ]
    ## train Random Forest model on train_data
    model <- train(x = train_data[, 1:4], y = train_data[, 5],
    method = 'rf')
    ## create an explainer object using train_data
    explainer <- lime(train_data, model)
    EXPLAIN

    View Slide

  42. set.seed(5658)
    ## load libraries
    library(caret)
    library(lime)
    ## partition the data
    intrain <- createDataPartition(y = iris$Species, p = 0.8, list = F)
    ## create train and test data
    train_data <- iris[intrain, ]
    test_data <- iris[-intrain, ]
    ## train Random Forest model on train_data
    model <- train(x = train_data[, 1:4], y = train_data[, 5],
    method = 'rf')
    ## create an explainer object using train_data
    explainer <- lime(train_data, model)
    ## explain new observations in test data
    explanation <- explain(test_data[, 1], explainer,
    n_labels = 1, n_features = 4)
    EXPLAIN

    View Slide

  43. set.seed(5658)
    ## load libraries
    library(caret)
    library(lime)
    ## partition the data
    intrain <- createDataPartition(y = iris$Species, p = 0.8, list = F)
    ## create train and test data
    train_data <- iris[intrain, ]
    test_data <- iris[-intrain, ]
    ## train Random Forest model on train_data
    model <- train(x = train_data[, 1:4], y = train_data[, 5],
    method = 'rf')
    ## create an explainer object using train_data
    explainer <- lime(train_data, model)
    ## explain new observations in test data
    explanation <- explain(test_data[, 1:4], explainer,
    n_labels = 1, n_features = 4)
    https:/
    /github.com/OmaymaS/satRday2019_talk_scripts/blob/master/R/lime_tabular.R

    View Slide

  44. ## plot features
    plot_features(explanation)
    LIME (Tabular Data)

    View Slide

  45. LIME (Tabular Data)

    View Slide

  46. LIME (Tabular Data)

    View Slide

  47. Label: toy terrier
    Probability: 0.81
    Explanation Fit: 0.38
    LIME (Images)
    pre-trained ImageNet model

    View Slide

  48. LIME (Images)
    superpixels
    50 100 150

    View Slide

  49. Label: tabby, tabby cat
    Probability: 0.29
    Explanation Fit: 0.77
    Label: Egyptian Cat
    Probability: 0.28
    Explanation Fit: 0.69
    LIME (Images)
    pre-trained ImageNet model

    View Slide

  50. Label: tabby, tabby cat
    Probability: 0.29
    Explanation Fit: 0.77
    Label: Egyptian Cat
    Probability: 0.28
    Explanation Fit: 0.69
    Type: Supports
    Type: Contradicts
    LIME (Images)
    pre-trained ImageNet model

    View Slide

  51. LIME (Images)
    “Why Should I Trust You?” Explaining the Predictions of Any Classifier
    https:/
    /arxiv.org/pdf/1602.04938.pdf

    View Slide

  52. LIME (Images)
    “Why Should I Trust You?” Explaining the Predictions of Any Classifier
    https:/
    /arxiv.org/pdf/1602.04938.pdf
    SNOW

    View Slide

  53. IMDB reviews sentiment classification
    LIME (Text)
    Keras model (CNN+LSTM)

    View Slide

  54. Boring
    Stupid
    Dumb
    Waste
    information
    Label predicted: -ve sentiment
    LIME (Text)

    View Slide

  55. Pros
    LIME
    - Provides human-friendly
    explanations.
    - Gives a fidelity measure.
    - Can use other features than
    the black-box model.

    View Slide

  56. Pros
    LIME
    Cons
    - Provides human-friendly
    explanations.
    - Gives a fidelity measure.
    - Can use other features than
    the original model.
    - The definition of proximity is
    not totally resolved in tabular
    data.
    - Instability of explanations.

    View Slide

  57. Pros
    - Provides human-friendly
    explanations.
    - Gives a fidelity measure.
    - Can use other features than
    the original model.
    Cons
    - Instability of explanations.
    LIME
    - The definition of proximity is
    not totally resolved in tabular
    data.

    View Slide

  58. 2 SHAPLEY
    VALUES
    iml https:/
    /github.com/christophM/iml
    INTERPRETABLE MACHINE LEARNING: HOW?

    View Slide

  59. SHAPLEY VALUES
    Explain the difference between
    the actual prediction and the
    average prediction of the
    black-box model.
    coalitional game theory


    View Slide

  60. UCI Bike Sharing
    (Regression)
    SHAPLEY VALUES

    View Slide

  61. library(tidyverse)
    library(caret)
    library(iml)
    ## partition the data
    intrain <- createDataPartition(y = bike$cnt, p = 0.9, list = F)
    ## create train and test data
    train_data <- bike[intrain, ]
    test_data <- bike[-intrain, ]
    train_x <- select(train_data, -cnt)
    test_x <- select(test_data, -cnt)
    ## train model
    model <- train(x = train_x, y = train_data$cnt,
    method = 'rf', ntree = 30, maximise = FALSE)
    TRAIN

    View Slide

  62. library(tidyverse)
    library(caret)
    library(iml)
    ## partition the data
    intrain <- createDataPartition(y = bike$cnt, p = 0.9, list = F)
    ## create train and test data
    train_data <- bike[intrain, ]
    test_data <- bike[-intrain, ]
    train_x <- select(train_data, -cnt)
    test_x <- select(test_data, -cnt)
    ## train model
    model <- train(x = train_x, y = train_data$cnt,
    method = 'rf', ntree = 30, maximise = FALSE)
    ## create predictor
    predictor <- Predictor$new(model, data = train_x)
    EXPLAIN

    View Slide

  63. library(tidyverse)
    library(caret)
    library(iml)
    ## partition the data
    intrain <- createDataPartition(y = bike$cnt, p = 0.9, list = F)
    ## create train and test data
    train_data <- bike[intrain, ]
    test_data <- bike[-intrain, ]
    train_x <- select(train_data, -cnt)
    test_x <- select(test_data, -cnt)
    ## train model
    model <- train(x = train_x, y = train_data$cnt,
    method = 'rf', ntree = 30, maximise = FALSE)
    ## create predictor
    predictor <- Predictor$new(model, data = train_x)
    ## calculate shapley values for a new instance
    shapley_values <- Shapley$new(predictor, x.interest = test_x[10,])
    EXPLAIN

    View Slide

  64. library(tidyverse)
    library(caret)
    library(iml)
    ## partition the data
    intrain <- createDataPartition(y = bike$cnt, p = 0.9, list = F)
    ## create train and test data
    train_data <- bike[intrain, ]
    test_data <- bike[-intrain, ]
    train_x <- select(train_data, -cnt)
    test_x <- select(test_data, -cnt)
    ## train model
    model <- train(x = train_x, y = train_data$cnt,
    method = 'rf', ntree = 30, maximise = FALSE)
    ## create predictor
    predictor <- Predictor$new(model, data = train_x)
    ## calculate shapley values for a new instance
    shapley_values <- Shapley$new(predictor, x.interest = new_istance)
    https:/
    /github.com/OmaymaS/satRday2019_talk_scripts/blob/master/R/shapley_tabular.R

    View Slide

  65. SHAPLEY VALUES

    View Slide

  66. SHAPLEY VALUES
    The contribution of temp value (4.416) to the difference between
    the actual prediction and the mean prediction is the estimated
    Shapley value (~ -1000).

    View Slide

  67. SHAPLEY VALUES

    View Slide

  68. Pros
    - Solid theory.
    - The difference between the
    prediction and the average
    prediction is fairly distributed
    among the feature values of
    the instance.
    SHAPLEY VALUES

    View Slide

  69. Pros Cons
    - Solid theory
    - The difference between the
    prediction and the average
    prediction is fairly distributed
    among the feature values of
    the instance.
    - Computationally expensive.
    - Can be misinterpreted.
    SHAPLEY VALUES
    - Uses all features (not ideal for
    explanations that contain few
    features).

    View Slide

  70. Pros Cons
    - Solid theory.
    - The difference between the
    prediction and the average
    prediction is fairly distributed
    among the feature values of
    the instance.
    - Computationally expensive.
    - Can be misinterpreted.
    SHAPLEY VALUES
    - Uses all features (not ideal for
    explanations that contain few
    features).

    View Slide

  71. > LIME
    > SHAPLEY VALUES
    > ANCHORS
    …. And more
    iml
    METHODS AND PACKAGES

    View Slide

  72. EXTRA READINGS

    View Slide

  73. View Slide

  74. INTERPRETING MACHINE
    LEARNING MODELS:
    WHY AND HOW!
    OMAYMA SAID

    View Slide