Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Explainable Artificial Intelligence to open black-box models

Using Explainable Artificial Intelligence to open black-box models

As machine learning becomes a crucial component of a growing number of user-facing applications, interpretable machine learning has become an increasingly important area of research for several reasons. First, as humans are the ones who train, deploy, and often use the predictions of machine learning models in the real world, it is of utmost importance for us to be able to trust the model. Apart from indicators such as accuracy on sample instances, a user’s trust is directly impacted by how much they can understand and predict the model’s behavior, as opposed to treating it as a black box. The good news is that we have made great strides in some areas of explainable AI. The bad news is that creating explainable AI is not easy and simple as related in medium articles. In this talk, I defend that we should separate explanations from the model (i.e. being model agnostic) because true model interpretability will cost performance and accuracy.

Carla Vieira

August 06, 2020
Tweet

More Decks by Carla Vieira

Other Decks in Technology

Transcript

  1. Using Explainable
    Artificial Intelligence to
    open black-box models
    Carla Vieira
    @carlaprvieira
    Illustration: Hanne Mostard

    View full-size slide

  2. Software Engineer (python <3), master student in
    Artificial Intelligence and Google Developer Expert in
    Machine Learning. Co-organizer of perifaCode
    Community.
    [email protected] | carlavieira.dev
    Carla Vieira

    View full-size slide

  3. bias
    data
    privacy legislation
    ethics

    View full-size slide

  4. We need to talk less about Artificial Intelligence
    hype …
    … and more about how we are using this
    technology.

    View full-size slide

  5. COMPAS
    Software
    (2016)
    Estudo do software COMPAS

    View full-size slide

  6. Gender Shades
    (2018)

    View full-size slide

  7. Black people
    affected by racial
    bias in health-care
    (2019)
    5 states: Bahia, Santa Catarina, Paraíba, Rio e Ceará
    https://www.nature.com/articles/d41586-019-03228-6

    View full-size slide

  8. Black people
    affected by racial
    bias in health-care
    (2019)
    5 states: Bahia, Santa Catarina, Paraíba, Rio e Ceará
    https://www.nature.com/articles/d41586-019-03228-6
    "The researchers found that the algorithm assigned risk scores to
    patients on the basis of total health-care costs accrued in one
    year."
    "The scientists speculate that this reduced access to care is due to
    the effects of systemic racism, ranging from distrust of the
    health-care system to direct racial discrimination by health-care
    providers."

    View full-size slide

  9. Brazil
    (2019) 5 states: Bahia, Santa Catarina, Paraíba, Rio e Ceará

    View full-size slide

  10. Research and
    Development of
    Facial recognition
    (2019)
    Joy Buolamwini
    Founder Algorithmic
    Justice League
    Deb Raji
    AI Now Institute
    Actionable Auditing: Investigating
    the Impact of Publicly Naming
    Biased Performance Results of
    Commercial AI Products
    https://www.media.mit.edu/publications/actionable-auditing-investigating-the-imp
    act-of-publicly-naming-biased-performance-results-of-commercial-ai-products/

    View full-size slide

  11. Wrongfully
    Accused by an
    Algorithm
    (2020)
    5 estados: Bahia, Santa Catarina, Paraíba, Rio e
    Ceará
    https://www.nytimes.com/2020/06/24/technology/facial-rec
    ognition-arrest.html

    View full-size slide

  12. Research and
    Development of
    Facial recognition
    (2020)

    View full-size slide

  13. Developer creates
    and publishes a tool
    that reinforces white
    supremacy
    (2020)
    https://altdeep.substack.com/p/two-things-you-might-have-missed

    View full-size slide

  14. MIT removes
    dataset
    (2020)
    https://www.theregister.com/2020/07/01/mit_dataset_removed/

    View full-size slide

  15. Problem
    We can talk about a algorithmic society" (O'NEIL, 2016) that incorporates
    new technology in your everyday lives without having a critical thought
    about it, seeing only the utility point of view, not thinking about how
    meritocracy e surveillance are scaled and automated into black-boxes
    we don't have access.

    View full-size slide

  16. Human bias Technology
    How this happens?

    View full-size slide

  17. Is Technology neutral?

    View full-size slide

  18. "When the field of AI believes it is neutral, it both fails to notice biased data
    and builds systems that sanctify the status quo and advance the interests
    of the powerful. What is needed is a field that exposes and critiques
    systems that concentrate power, while co-creating new systems with
    impacted communities: AI by and for the people."
    Pratyusha Kalluri (PhD Computer Science, Stanford)
    Is Technology neutral?

    View full-size slide

  19. Human bias Technology
    How to remove bias?

    View full-size slide

  20. Human bias Technology
    How to avoid bias?

    View full-size slide

  21. Human bias Technology
    Diversity

    View full-size slide

  22. Gender Gap in Artificial Intelligence
    “Only 22% of AI professionals globally are female,
    compared to 78% who are male.”
    (The Global Gender Gap Report 2018 - p.28)

    View full-size slide

  23. “Whether AI will help us reach our aspirations or
    reinforce the unjust inequalities is ultimately up to
    us.”
    Joy Buolamwini

    View full-size slide

  24. Even though these decisions affect humans,
    to optimize task performance ML models
    often become too complex to be intelligible to
    humans: black-box models .

    View full-size slide

  25. BLACK-BOX
    INPUT OUTPUT

    View full-size slide

  26. BLACK-BOX
    INPUT OUTPUT
    ?

    View full-size slide

  27. Black-box models?
    RANDOM FOREST DEEP NEURAL
    NETWORKS

    View full-size slide

  28. How to open the Black-box?

    View full-size slide

  29. How to open the Black-box?
    TRUST
    EXPLAINABILITY
    TRANSPARENCY

    View full-size slide

  30. Explainable AI (XAI) refers to methods and techniques in the
    application of artificial intelligence technology (AI) such that the
    results of the solution can be understood by humans.
    39
    Explainable AI (XAI)?

    View full-size slide

  31. • Interpretability: "systems are interpretable if their operations can be
    understood by a human, either through introspection or through a
    produced explanation." (BIRAN, 2017)
    • Explainability: is the model ability of offering an explanation of its
    predictions
    40
    Interpretability x Explainability

    View full-size slide

  32. WHITE-BOX MODELS
    BLACK-BOX EXPLANATION
    OUTPUT EXPLANATION
    MODEL EXPLANATION MODEL INSPECTION
    OPEN BLACK-BOX
    Explainability methods (GUIDOTTI et al., 2018)

    View full-size slide

  33. 42
    Model Agnostic Methods
    BLACK-BOX
    MODEL
    OUTPUT
    DATA METHODS EXPLANATIONS
    "Model-agnostic methods allow explaining predictions of arbitrary
    machine learning models independent of the implementation."
    (MENGNAN et al., 2019)

    View full-size slide

  34. "Local explanations target to identify the contributions of each feature in
    the input toward a specific model prediction." (MENGNAN et al., 2019)
    "Global explanations aims to provide a global understanding about what
    knowledge has been acquired by these pretrained models and illuminate
    the parameters or learned representations in an intuitive manner to
    humans." (MENGNAN et al., 2019)
    43
    Local x Global explanations

    View full-size slide

  35. - Model-Agnostic Methods
    - Local explanation
    - LIME focuses on training local surrogate models to explain
    individual predictions, using data perturbation
    - Data: tabular data, text, images
    Python - LIME
    44
    LIME (RIBEIRO et al., 2016)

    View full-size slide

  36. SHAP (SHapley Additive exPlanations) is a game
    theoretic approach to explain the output of any
    machine learning model.
    - Model-Agnostic Methods
    - Local explanation
    Python - SHAP
    45
    SHAP (LUND-BERG e LEE, 2017)

    View full-size slide

  37. A counterfactual explanation describes a causal situation in the form:
    "If A had not occurred, B would not have occurred"
    In interpretable machine learning, counterfactual explanations can be
    used to explain predictions of individual instances.
    46
    Counterfactual Explanations

    View full-size slide

  38. http://aix360.mybluemix.net/explanation_cust#
    Counterfactual Explanations

    View full-size slide

  39. http://aix360.mybluemix.net/explanation_cust#
    Counterfactual Explanations

    View full-size slide

  40. http://aix360.mybluemix.net/explanation_cust#
    Counterfactual Explanations

    View full-size slide

  41. - While global understanding is particularly important for assessing trust in a
    model as a whole (before deployment), most of current research has been
    devoted to explaining individual predictions.
    - Ribeiro (2018) believes there is an unexplored opportunity in coming up
    with explanations that are global in nature.
    50
    Explainable AI

    View full-size slide

  42. “What this new wave of XAI
    researchers agree on is that if AI
    systems are to be used by more
    people, those people must be part of
    the design from the start—and
    different people need different kinds
    of explanations.”

    View full-size slide

  43. "When the field of AI believes it is neutral, it both fails to notice biased data
    and builds systems that sanctify the status quo and advance the interests
    of the powerful. What is needed is a field that exposes and critiques
    systems that concentrate power, while co-creating new systems with
    impacted communities: AI by and for the people."
    Pratyusha Kalluri (PhD Computer Science, Stanford)

    View full-size slide

  44. The choices we are making today about
    Artificial Intelligence are going to define
    our future.

    View full-size slide

  45. Race After Technology: Abolitionist Tools
    for the New Jim Code (Ruha Benjamin)
    From everyday apps to complex algorithms, Ruha Benjamin
    cuts through tech-industry hype to understand how emerging
    technologies can reinforce White supremacy and deepen
    social inequity.
    Benjamin argues that automation, far from being a sinister
    story of racist programmers scheming on the dark web, has
    the potential to hide, speed up, and deepen discrimination
    while appearing neutral and even benevolent when compared
    to the racism of a previous era.

    View full-size slide

  46. Weapons of Math Destruction: How Big
    Data Increases Inequality and Threatens
    Democracy (Cathy O'Neil)
    We live in the age of the algorithm. Increasingly, the decisions
    that affect our lives—where we go to school, whether we can get
    a job or a loan, how much we pay for health insurance—are being
    made not by humans, but by machines. In theory, this should
    lead to greater fairness: Everyone is judged according to the
    same rules.

    View full-size slide

  47. Communities, Algorithms and Digital
    Activisms: African Diaspora views (Tarcízio
    Silva)
    Organized by Tarcízio Silva and published by the
    LiteraRUA publisher, this book brings together 14 chapters
    of researchers from Brazil and countries from African
    Diaspora and Africa, such as Congo, Ethiopia, Ghana,
    Nigeria, Colombia, United States and United Kingdom.

    View full-size slide

  48. Ogunhê Podcast (Ana Carolina da
    Hora)
    Ogunhê is the salute to the orisha OGUM: the
    orisha of war and technology. Ogun, to survive in
    the forest and in wars, created his weapons
    (technologies) among other objects to generate
    changes around him. This project aims to share
    and present scientists from the African continent
    and their scientific contributions that help society.

    View full-size slide

  49. @carlaprvieira (instagram/twitter)
    [email protected] | carlavieira.dev
    youtube.com/EAICarla/ | twitch.tv/carlaprv
    perifacode.com
    Thanks!

    View full-size slide

  50. − Newsletters
    − MIT Review – The Download (en)
    − MIT Review – The Algorithm (en)
    − Data & Society (en): Data & Society advances public understanding of the social implications of
    data-centric technologies and automation.
    − AI Weekly by Khari Johson (en)
    − Data Hackers (pt):
    − Desvelar (Tecnologia e Sociedade) (pt):
    − Podcasts
    − Artificial Intelligence: AI Podcast by Lex Fridman (en)
    − Crazy for data (pt)
    − The Received Wisdom Podcast (en)
    − 10 TED Talks about AI

    View full-size slide

  51. References
    BARAKAT, N. H.; BRADLEY, A. P. (2007) Rule extraction from support vector machines: Asequential covering approach. IEEE
    Transactions on Knowledge and Data Engineering. Disponível em: https://doi.org/10.1109/TKDE.2007.190610
    Doran, D., Schulz, S.C. e Besold, T. R. 2018. What Does Explainable AI Really Mean? A New Conceptualization of Perspectives. CEUR
    Workshop Proceedings, 2018. Disponível em: https://arxiv.org/abs/1710.00794
    Freitas, A. 2014. Comprehensible classification models: A position paper. Disponível em: https://doi.org/10.1145/2594473.2594475
    Guidotti, R., Monreale, A., Ruggieri, S., Turini, F.,Pedreschi, D. e Giannotti, F. 2018. A survey of methods for explaining black box models.
    Disponível em: https://arxiv.org/abs/1802.01933
    Hymas, C. (2019) AI used for first time in job interviews in UK to find best applicants. The Telegraph Disponível em:
    https://www.telegraph.co.uk/news/2019/09/27/ai-facial-recognition-used-first-time-job-interviews-uk-find/
    Ribeiro, M. T., Singh, S. e Guestrin, C. 2016.Model-agnostic interpretability of machine learning,Cornell University. Disponível em:
    https://arxiv.org/abs/1606.05386
    Ledford, H. 2019. Millions of black people affected by racial bias in health-care algorithms. Nature. Disponível em:
    https://www.nature.com/articles/d41586-019-03228-6
    60

    View full-size slide

  52. References
    LUNDBERG, S. M.; ERION, G. G.; CHEN, H.; DEGRAVE, A.; PRUTKIN, J. M.; NAIR,B.; KATZ, R.; HIMMELFARB, J.; BANSAL, N.; LEE, S. 2019. Explainable AI for
    trees: From local explanations to global understanding. Disponível em:http://arxiv.org/abs/1905.04610
    LUNDBERG, S. M.; LEE, S.-I. 2017. A unified approach to interpreting model predictions.In: GUYON, I.; LUXBURG, U. V.; BENGIO, S.; WALLACH, H.; FERGUS,
    R.;VISHWANATHAN, S.; GARNETT, R. (Ed.). Disponível em:http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf
    Mengnan, D., Ninghao, L., e Xia, H. 2019. Techniques for interpretable machine learning. Disponível em: https://dl.acm.org/doi/10.1145/3359786
    Obermeyer, Z., Powers, B., Vogeli, C. e Mullainathan, S. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Disponível
    em: https://science.sciencemag.org/content/366/6464/447
    O’NEIL, C.Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. USA: Crown Publishing Group, 2016. ISBN
    0553418815.
    Papadopoulos, P. e Walkinshaw, N. 2015. "Black-Box Test Generation from Inferred Models".Disponível em: https://ieeexplore.ieee.org/document/7168327
    Vieira, C. P. R.; Digiampietri, L. A. 2020. A study about Explainable Artificial Intelligence: using decision tree to explain SVM. Disponível em:
    http://seer.upf.br/index.php/rbca/article/view/10247
    61

    View full-size slide