Using Explainable Artificial Intelligence to open black-box models

As machine learning becomes a crucial component of a growing number of user-facing applications, interpretable machine learning has become an increasingly important area of research for several reasons. First, as humans are the ones who train, deploy, and often use the predictions of machine learning models in the real world, it is of utmost importance for us to be able to trust the model. Apart from indicators such as accuracy on sample instances, a user’s trust is directly impacted by how much they can understand and predict the model’s behavior, as opposed to treating it as a black box. The good news is that we have made great strides in some areas of explainable AI. The bad news is that creating explainable AI is not easy and simple as related in medium articles. In this talk, I defend that we should separate explanations from the model (i.e. being model agnostic) because true model interpretability will cost performance and accuracy.

Transcript

1. Using Explainable
   Artificial Intelligence to
   open black-box models
   Carla Vieira
   @carlaprvieira
   Illustration: Hanne Mostard
   

   View Slide

2. Software Engineer (python <3), master student in
   Artificial Intelligence and Google Developer Expert in
   Machine Learning. Co-organizer of perifaCode
   Community.
   contato@carlavieira.dev | carlavieira.dev
   Carla Vieira
   

   View Slide

3. ?
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. ?
   

   View Slide

8. bias
   data
   privacy legislation
   ethics
   

   View Slide

9. View Slide

10. View Slide

11. We need to talk less about Artificial Intelligence
    hype …
    … and more about how we are using this
    technology.
    

    View Slide

12. COMPAS
    Software
    (2016)
    Estudo do software COMPAS
    

    View Slide

13. Gender Shades
    (2018)
    

    View Slide

14. Black people
    affected by racial
    bias in health-care
    (2019)
    5 states: Bahia, Santa Catarina, Paraíba, Rio e Ceará
    https://www.nature.com/articles/d41586-019-03228-6
    

    View Slide

15. Black people
    affected by racial
    bias in health-care
    (2019)
    5 states: Bahia, Santa Catarina, Paraíba, Rio e Ceará
    https://www.nature.com/articles/d41586-019-03228-6
    "The researchers found that the algorithm assigned risk scores to
    patients on the basis of total health-care costs accrued in one
    year."
    "The scientists speculate that this reduced access to care is due to
    the effects of systemic racism, ranging from distrust of the
    health-care system to direct racial discrimination by health-care
    providers."
    

    View Slide

16. Brazil
    (2019) 5 states: Bahia, Santa Catarina, Paraíba, Rio e Ceará
    

    View Slide

17. Research and
    Development of
    Facial recognition
    (2019)
    Joy Buolamwini
    Founder Algorithmic
    Justice League
    Deb Raji
    AI Now Institute
    Actionable Auditing: Investigating
    the Impact of Publicly Naming
    Biased Performance Results of
    Commercial AI Products
    https://www.media.mit.edu/publications/actionable-auditing-investigating-the-imp
    act-of-publicly-naming-biased-performance-results-of-commercial-ai-products/
    

    View Slide

18. Wrongfully
    Accused by an
    Algorithm
    (2020)
    5 estados: Bahia, Santa Catarina, Paraíba, Rio e
    Ceará
    https://www.nytimes.com/2020/06/24/technology/facial-rec
    ognition-arrest.html
    

    View Slide

19. Research and
    Development of
    Facial recognition
    (2020)
    

    View Slide

20. Developer creates
    and publishes a tool
    that reinforces white
    supremacy
    (2020)
    https://altdeep.substack.com/p/two-things-you-might-have-missed
    

    View Slide

21. MIT removes
    dataset
    (2020)
    https://www.theregister.com/2020/07/01/mit_dataset_removed/
    

    View Slide

22. Problem
    We can talk about a algorithmic society" (O'NEIL, 2016) that incorporates
    new technology in your everyday lives without having a critical thought
    about it, seeing only the utility point of view, not thinking about how
    meritocracy e surveillance are scaled and automated into black-boxes
    we don't have access.
    

    View Slide

23. Human bias Technology
    How this happens?
    

    View Slide

24. Is Technology neutral?
    

    View Slide

25. "When the field of AI believes it is neutral, it both fails to notice biased data
    and builds systems that sanctify the status quo and advance the interests
    of the powerful. What is needed is a field that exposes and critiques
    systems that concentrate power, while co-creating new systems with
    impacted communities: AI by and for the people."
    Pratyusha Kalluri (PhD Computer Science, Stanford)
    Is Technology neutral?
    

    View Slide

26. Human bias Technology
    How to remove bias?
    

    View Slide

27. Human bias Technology
    How to avoid bias?
    

    View Slide

28. Human bias Technology
    Diversity
    

    View Slide

29. Gender Gap in Artificial Intelligence
    “Only 22% of AI professionals globally are female,
    compared to 78% who are male.”
    (The Global Gender Gap Report 2018 - p.28)
    

    View Slide

30. “Whether AI will help us reach our aspirations or
    reinforce the unjust inequalities is ultimately up to
    us.”
    Joy Buolamwini
    

    View Slide

31. Even though these decisions affect humans,
    to optimize task performance ML models
    often become too complex to be intelligible to
    humans: black-box models .
    

    View Slide

32. BLACK-BOX
    INPUT OUTPUT
    

    View Slide

33. BLACK-BOX
    INPUT OUTPUT
    ?
    

    View Slide

34. View Slide

35. View Slide

36. Black-box models?
    RANDOM FOREST DEEP NEURAL
    NETWORKS
    

    View Slide

37. How to open the Black-box?
    

    View Slide

38. How to open the Black-box?
    TRUST
    EXPLAINABILITY
    TRANSPARENCY
    

    View Slide

39. Explainable AI (XAI) refers to methods and techniques in the
    application of artificial intelligence technology (AI) such that the
    results of the solution can be understood by humans.
    39
    Explainable AI (XAI)?
    

    View Slide

40. • Interpretability: "systems are interpretable if their operations can be
    understood by a human, either through introspection or through a
    produced explanation." (BIRAN, 2017)
    • Explainability: is the model ability of offering an explanation of its
    predictions
    40
    Interpretability x Explainability
    

    View Slide

41. WHITE-BOX MODELS
    BLACK-BOX EXPLANATION
    OUTPUT EXPLANATION
    MODEL EXPLANATION MODEL INSPECTION
    OPEN BLACK-BOX
    Explainability methods (GUIDOTTI et al., 2018)
    

    View Slide

42. 42
    Model Agnostic Methods
    BLACK-BOX
    MODEL
    OUTPUT
    DATA METHODS EXPLANATIONS
    "Model-agnostic methods allow explaining predictions of arbitrary
    machine learning models independent of the implementation."
    (MENGNAN et al., 2019)
    

    View Slide

43. "Local explanations target to identify the contributions of each feature in
    the input toward a specific model prediction." (MENGNAN et al., 2019)
    "Global explanations aims to provide a global understanding about what
    knowledge has been acquired by these pretrained models and illuminate
    the parameters or learned representations in an intuitive manner to
    humans." (MENGNAN et al., 2019)
    43
    Local x Global explanations
    

    View Slide

44. - Model-Agnostic Methods
    - Local explanation
    - LIME focuses on training local surrogate models to explain
    individual predictions, using data perturbation
    - Data: tabular data, text, images
    Python - LIME
    44
    LIME (RIBEIRO et al., 2016)
    

    View Slide

45. SHAP (SHapley Additive exPlanations) is a game
    theoretic approach to explain the output of any
    machine learning model.
    - Model-Agnostic Methods
    - Local explanation
    Python - SHAP
    45
    SHAP (LUND-BERG e LEE, 2017)
    

    View Slide

46. A counterfactual explanation describes a causal situation in the form:
    "If A had not occurred, B would not have occurred"
    In interpretable machine learning, counterfactual explanations can be
    used to explain predictions of individual instances.
    46
    Counterfactual Explanations
    

    View Slide

47. http://aix360.mybluemix.net/explanation_cust#
    Counterfactual Explanations
    

    View Slide

48. http://aix360.mybluemix.net/explanation_cust#
    Counterfactual Explanations
    

    View Slide

49. http://aix360.mybluemix.net/explanation_cust#
    Counterfactual Explanations
    

    View Slide

50. - While global understanding is particularly important for assessing trust in a
    model as a whole (before deployment), most of current research has been
    devoted to explaining individual predictions.
    - Ribeiro (2018) believes there is an unexplored opportunity in coming up
    with explanations that are global in nature.
    50
    Explainable AI
    

    View Slide

51. “What this new wave of XAI
    researchers agree on is that if AI
    systems are to be used by more
    people, those people must be part of
    the design from the start—and
    different people need different kinds
    of explanations.”
    

    View Slide

52. "When the field of AI believes it is neutral, it both fails to notice biased data
    and builds systems that sanctify the status quo and advance the interests
    of the powerful. What is needed is a field that exposes and critiques
    systems that concentrate power, while co-creating new systems with
    impacted communities: AI by and for the people."
    Pratyusha Kalluri (PhD Computer Science, Stanford)
    

    View Slide

53. The choices we are making today about
    Artificial Intelligence are going to define
    our future.
    

    View Slide

54. Race After Technology: Abolitionist Tools
    for the New Jim Code (Ruha Benjamin)
    From everyday apps to complex algorithms, Ruha Benjamin
    cuts through tech-industry hype to understand how emerging
    technologies can reinforce White supremacy and deepen
    social inequity.
    Benjamin argues that automation, far from being a sinister
    story of racist programmers scheming on the dark web, has
    the potential to hide, speed up, and deepen discrimination
    while appearing neutral and even benevolent when compared
    to the racism of a previous era.
    

    View Slide

55. Weapons of Math Destruction: How Big
    Data Increases Inequality and Threatens
    Democracy (Cathy O'Neil)
    We live in the age of the algorithm. Increasingly, the decisions
    that affect our lives—where we go to school, whether we can get
    a job or a loan, how much we pay for health insurance—are being
    made not by humans, but by machines. In theory, this should
    lead to greater fairness: Everyone is judged according to the
    same rules.
    

    View Slide

56. Communities, Algorithms and Digital
    Activisms: African Diaspora views (Tarcízio
    Silva)
    Organized by Tarcízio Silva and published by the
    LiteraRUA publisher, this book brings together 14 chapters
    of researchers from Brazil and countries from African
    Diaspora and Africa, such as Congo, Ethiopia, Ghana,
    Nigeria, Colombia, United States and United Kingdom.
    

    View Slide

57. Ogunhê Podcast (Ana Carolina da
    Hora)
    Ogunhê is the salute to the orisha OGUM: the
    orisha of war and technology. Ogun, to survive in
    the forest and in wars, created his weapons
    (technologies) among other objects to generate
    changes around him. This project aims to share
    and present scientists from the African continent
    and their scientific contributions that help society.
    

    View Slide

58. @carlaprvieira (instagram/twitter)
    contato@carlavieira.dev | carlavieira.dev
    youtube.com/EAICarla/ | twitch.tv/carlaprv
    perifacode.com
    Thanks!
    

    View Slide

59. − Newsletters
    − MIT Review – The Download (en)
    − MIT Review – The Algorithm (en)
    − Data & Society (en): Data & Society advances public understanding of the social implications of
    data-centric technologies and automation.
    − AI Weekly by Khari Johson (en)
    − Data Hackers (pt):
    − Desvelar (Tecnologia e Sociedade) (pt):
    − Podcasts
    − Artificial Intelligence: AI Podcast by Lex Fridman (en)
    − Crazy for data (pt)
    − The Received Wisdom Podcast (en)
    − 10 TED Talks about AI
    

    View Slide

60. References
    BARAKAT, N. H.; BRADLEY, A. P. (2007) Rule extraction from support vector machines: Asequential covering approach. IEEE
    Transactions on Knowledge and Data Engineering. Disponível em: https://doi.org/10.1109/TKDE.2007.190610
    Doran, D., Schulz, S.C. e Besold, T. R. 2018. What Does Explainable AI Really Mean? A New Conceptualization of Perspectives. CEUR
    Workshop Proceedings, 2018. Disponível em: https://arxiv.org/abs/1710.00794
    Freitas, A. 2014. Comprehensible classification models: A position paper. Disponível em: https://doi.org/10.1145/2594473.2594475
    Guidotti, R., Monreale, A., Ruggieri, S., Turini, F.,Pedreschi, D. e Giannotti, F. 2018. A survey of methods for explaining black box models.
    Disponível em: https://arxiv.org/abs/1802.01933
    Hymas, C. (2019) AI used for first time in job interviews in UK to find best applicants. The Telegraph Disponível em:
    https://www.telegraph.co.uk/news/2019/09/27/ai-facial-recognition-used-first-time-job-interviews-uk-find/
    Ribeiro, M. T., Singh, S. e Guestrin, C. 2016.Model-agnostic interpretability of machine learning,Cornell University. Disponível em:
    https://arxiv.org/abs/1606.05386
    Ledford, H. 2019. Millions of black people affected by racial bias in health-care algorithms. Nature. Disponível em:
    https://www.nature.com/articles/d41586-019-03228-6
    60
    

    View Slide

61. References
    LUNDBERG, S. M.; ERION, G. G.; CHEN, H.; DEGRAVE, A.; PRUTKIN, J. M.; NAIR,B.; KATZ, R.; HIMMELFARB, J.; BANSAL, N.; LEE, S. 2019. Explainable AI for
    trees: From local explanations to global understanding. Disponível em:http://arxiv.org/abs/1905.04610
    LUNDBERG, S. M.; LEE, S.-I. 2017. A unified approach to interpreting model predictions.In: GUYON, I.; LUXBURG, U. V.; BENGIO, S.; WALLACH, H.; FERGUS,
    R.;VISHWANATHAN, S.; GARNETT, R. (Ed.). Disponível em:http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf
    Mengnan, D., Ninghao, L., e Xia, H. 2019. Techniques for interpretable machine learning. Disponível em: https://dl.acm.org/doi/10.1145/3359786
    Obermeyer, Z., Powers, B., Vogeli, C. e Mullainathan, S. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Disponível
    em: https://science.sciencemag.org/content/366/6464/447
    O’NEIL, C.Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. USA: Crown Publishing Group, 2016. ISBN
    0553418815.
    Papadopoulos, P. e Walkinshaw, N. 2015. "Black-Box Test Generation from Inferred Models".Disponível em: https://ieeexplore.ieee.org/document/7168327
    Vieira, C. P. R.; Digiampietri, L. A. 2020. A study about Explainable Artificial Intelligence: using decision tree to explain SVM. Disponível em:
    http://seer.upf.br/index.php/rbca/article/view/10247
    61
    

    View Slide