Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Explainable Artificial Intelligence to open black-box models

Using Explainable Artificial Intelligence to open black-box models

As machine learning becomes a crucial component of a growing number of user-facing applications, interpretable machine learning has become an increasingly important area of research for several reasons. First, as humans are the ones who train, deploy, and often use the predictions of machine learning models in the real world, it is of utmost importance for us to be able to trust the model. Apart from indicators such as accuracy on sample instances, a user’s trust is directly impacted by how much they can understand and predict the model’s behavior, as opposed to treating it as a black box. The good news is that we have made great strides in some areas of explainable AI. The bad news is that creating explainable AI is not easy and simple as related in medium articles. In this talk, I defend that we should separate explanations from the model (i.e. being model agnostic) because true model interpretability will cost performance and accuracy.


Carla Vieira

August 06, 2020


  1. Using Explainable Artificial Intelligence to open black-box models Carla Vieira

    @carlaprvieira Illustration: Hanne Mostard
  2. Software Engineer (python <3), master student in Artificial Intelligence and

    Google Developer Expert in Machine Learning. Co-organizer of perifaCode Community. contato@carlavieira.dev | carlavieira.dev Carla Vieira
  3. ?

  4. None
  5. None
  6. None
  7. ?

  8. bias data privacy legislation ethics

  9. None
  10. None
  11. We need to talk less about Artificial Intelligence hype …

    … and more about how we are using this technology.
  12. COMPAS Software (2016) Estudo do software COMPAS

  13. Gender Shades (2018)

  14. Black people affected by racial bias in health-care (2019) 5

    states: Bahia, Santa Catarina, Paraíba, Rio e Ceará https://www.nature.com/articles/d41586-019-03228-6
  15. Black people affected by racial bias in health-care (2019) 5

    states: Bahia, Santa Catarina, Paraíba, Rio e Ceará https://www.nature.com/articles/d41586-019-03228-6 "The researchers found that the algorithm assigned risk scores to patients on the basis of total health-care costs accrued in one year." "The scientists speculate that this reduced access to care is due to the effects of systemic racism, ranging from distrust of the health-care system to direct racial discrimination by health-care providers."
  16. Brazil (2019) 5 states: Bahia, Santa Catarina, Paraíba, Rio e

  17. Research and Development of Facial recognition (2019) Joy Buolamwini Founder

    Algorithmic Justice League Deb Raji AI Now Institute Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products https://www.media.mit.edu/publications/actionable-auditing-investigating-the-imp act-of-publicly-naming-biased-performance-results-of-commercial-ai-products/
  18. Wrongfully Accused by an Algorithm (2020) 5 estados: Bahia, Santa

    Catarina, Paraíba, Rio e Ceará https://www.nytimes.com/2020/06/24/technology/facial-rec ognition-arrest.html
  19. Research and Development of Facial recognition (2020)

  20. Developer creates and publishes a tool that reinforces white supremacy

    (2020) https://altdeep.substack.com/p/two-things-you-might-have-missed
  21. MIT removes dataset (2020) https://www.theregister.com/2020/07/01/mit_dataset_removed/

  22. Problem We can talk about a algorithmic society" (O'NEIL, 2016)

    that incorporates new technology in your everyday lives without having a critical thought about it, seeing only the utility point of view, not thinking about how meritocracy e surveillance are scaled and automated into black-boxes we don't have access.
  23. Human bias Technology How this happens?

  24. Is Technology neutral?

  25. "When the field of AI believes it is neutral, it

    both fails to notice biased data and builds systems that sanctify the status quo and advance the interests of the powerful. What is needed is a field that exposes and critiques systems that concentrate power, while co-creating new systems with impacted communities: AI by and for the people." Pratyusha Kalluri (PhD Computer Science, Stanford) Is Technology neutral?
  26. Human bias Technology How to remove bias?

  27. Human bias Technology How to avoid bias?

  28. Human bias Technology Diversity

  29. Gender Gap in Artificial Intelligence “Only 22% of AI professionals

    globally are female, compared to 78% who are male.” (The Global Gender Gap Report 2018 - p.28)
  30. “Whether AI will help us reach our aspirations or reinforce

    the unjust inequalities is ultimately up to us.” Joy Buolamwini
  31. Even though these decisions affect humans, to optimize task performance

    ML models often become too complex to be intelligible to humans: black-box models .


  34. None
  35. None

  37. How to open the Black-box?

  38. How to open the Black-box? TRUST EXPLAINABILITY TRANSPARENCY

  39. Explainable AI (XAI) refers to methods and techniques in the

    application of artificial intelligence technology (AI) such that the results of the solution can be understood by humans. 39 Explainable AI (XAI)?
  40. • Interpretability: "systems are interpretable if their operations can be

    understood by a human, either through introspection or through a produced explanation." (BIRAN, 2017) • Explainability: is the model ability of offering an explanation of its predictions 40 Interpretability x Explainability

    OPEN BLACK-BOX Explainability methods (GUIDOTTI et al., 2018)

    "Model-agnostic methods allow explaining predictions of arbitrary machine learning models independent of the implementation." (MENGNAN et al., 2019)
  43. "Local explanations target to identify the contributions of each feature

    in the input toward a specific model prediction." (MENGNAN et al., 2019) "Global explanations aims to provide a global understanding about what knowledge has been acquired by these pretrained models and illuminate the parameters or learned representations in an intuitive manner to humans." (MENGNAN et al., 2019) 43 Local x Global explanations
  44. - Model-Agnostic Methods - Local explanation - LIME focuses on

    training local surrogate models to explain individual predictions, using data perturbation - Data: tabular data, text, images Python - LIME 44 LIME (RIBEIRO et al., 2016)
  45. SHAP (SHapley Additive exPlanations) is a game theoretic approach to

    explain the output of any machine learning model. - Model-Agnostic Methods - Local explanation Python - SHAP 45 SHAP (LUND-BERG e LEE, 2017)
  46. A counterfactual explanation describes a causal situation in the form:

    "If A had not occurred, B would not have occurred" In interpretable machine learning, counterfactual explanations can be used to explain predictions of individual instances. 46 Counterfactual Explanations
  47. http://aix360.mybluemix.net/explanation_cust# Counterfactual Explanations

  48. http://aix360.mybluemix.net/explanation_cust# Counterfactual Explanations

  49. http://aix360.mybluemix.net/explanation_cust# Counterfactual Explanations

  50. - While global understanding is particularly important for assessing trust

    in a model as a whole (before deployment), most of current research has been devoted to explaining individual predictions. - Ribeiro (2018) believes there is an unexplored opportunity in coming up with explanations that are global in nature. 50 Explainable AI
  51. “What this new wave of XAI researchers agree on is

    that if AI systems are to be used by more people, those people must be part of the design from the start—and different people need different kinds of explanations.”
  52. "When the field of AI believes it is neutral, it

    both fails to notice biased data and builds systems that sanctify the status quo and advance the interests of the powerful. What is needed is a field that exposes and critiques systems that concentrate power, while co-creating new systems with impacted communities: AI by and for the people." Pratyusha Kalluri (PhD Computer Science, Stanford)
  53. The choices we are making today about Artificial Intelligence are

    going to define our future.
  54. Race After Technology: Abolitionist Tools for the New Jim Code

    (Ruha Benjamin) From everyday apps to complex algorithms, Ruha Benjamin cuts through tech-industry hype to understand how emerging technologies can reinforce White supremacy and deepen social inequity. Benjamin argues that automation, far from being a sinister story of racist programmers scheming on the dark web, has the potential to hide, speed up, and deepen discrimination while appearing neutral and even benevolent when compared to the racism of a previous era.
  55. Weapons of Math Destruction: How Big Data Increases Inequality and

    Threatens Democracy (Cathy O'Neil) We live in the age of the algorithm. Increasingly, the decisions that affect our lives—where we go to school, whether we can get a job or a loan, how much we pay for health insurance—are being made not by humans, but by machines. In theory, this should lead to greater fairness: Everyone is judged according to the same rules.
  56. Communities, Algorithms and Digital Activisms: African Diaspora views (Tarcízio Silva)

    Organized by Tarcízio Silva and published by the LiteraRUA publisher, this book brings together 14 chapters of researchers from Brazil and countries from African Diaspora and Africa, such as Congo, Ethiopia, Ghana, Nigeria, Colombia, United States and United Kingdom.
  57. Ogunhê Podcast (Ana Carolina da Hora) Ogunhê is the salute

    to the orisha OGUM: the orisha of war and technology. Ogun, to survive in the forest and in wars, created his weapons (technologies) among other objects to generate changes around him. This project aims to share and present scientists from the African continent and their scientific contributions that help society.
  58. @carlaprvieira (instagram/twitter) contato@carlavieira.dev | carlavieira.dev youtube.com/EAICarla/ | twitch.tv/carlaprv perifacode.com Thanks!

  59. − Newsletters − MIT Review – The Download (en) −

    MIT Review – The Algorithm (en) − Data & Society (en): Data & Society advances public understanding of the social implications of data-centric technologies and automation. − AI Weekly by Khari Johson (en) − Data Hackers (pt): − Desvelar (Tecnologia e Sociedade) (pt): − Podcasts − Artificial Intelligence: AI Podcast by Lex Fridman (en) − Crazy for data (pt) − The Received Wisdom Podcast (en) − 10 TED Talks about AI
  60. References BARAKAT, N. H.; BRADLEY, A. P. (2007) Rule extraction

    from support vector machines: Asequential covering approach. IEEE Transactions on Knowledge and Data Engineering. Disponível em: https://doi.org/10.1109/TKDE.2007.190610 Doran, D., Schulz, S.C. e Besold, T. R. 2018. What Does Explainable AI Really Mean? A New Conceptualization of Perspectives. CEUR Workshop Proceedings, 2018. Disponível em: https://arxiv.org/abs/1710.00794 Freitas, A. 2014. Comprehensible classification models: A position paper. Disponível em: https://doi.org/10.1145/2594473.2594475 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F.,Pedreschi, D. e Giannotti, F. 2018. A survey of methods for explaining black box models. Disponível em: https://arxiv.org/abs/1802.01933 Hymas, C. (2019) AI used for first time in job interviews in UK to find best applicants. The Telegraph Disponível em: https://www.telegraph.co.uk/news/2019/09/27/ai-facial-recognition-used-first-time-job-interviews-uk-find/ Ribeiro, M. T., Singh, S. e Guestrin, C. 2016.Model-agnostic interpretability of machine learning,Cornell University. Disponível em: https://arxiv.org/abs/1606.05386 Ledford, H. 2019. Millions of black people affected by racial bias in health-care algorithms. Nature. Disponível em: https://www.nature.com/articles/d41586-019-03228-6 60
  61. References LUNDBERG, S. M.; ERION, G. G.; CHEN, H.; DEGRAVE,

    A.; PRUTKIN, J. M.; NAIR,B.; KATZ, R.; HIMMELFARB, J.; BANSAL, N.; LEE, S. 2019. Explainable AI for trees: From local explanations to global understanding. Disponível em:http://arxiv.org/abs/1905.04610 LUNDBERG, S. M.; LEE, S.-I. 2017. A unified approach to interpreting model predictions.In: GUYON, I.; LUXBURG, U. V.; BENGIO, S.; WALLACH, H.; FERGUS, R.;VISHWANATHAN, S.; GARNETT, R. (Ed.). Disponível em:http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf Mengnan, D., Ninghao, L., e Xia, H. 2019. Techniques for interpretable machine learning. Disponível em: https://dl.acm.org/doi/10.1145/3359786 Obermeyer, Z., Powers, B., Vogeli, C. e Mullainathan, S. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Disponível em: https://science.sciencemag.org/content/366/6464/447 O’NEIL, C.Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. USA: Crown Publishing Group, 2016. ISBN 0553418815. Papadopoulos, P. e Walkinshaw, N. 2015. "Black-Box Test Generation from Inferred Models".Disponível em: https://ieeexplore.ieee.org/document/7168327 Vieira, C. P. R.; Digiampietri, L. A. 2020. A study about Explainable Artificial Intelligence: using decision tree to explain SVM. Disponível em: http://seer.upf.br/index.php/rbca/article/view/10247 61