Using Explainable Artificial Intelligence to open black-box models Carla Vieira @carlaprvieira Illustration: Hanne Mostard 

Software Engineer (python <3), master student in Artificial Intelligence and Google Developer Expert in Machine Learning. Co-organizer of perifaCode Community. [email protected] | carlavieira.dev Carla Vieira 

? 

No content

No content

No content

? 

bias data privacy legislation ethics 

No content

No content

We need to talk less about Artificial Intelligence hype … … and more about how we are using this technology. 

COMPAS Software (2016) Estudo do software COMPAS 

Gender Shades (2018) 

Black people affected by racial bias in health-care (2019) 5 states: Bahia, Santa Catarina, Paraíba, Rio e Ceará https://www.nature.com/articles/d41586-019-03228-6 

Black people affected by racial bias in health-care (2019) 5 states: Bahia, Santa Catarina, Paraíba, Rio e Ceará https://www.nature.com/articles/d41586-019-03228-6 "The researchers found that the algorithm assigned risk scores to patients on the basis of total health-care costs accrued in one year." "The scientists speculate that this reduced access to care is due to the effects of systemic racism, ranging from distrust of the health-care system to direct racial discrimination by health-care providers." 

Brazil (2019) 5 states: Bahia, Santa Catarina, Paraíba, Rio e Ceará 

Research and Development of Facial recognition (2019) Joy Buolamwini Founder Algorithmic Justice League Deb Raji AI Now Institute Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products https://www.media.mit.edu/publications/actionable-auditing-investigating-the-imp act-of-publicly-naming-biased-performance-results-of-commercial-ai-products/ 

Wrongfully Accused by an Algorithm (2020) 5 estados: Bahia, Santa Catarina, Paraíba, Rio e Ceará https://www.nytimes.com/2020/06/24/technology/facial-rec ognition-arrest.html 

Research and Development of Facial recognition (2020) 

Developer creates and publishes a tool that reinforces white supremacy (2020) https://altdeep.substack.com/p/two-things-you-might-have-missed 

MIT removes dataset (2020) https://www.theregister.com/2020/07/01/mit_dataset_removed/ 

Problem We can talk about a algorithmic society" (O'NEIL, 2016) that incorporates new technology in your everyday lives without having a critical thought about it, seeing only the utility point of view, not thinking about how meritocracy e surveillance are scaled and automated into black-boxes we don't have access. 

Human bias Technology How this happens? 

Is Technology neutral? 

"When the field of AI believes it is neutral, it both fails to notice biased data and builds systems that sanctify the status quo and advance the interests of the powerful. What is needed is a field that exposes and critiques systems that concentrate power, while co-creating new systems with impacted communities: AI by and for the people." Pratyusha Kalluri (PhD Computer Science, Stanford) Is Technology neutral? 

Human bias Technology How to remove bias? 

Human bias Technology How to avoid bias? 

Human bias Technology Diversity 

Gender Gap in Artificial Intelligence “Only 22% of AI professionals globally are female, compared to 78% who are male.” (The Global Gender Gap Report 2018 - p.28) 

“Whether AI will help us reach our aspirations or reinforce the unjust inequalities is ultimately up to us.” Joy Buolamwini 

Even though these decisions affect humans, to optimize task performance ML models often become too complex to be intelligible to humans: black-box models . 

BLACK-BOX INPUT OUTPUT 

BLACK-BOX INPUT OUTPUT ? 

No content

No content

Black-box models? RANDOM FOREST DEEP NEURAL NETWORKS 

How to open the Black-box? 

How to open the Black-box? TRUST EXPLAINABILITY TRANSPARENCY 

Explainable AI (XAI) refers to methods and techniques in the application of artificial intelligence technology (AI) such that the results of the solution can be understood by humans. 39 Explainable AI (XAI)? 

• Interpretability: "systems are interpretable if their operations can be understood by a human, either through introspection or through a produced explanation." (BIRAN, 2017) • Explainability: is the model ability of offering an explanation of its predictions 40 Interpretability x Explainability 

WHITE-BOX MODELS BLACK-BOX EXPLANATION OUTPUT EXPLANATION MODEL EXPLANATION MODEL INSPECTION OPEN BLACK-BOX Explainability methods (GUIDOTTI et al., 2018) 

42 Model Agnostic Methods BLACK-BOX MODEL OUTPUT DATA METHODS EXPLANATIONS "Model-agnostic methods allow explaining predictions of arbitrary machine learning models independent of the implementation." (MENGNAN et al., 2019) 

"Local explanations target to identify the contributions of each feature in the input toward a specific model prediction." (MENGNAN et al., 2019) "Global explanations aims to provide a global understanding about what knowledge has been acquired by these pretrained models and illuminate the parameters or learned representations in an intuitive manner to humans." (MENGNAN et al., 2019) 43 Local x Global explanations 

- Model-Agnostic Methods - Local explanation - LIME focuses on training local surrogate models to explain individual predictions, using data perturbation - Data: tabular data, text, images Python - LIME 44 LIME (RIBEIRO et al., 2016) 

SHAP (SHapley Additive exPlanations) is a game theoretic approach to explain the output of any machine learning model. - Model-Agnostic Methods - Local explanation Python - SHAP 45 SHAP (LUND-BERG e LEE, 2017) 

A counterfactual explanation describes a causal situation in the form: "If A had not occurred, B would not have occurred" In interpretable machine learning, counterfactual explanations can be used to explain predictions of individual instances. 46 Counterfactual Explanations 

http://aix360.mybluemix.net/explanation_cust# Counterfactual Explanations 

http://aix360.mybluemix.net/explanation_cust# Counterfactual Explanations 

http://aix360.mybluemix.net/explanation_cust# Counterfactual Explanations 

- While global understanding is particularly important for assessing trust in a model as a whole (before deployment), most of current research has been devoted to explaining individual predictions. - Ribeiro (2018) believes there is an unexplored opportunity in coming up with explanations that are global in nature. 50 Explainable AI 

“What this new wave of XAI researchers agree on is that if AI systems are to be used by more people, those people must be part of the design from the start—and different people need different kinds of explanations.” 

"When the field of AI believes it is neutral, it both fails to notice biased data and builds systems that sanctify the status quo and advance the interests of the powerful. What is needed is a field that exposes and critiques systems that concentrate power, while co-creating new systems with impacted communities: AI by and for the people." Pratyusha Kalluri (PhD Computer Science, Stanford) 

The choices we are making today about Artificial Intelligence are going to define our future. 

Race After Technology: Abolitionist Tools for the New Jim Code (Ruha Benjamin) From everyday apps to complex algorithms, Ruha Benjamin cuts through tech-industry hype to understand how emerging technologies can reinforce White supremacy and deepen social inequity. Benjamin argues that automation, far from being a sinister story of racist programmers scheming on the dark web, has the potential to hide, speed up, and deepen discrimination while appearing neutral and even benevolent when compared to the racism of a previous era. 

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (Cathy O'Neil) We live in the age of the algorithm. Increasingly, the decisions that affect our lives—where we go to school, whether we can get a job or a loan, how much we pay for health insurance—are being made not by humans, but by machines. In theory, this should lead to greater fairness: Everyone is judged according to the same rules. 

Communities, Algorithms and Digital Activisms: African Diaspora views (Tarcízio Silva) Organized by Tarcízio Silva and published by the LiteraRUA publisher, this book brings together 14 chapters of researchers from Brazil and countries from African Diaspora and Africa, such as Congo, Ethiopia, Ghana, Nigeria, Colombia, United States and United Kingdom. 

Ogunhê Podcast (Ana Carolina da Hora) Ogunhê is the salute to the orisha OGUM: the orisha of war and technology. Ogun, to survive in the forest and in wars, created his weapons (technologies) among other objects to generate changes around him. This project aims to share and present scientists from the African continent and their scientific contributions that help society. 

@carlaprvieira (instagram/twitter) [email protected] | carlavieira.dev youtube.com/EAICarla/ | twitch.tv/carlaprv perifacode.com Thanks! 

− Newsletters − MIT Review – The Download (en) − MIT Review – The Algorithm (en) − Data & Society (en): Data & Society advances public understanding of the social implications of data-centric technologies and automation. − AI Weekly by Khari Johson (en) − Data Hackers (pt): − Desvelar (Tecnologia e Sociedade) (pt): − Podcasts − Artificial Intelligence: AI Podcast by Lex Fridman (en) − Crazy for data (pt) − The Received Wisdom Podcast (en) − 10 TED Talks about AI 

References BARAKAT, N. H.; BRADLEY, A. P. (2007) Rule extraction from support vector machines: Asequential covering approach. IEEE Transactions on Knowledge and Data Engineering. Disponível em: https://doi.org/10.1109/TKDE.2007.190610 Doran, D., Schulz, S.C. e Besold, T. R. 2018. What Does Explainable AI Really Mean? A New Conceptualization of Perspectives. CEUR Workshop Proceedings, 2018. Disponível em: https://arxiv.org/abs/1710.00794 Freitas, A. 2014. Comprehensible classification models: A position paper. Disponível em: https://doi.org/10.1145/2594473.2594475 Guidotti, R., Monreale, A., Ruggieri, S., Turini, F.,Pedreschi, D. e Giannotti, F. 2018. A survey of methods for explaining black box models. Disponível em: https://arxiv.org/abs/1802.01933 Hymas, C. (2019) AI used for first time in job interviews in UK to find best applicants. The Telegraph Disponível em: https://www.telegraph.co.uk/news/2019/09/27/ai-facial-recognition-used-first-time-job-interviews-uk-find/ Ribeiro, M. T., Singh, S. e Guestrin, C. 2016.Model-agnostic interpretability of machine learning,Cornell University. Disponível em: https://arxiv.org/abs/1606.05386 Ledford, H. 2019. Millions of black people affected by racial bias in health-care algorithms. Nature. Disponível em: https://www.nature.com/articles/d41586-019-03228-6 60 

References LUNDBERG, S. M.; ERION, G. G.; CHEN, H.; DEGRAVE, A.; PRUTKIN, J. M.; NAIR,B.; KATZ, R.; HIMMELFARB, J.; BANSAL, N.; LEE, S. 2019. Explainable AI for trees: From local explanations to global understanding. Disponível em:http://arxiv.org/abs/1905.04610 LUNDBERG, S. M.; LEE, S.-I. 2017. A unified approach to interpreting model predictions.In: GUYON, I.; LUXBURG, U. V.; BENGIO, S.; WALLACH, H.; FERGUS, R.;VISHWANATHAN, S.; GARNETT, R. (Ed.). Disponível em:http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf Mengnan, D., Ninghao, L., e Xia, H. 2019. Techniques for interpretable machine learning. Disponível em: https://dl.acm.org/doi/10.1145/3359786 Obermeyer, Z., Powers, B., Vogeli, C. e Mullainathan, S. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Disponível em: https://science.sciencemag.org/content/366/6464/447 O’NEIL, C.Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. USA: Crown Publishing Group, 2016. ISBN 0553418815. Papadopoulos, P. e Walkinshaw, N. 2015. "Black-Box Test Generation from Inferred Models".Disponível em: https://ieeexplore.ieee.org/document/7168327 Vieira, C. P. R.; Digiampietri, L. A. 2020. A study about Explainable Artificial Intelligence: using decision tree to explain SVM. Disponível em: http://seer.upf.br/index.php/rbca/article/view/10247 61