Slide 1

Slide 1 text

Deconstructing Online Hate UX AUSTRALIA 2021 carnagenyc, Hater, https://flic.kr/p/93xwdV

Slide 2

Slide 2 text

DECONSTRUCTING ONLINE HATE Who Am I ▸ 25+ year veteran interaction designer ▸ Currently consulting with Anti-Defamation League’s (ADL) Center for Technology and Society (CTS) ▸ Chair, Interaction Design BFA program at California College of the Arts ▸ Co-author Designing Social Interfaces, 1st & 2nd editions from O’Reilly Media ▸ Design specialties include modeling systems & architectures, designing and advising on social interfaces ▸ Writer ▸ Photographer ▸ Linoleum block printer & letterpress printer ▸ t: @emalone ▸ ig: erin_8pawpress ▸ [email protected]

Slide 3

Slide 3 text

DECONSTRUCTING ONLINE HATE Who is the ADL? ▸ Oldest anti-hate org in world and one of leading 
 civil rights groups in US ▸ Mission: Stop the defamation of the Jewish people and secure justice and fair treatment to all ▸ www.adl.org

Slide 4

Slide 4 text

DECONSTRUCTING ONLINE HATE What is the Center for Technology & Society ▸ ADL actively working with leading companies across Silicon Valley to problem-solve around fighting hate ▸ Actively advising Congress and state elected officials on policy ▸ Tracking & measuring hate online in social platforms and games ▸ Partner with the FairPlay Alliance, a group of over 160 game developers to create a framework for addressing toxic behavior in games (includes definitions, policies, ux design & moderation recommendations) ▸ Building a kinder, gentler Social Patterns interaction design library as a reference for teams battling these issues ▸ https://www.adl.org/who-we-are/ our-organization/advocacy-centers/ center-for-technology-and-society

Slide 5

Slide 5 text

What is the online hate ecosystem? London, England, United Kingdom - June 6th, 2020: Group of people at BLM protest in London by Avel Shah from Noun Project

Slide 6

Slide 6 text

STAKEHOLDER INTERVIEWS

Slide 7

Slide 7 text

How might we visualize DECONSTRUCTING ONLINE HATE

Slide 8

Slide 8 text

2x2 Buckets of categories Bubble Diagram Cluster Model Concept Model Ecosystem Diagram Genealogy of Hate Honeycomb Diagram Inflows & Outflows Journeymap Mindmap Networks Relationships Taxonomy Timeline Venn diagrams

Slide 9

Slide 9 text

Who are the players WHO•WHAT•WHY•WHERE•WHEN•HOW Group of men and women on anti-government protest by Jacob Lund from Noun Project

Slide 10

Slide 10 text

How I ended up down a rathole EXTREMISTS Craig O'Neal from Florida - KKK Rally in Georgia, Wikimedia Commons

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Alt-Right Anti-Abortion Anti-Feminist Anti-Gypsyism Anti-Immigrant Anti-LGBT Anti-Muslim Anti-Semitism Antigovernment Movement Black Nationalist Christian Identity General Hate Hate Music Holocaust Denial Identitarian Movement Islamaphobia Isolationism Ku Klux Klan Male Supremacy Misogyny Militias Nativism Neo-Confederate Neo-Fascim Neo-Nazi Neo-Nationalism Neo Volkisch Paleoconservatism Paleolibertarianism Phineas Priesthood Pro-Life Protectionism Racist Skinhead Radical Right Radical Traditional Catholicism Right Wing Populism Sovereign Citizens 
 Movement White Nationalist White Ethnostate White Separatist White Supremacy Anti-American Indian or Alaska Native Anti-Arab Anti-Asian Anti-Black or African American Anti-Hispanic or Latino Anti-Multiple Races, Group Anti-Native Hawaiian or Other Pacific Islander Anti-Other Race/Ethnicity/Ancestry Anti-White Anti-Buddhist Anti-Catholic Anti-Eastern Orthodox 
 (Russian, Greek, Other) Anti-Hindu Anti-Islamic Anti-Jehovah’s Witness Anti-Jewish Anti-Mormon Anti-Multiple Religions, Group Anti-Other Christian Anti-Other Religion Anti-Protestant Anti-Atheism/Agnosticism/etc. Anti-Bisexual Anti-Gay (Male) Anti-Heterosexual Anti-Lesbian Anti-Lesbian, Gay, Bisexual, or 
 Transgender (Mixed Group) Anti-Mental Disability Anti-Physical Disability Anti-Male Anti-Female Gender Identity Anti-Transgender Anti-Gender Non-Conforming

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

First find the parts BACK TO ONLINE HATE ECOSYSTEM Photo by Mika Baumeister on Unsplash

Slide 20

Slide 20 text

Othering Amplifiers Perpetrators Location Relationships Biases People Organizations Dating Apps Sentiment Victimhood Offline Online Anonymous Incentives Section 230 Dehumanization US vs Europe Privacy Data Ideologies Hate Groups Propaganda Mitigators Inflows & Outflows Money Social Platforms Governments Online Games Liability Infrastructure Community Narratives 1st Amendment Conspiracy Theories Non-Profits Schools Misinformation Memes Trolls Mainstream Fringe Extremists Academics Activists Alt-Right NGOs Targets Harassment Discrimination Conspiracy Theories Discrimination Religion Gender Sex Immigrant Anger Fear Ability Rejection Marginalized Marginalized Research Silenced Racism Misogyny Abuse Doxing Violence Death Genocide Stress Education Counter-Speech Players Patriotism Intelligence Hate Manosphere Bigotry Race Stalking Brainwashing Toxic Threats Stereotyping Slurs Ridicule Women LGBTQ+ Reputation Identity Virality Engagement RedPill

Slide 21

Slide 21 text

Othering Amplifiers Perpetrators Location Relationships Biases People Organizations Dating Apps Sentiment Victimhood Offline Online Anonymous Incentives Section 230 Dehumanization US vs Europe Privacy Data Ideologies Hate Groups Propaganda Mitigators Inflows & Outflows Money Social Platforms Governments Online Games Liability Infrastructure Community Narratives 1st Amendment Conspiracy Theories Non-Profits Schools Misinformation Memes Trolls Mainstream Fringe Extremists Academics Activists Alt-Right NGOs Targets Harassment Discrimination Conspiracy Theories Discrimination Religion Gender Sex Immigrant Anger Fear Ability Rejection Marginalized Marginalized Research Silenced Racism Misogyny Abuse Doxing Violence Death Genocide Stress Education Counter-Speech Players Patriotism Intelligence Hate Manosphere Bigotry Race Stalking Brainwashing Toxic Threats Stereotyping Slurs Ridicule Women LGBTQ+ Reputation Identity Virality Engagement RedPill

Slide 22

Slide 22 text

Robert Thivierge, Aryan Guard in Kensington 2, https://flic.kr/p/2UMZUh Ideologies & Biases

Slide 23

Slide 23 text

Large group of people at BLM protest at the Lincoln Memorial, Washington, D.C. by Carol M. Highsmith from Noun Project Race/Ethnicity/ Ancestry Anti-American Indian or Alaska Native Anti-Arab Anti-Asian Anti-Black or African American Anti-Hispanic or Latino Anti-Multiple Races, Group Anti-Native Hawaiian or Other Pacific Islander Anti-Other Race/ Ethnicity/Ancestry Anti-White Religion Anti-Buddhist Anti-Catholic Anti-Eastern Orthodox (Russian, Greek, Other) Anti-Hindu Anti-Islamic Anti-Jehovah’s Witness Anti-Jewish Anti-Mormon Anti-Multiple Religions, Group Anti-Other Christian Anti-Other Religion Anti-Protestant Anti-Atheism/ Agnosticism/etc. Sexual Orientation Anti-Bisexual Anti-Gay (Male) Anti-Heterosexual Anti-Lesbian Anti-Lesbian, Gay, Bisexual, or Transgender (Mixed Group) Disability Anti-Mental Disability Anti-Physical Disability Gender Anti-Male Anti-Female Gender Identity Anti-Transgender Anti-Gender Non- Conforming FBI LIST

Slide 24

Slide 24 text

carnagenyc, Hater, https://flic.kr/p/93xwdV Ageism Ableism Anti-LGBTQ+ Conspiracy Theories Misogyny Nativism Racism Religious Bigotry Xenophobia White Supremacy Hate Groups Individuals Institutions Governments Media Who What

Slide 25

Slide 25 text

Janusz Kaliszczak, 2018-07-14-AntiFascist-005, https://flic.kr/p/M161hm Why Bullied Lashing Out Fear Victim of Hate Looking for and Finding Community Religious Doctrine Radicalized Suffered Abuse Rejection Blaming Others Pain Masquerading as Hate Protection Violent Childhood Mistrust of Other Belief in Stereotypes Learned Cultural Biases Feeling Threatened War

Slide 26

Slide 26 text

Biases & Hate Ideologies Perpetrators

Slide 27

Slide 27 text

Why are people targeted? London, England, United Kingdom - July 7th, 2019: Revelers pictured atop the Pride Bus in London by Avel Shah from Noun Project

Slide 28

Slide 28 text

Photo by Crawford Jolly on Unsplash Racist Hate Speech Death Threats Sexist Hate Speech Discrimination Dehumanization Doxxing Insults Blocking Play Othering Rape Threats Harassment Identity Impersonation Non-consensual Pornography Deepfakes Swatting Bullying Throwing the Game Mobbing Overt Negativity Digital Voyeurism Cyber Stalking Race Religion Gender Gender Identity Sexual Orientation Age National Origin Ethnicity Ability Profession Why How

Slide 29

Slide 29 text

super awesome, riot, https://flic.kr/p/9fNVGx Outcomes Apathy Discrimination Suicides Ethnic Cleansing Silenced Voices Lost Reputation Lost Income Undermined Elections Radicalization Offline Attacks Escalation & Retaliation

Slide 30

Slide 30 text

Targets Behaviors Aga inst Targets

Slide 31

Slide 31 text

Woman taking selfie with her friend for her social media post by Jacob Lund from Noun Project Where is this happening?

Slide 32

Slide 32 text

Photo by Thais Do Rio on Unsplash Arcade Games Role Playing Games Collaborative Games Action & Adventure Games 1st Person Shooter Games Sports Massively Multiplayer Simulations Photo Sharing Videos Live Streaming Podcasting Blogging Chat Messaging Email Message Boards Groups Collecting Events Professional Traditional Media Online Radio Online Games Social Platforms & Apps Dating Apps Media Where Types

Slide 33

Slide 33 text

Three friends hanging out and playing video games by Jacob Lund from Noun Project Activities Live Streams Text Chats Voice Chats Comments Connections Conversations Comments Hashtags Text Photos

Slide 34

Slide 34 text

Amplifiers Activities That Am plify Hate Speech

Slide 35

Slide 35 text

Photo by Sushil Nash on Unsplash Who is working to stop online hate?

Slide 36

Slide 36 text

Community Guidelines Terms of Service Moderators Best Practices Artificial Intelligence Algorithms Product Goals Ethics Principles Designers Journalists Editors Publishers Researchers Academics Authors Activists Youth Groups Schools Teachers & Parents Civil Rights Feminist Policy Immigration Human Rights Discrimination International Federal State Local Social Platforms Online Games Dating Apps Media Individuals NGOs & Non-Profits Governments Who How Photo by Matt Popovich on Unsplash

Slide 37

Slide 37 text

Outputs Rewarding Desired Behavior User Reputation Light Friction Built-in User Tools Algorithmic Filtering Moderator Tools Celebrity Influence Articles Research Papers Books Fact Checking Counter Speech Market Reform Campaigns Workshops Policies Laws Photo by Anna Sullivan on Unsplash

Slide 38

Slide 38 text

Mitigators Mitigator Tactics & Tools

Slide 39

Slide 39 text

Mitigator Tactics & Tools OUTP UTS Activities That Am plify Hate Speech ACTIV ITIES Biases & Hate Ideologies W H Y Behaviors Aga inst Targets OUTC OM ES Hate Groups Media Governments Individuals Institutions Race Death Threats Racist Hate Speech Anti-LGBTQ+ Racism Misogyny Sexist Hate Speech Insults Blocking Play Othering Dehumanization Doxing Rape Threats Harassm ent Identity Im personation Non-consensual Pornography Deepfakes Threats of Violence Swatting Bullying Throwing the Game Mobbing Overt Negativity Digital Voyeurism Stereotyping Gender National Origin Ethnicity Ability Social Platforms & Games Individuals NGOs & Non-Profits Governments Media Social Platform s & Social Apps Online Games Nativism Xenophobia Ageism Religious Bigotry Conspiracy Theories Ableism Religion Traditional Media Blogs Podcasts Online Radio VLogs Academ ics Activists Journalists Civil Rights Feminist Policy Immigration Human Rights Discrimination International Federal State Local Community Guidelines Best Practices Moderators Terms of Service Researchers Artifical Intelligence Algorithms Cyber Stalking Discrimination Photo Sharing Videos Podcasting Live Streaming Blogging Collecting Dating Professional Events M essage Boards M essaging Em ail Groups Chat Perpetrators M itigators Targets A m plifiers Live Streams Text Chats Voice Chats Comments Role Playing Arcade Collaborative Action & Adventure 1st Person Shooter Sports Massively Multiplayer Simulation Youth Groups Schools Teachers & Parents Conversations Photos Com m ents Hashtags Audio Video Text Connections Designers Product Goals Age Market Reform Campaigns User Tools Algorithmic Filtering User Reputation Light Friction Built-in Rewarding Desired Behavior Moderator Tools Counter Speech Celebrity Influence Laws Policies Workshops Research Papers Fact Checking Silenced Voices Radicalization Undermined Elections Ethnic Cleansing Suicides Offline Attacks Escalation & Retaliation Lost Reputation Lost Income Radicalized Suffered Abuse Rejection Bullied Looking for and Finding Community Lashing Out Protection Mistrust of Other Violent Childhood Belief in Stereotypes Learned Cultural Biases Feeling Threatened Victim of Hate Religious Doctrine Pain m asquerading as Hate War Fear Blam ing Others Discrimination Apathy Online Hate Media Editors Authors Publishers Gender Identity Sexual Orientation Profession Articles White Supremacy

Slide 40

Slide 40 text

Understanding the relationships ONLINE HATE ECOSYSTEM Photo by Diego Lozano on Unsplash

Slide 41

Slide 41 text

ONLINE HATE ECOSYSTEM - THE RELATIONSHIPS What are the relationships between the parts? What paths can we see connecting them? Where do we see perpetrators in the system? Where do we see mitigators the system? How do these all interconnect? Photo by Nitish Meena on Unsplash

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Follow the money ONLINE HATE ECOSYSTEM Photo by Diego Lozano on Unsplash

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

Anthony Crider, CC BY 2.0 , via Wikimedia Commons Escalation path A POTENTIAL

Slide 59

Slide 59 text

ONLINE HATE ECOSYSTEM - A POTENTIAL ESCALATION PATH What is a single person’s experience? Where do they get immersed? Where do bad actors target them? Where might we find opportunities for intervention? Where can platforms intervene or suppress hate content? Who should be intervening and disrupting the cycle? How can design intervene? Photo by Clay Banks on Unsplash

Slide 60

Slide 60 text

A straight path of negative possibility

Slide 61

Slide 61 text

and consumes which in fluences and continues Influences, amplification & confirmation biases reinforce the path

Slide 62

Slide 62 text

Interventions & places of disruption invite discussion around programs, education and other initiatives. and consumes which in fluences and continues

Slide 63

Slide 63 text

Platforms can interrupt & suppress before hate is posted and consumes which in fluences and continues

Slide 64

Slide 64 text

Research HOW DID I GET HERE Photo by Gayatri Malhotra on Unsplash

Slide 65

Slide 65 text

UNDERSTANDING BY THE NUMBERS - PRIMARY & SECONDARY RESEARCH carnagenyc, Hater, https://flic.kr/p/93xwdV Mainstream articles, NGO reports, academic papers & expert authored books read 300+ NGO and Non-profit websites, reports, articles and missions reviewed 50+ Left-leaning & mainstream online magazines, news sites & blogs 19+ & a handful of Alt-Right media sites

Slide 66

Slide 66 text

UNDERSTANDING BY THE NUMBERS - PRIMARY & SECONDARY RESEARCH Janusz Kaliszczak, 2018-07-14-AntiFascist-005, https://flic.kr/p/M161hm External feedback and validation reviewer sessions 17+ Companies, disciplines and expertises represented in reviewer & feedback sessions 15 ADL Stakeholder & Academic interviews to start the research process 17+

Slide 67

Slide 67 text

Questions STILL TO ANSWER Photo by Gayatri Malhotra on Unsplash

Slide 68

Slide 68 text

How do games differ than social platforms? How should our approach differ? QUESTIONS

Slide 69

Slide 69 text

How can we manipulate “engagement” and “virality” for good (kinder) outcomes? QUESTIONS

Slide 70

Slide 70 text

How can we continue to encourage connections but design better guardrails? 
 How much is too much? QUESTIONS

Slide 71

Slide 71 text

How do we know if our design intentions make positive change? QUESTIONS

Slide 72

Slide 72 text

How can understanding the ecosystem help “design” improve in small places for big impact? QUESTIONS

Slide 73

Slide 73 text

Where are the most salient opportunities for change? Where are there time delays? QUESTIONS

Slide 74

Slide 74 text

What policies, laws & legislation should I understand in my role as a designer? QUESTIONS

Slide 75

Slide 75 text

What is our responsibility for pushing back — for keeping kindness & community at the forefront over profits? QUESTIONS

Slide 76

Slide 76 text

Thank You ▸ t: @emalone ▸ ig: erin_8pawpress ▸ www.experiencemattersdesign.com Photo by Gayatri Malhotra on Unsplash Diagrams: https://www.adl.org/online-hate-ecosystem-primer

Slide 77

Slide 77 text

Questions? Group of protestors with their fists raised up in the air by Jacob Lund from Noun Project

Slide 78

Slide 78 text

REFERENCES Royalty Free & Creative Commons Images from: ▸ London, England, United Kingdom - June 6th, 2020: Group of people at BLM protest in London by Avel Shah from Noun Project ▸ Anti-government protest by the social activist group by Jacob Lund from Noun Project ▸ Group of men and women on anti-government protest by Jacob Lund from Noun Project ▸ Large group of people at BLM protest at the Lincoln Memorial, Washington, D.C. by Carol M. Highsmith from Noun Project ▸ Three friends hanging out and playing video games by Jacob Lund from Noun Project ▸ Woman taking selfie with her friend for her social media post by Jacob Lund from Noun Project ▸ London, England, United Kingdom - July 7th, 2019: Revelers pictured atop the Pride Bus in London by Avel Shah from Noun Project ▸ Robert Thivierge, Aryan Guard in Kensington 2, https://flic.kr/p/2UMZUh ▸ Janusz Kaliszczak, 2018-07-14-AntiFascist-005, https://flic.kr/p/M161hm ▸ carnagenyc, Hater, https://flic.kr/p/93xwdV ▸ super awesome, riot, https://flic.kr/p/9fNVGx ▸ Newspaper boxes, photo by Matt Popovich on Unsplash ▸ Courthouse, Photo by Anna Sullivan on Unsplash ▸ Black Lives Matter Signs, Photo by Diego Lozano on Unsplash ▸ Black Lives Matters protesters, Photo by Sushil Nash on Unsplash ▸ Stalking mannequins, Photo by Mika Baumeister on Unsplash ▸ Southern confederate protesters, Anthony Crider, CC BY 2.0 , via Wikimedia Commons ▸ White Pride, Robert Thivierge, Aryan Guard in Kensington 2, https://flic.kr/p/2UMZUh ▸ KKK Rally in Georgia, Craig O'Neal from Florida, CC BY 2.0 , via Wikimedia Commons ▸ Anton Fredrikson / Keski-Suomen museo, Public domain, via Wikimedia Commons ▸ Activists, Photo by Nitish Meena on Unsplash ▸ Black Lives Matters protesters & activists, Photo by Clay Banks on Unsplash ▸ Black Lives Matters protesters, Photo by Gayatri Malhotra on Unsplash ▸ Group of protestors with their fists raised up in the air by Jacob Lund from Noun Project Anton Fredrikson / Keski-Suomen museo, Public domain, via Wikimedia Commons Typography - Font design: www.vocaltype.co - Headings Martin, Subheads/body VTC Carrie, Body text Avenir Next Condensed