Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Erin malone - Mapping Hate - Deconstructing the Ecosystem

Erin malone - Mapping Hate - Deconstructing the Ecosystem

UX Australia 2021 DAY 4

What does the landscape of online hate look like and how do you visualize it? Join Erin Malone as she talks about her project with ADL’s Center for Technology and Society to map the ecosystems of hate in digital social spaces.

Bff39f7d4e25a30fb4d28a9005de7363?s=128

uxaustralia
PRO

August 27, 2021
Tweet

Transcript

  1. Deconstructing Online Hate UX AUSTRALIA 2021 carnagenyc, Hater, https://flic.kr/p/93xwdV

  2. DECONSTRUCTING ONLINE HATE Who Am I ▸ 25+ year veteran

    interaction designer ▸ Currently consulting with Anti-Defamation League’s (ADL) Center for Technology and Society (CTS) ▸ Chair, Interaction Design BFA program at California College of the Arts ▸ Co-author Designing Social Interfaces, 1st & 2nd editions from O’Reilly Media ▸ Design specialties include modeling systems & architectures, designing and advising on social interfaces ▸ Writer ▸ Photographer ▸ Linoleum block printer & letterpress printer ▸ t: @emalone ▸ ig: erin_8pawpress ▸ erin@experiencemattersdesign.com
  3. DECONSTRUCTING ONLINE HATE Who is the ADL? ▸ Oldest anti-hate

    org in world and one of leading 
 civil rights groups in US ▸ Mission: Stop the defamation of the Jewish people and secure justice and fair treatment to all ▸ www.adl.org
  4. DECONSTRUCTING ONLINE HATE What is the Center for Technology &

    Society ▸ ADL actively working with leading companies across Silicon Valley to problem-solve around fighting hate ▸ Actively advising Congress and state elected officials on policy ▸ Tracking & measuring hate online in social platforms and games ▸ Partner with the FairPlay Alliance, a group of over 160 game developers to create a framework for addressing toxic behavior in games (includes definitions, policies, ux design & moderation recommendations) ▸ Building a kinder, gentler Social Patterns interaction design library as a reference for teams battling these issues ▸ https://www.adl.org/who-we-are/ our-organization/advocacy-centers/ center-for-technology-and-society
  5. What is the online hate ecosystem? London, England, United Kingdom

    - June 6th, 2020: Group of people at BLM protest in London by Avel Shah from Noun Project
  6. STAKEHOLDER INTERVIEWS

  7. How might we visualize DECONSTRUCTING ONLINE HATE

  8. 2x2 Buckets of categories Bubble Diagram Cluster Model Concept Model

    Ecosystem Diagram Genealogy of Hate Honeycomb Diagram Inflows & Outflows Journeymap Mindmap Networks Relationships Taxonomy Timeline Venn diagrams
  9. Who are the players WHO•WHAT•WHY•WHERE•WHEN•HOW Group of men and women

    on anti-government protest by Jacob Lund from Noun Project
  10. How I ended up down a rathole EXTREMISTS Craig O'Neal

    from Florida - KKK Rally in Georgia, Wikimedia Commons
  11. None
  12. None
  13. Alt-Right Anti-Abortion Anti-Feminist Anti-Gypsyism Anti-Immigrant Anti-LGBT Anti-Muslim Anti-Semitism Antigovernment Movement

    Black Nationalist Christian Identity General Hate Hate Music Holocaust Denial Identitarian Movement Islamaphobia Isolationism Ku Klux Klan Male Supremacy Misogyny Militias Nativism Neo-Confederate Neo-Fascim Neo-Nazi Neo-Nationalism Neo Volkisch Paleoconservatism Paleolibertarianism Phineas Priesthood Pro-Life Protectionism Racist Skinhead Radical Right Radical Traditional Catholicism Right Wing Populism Sovereign Citizens 
 Movement White Nationalist White Ethnostate White Separatist White Supremacy Anti-American Indian or Alaska Native Anti-Arab Anti-Asian Anti-Black or African American Anti-Hispanic or Latino Anti-Multiple Races, Group Anti-Native Hawaiian or Other Pacific Islander Anti-Other Race/Ethnicity/Ancestry Anti-White Anti-Buddhist Anti-Catholic Anti-Eastern Orthodox 
 (Russian, Greek, Other) Anti-Hindu Anti-Islamic Anti-Jehovah’s Witness Anti-Jewish Anti-Mormon Anti-Multiple Religions, Group Anti-Other Christian Anti-Other Religion Anti-Protestant Anti-Atheism/Agnosticism/etc. Anti-Bisexual Anti-Gay (Male) Anti-Heterosexual Anti-Lesbian Anti-Lesbian, Gay, Bisexual, or 
 Transgender (Mixed Group) Anti-Mental Disability Anti-Physical Disability Anti-Male Anti-Female Gender Identity Anti-Transgender Anti-Gender Non-Conforming
  14. None
  15. None
  16. None
  17. None
  18. None
  19. First find the parts BACK TO ONLINE HATE ECOSYSTEM Photo

    by Mika Baumeister on Unsplash
  20. Othering Amplifiers Perpetrators Location Relationships Biases People Organizations Dating Apps

    Sentiment Victimhood Offline Online Anonymous Incentives Section 230 Dehumanization US vs Europe Privacy Data Ideologies Hate Groups Propaganda Mitigators Inflows & Outflows Money Social Platforms Governments Online Games Liability Infrastructure Community Narratives 1st Amendment Conspiracy Theories Non-Profits Schools Misinformation Memes Trolls Mainstream Fringe Extremists Academics Activists Alt-Right NGOs Targets Harassment Discrimination Conspiracy Theories Discrimination Religion Gender Sex Immigrant Anger Fear Ability Rejection Marginalized Marginalized Research Silenced Racism Misogyny Abuse Doxing Violence Death Genocide Stress Education Counter-Speech Players Patriotism Intelligence Hate Manosphere Bigotry Race Stalking Brainwashing Toxic Threats Stereotyping Slurs Ridicule Women LGBTQ+ Reputation Identity Virality Engagement RedPill
  21. Othering Amplifiers Perpetrators Location Relationships Biases People Organizations Dating Apps

    Sentiment Victimhood Offline Online Anonymous Incentives Section 230 Dehumanization US vs Europe Privacy Data Ideologies Hate Groups Propaganda Mitigators Inflows & Outflows Money Social Platforms Governments Online Games Liability Infrastructure Community Narratives 1st Amendment Conspiracy Theories Non-Profits Schools Misinformation Memes Trolls Mainstream Fringe Extremists Academics Activists Alt-Right NGOs Targets Harassment Discrimination Conspiracy Theories Discrimination Religion Gender Sex Immigrant Anger Fear Ability Rejection Marginalized Marginalized Research Silenced Racism Misogyny Abuse Doxing Violence Death Genocide Stress Education Counter-Speech Players Patriotism Intelligence Hate Manosphere Bigotry Race Stalking Brainwashing Toxic Threats Stereotyping Slurs Ridicule Women LGBTQ+ Reputation Identity Virality Engagement RedPill
  22. Robert Thivierge, Aryan Guard in Kensington 2, https://flic.kr/p/2UMZUh Ideologies &

    Biases
  23. Large group of people at BLM protest at the Lincoln

    Memorial, Washington, D.C. by Carol M. Highsmith from Noun Project Race/Ethnicity/ Ancestry Anti-American Indian or Alaska Native Anti-Arab Anti-Asian Anti-Black or African American Anti-Hispanic or Latino Anti-Multiple Races, Group Anti-Native Hawaiian or Other Pacific Islander Anti-Other Race/ Ethnicity/Ancestry Anti-White Religion Anti-Buddhist Anti-Catholic Anti-Eastern Orthodox (Russian, Greek, Other) Anti-Hindu Anti-Islamic Anti-Jehovah’s Witness Anti-Jewish Anti-Mormon Anti-Multiple Religions, Group Anti-Other Christian Anti-Other Religion Anti-Protestant Anti-Atheism/ Agnosticism/etc. Sexual Orientation Anti-Bisexual Anti-Gay (Male) Anti-Heterosexual Anti-Lesbian Anti-Lesbian, Gay, Bisexual, or Transgender (Mixed Group) Disability Anti-Mental Disability Anti-Physical Disability Gender Anti-Male Anti-Female Gender Identity Anti-Transgender Anti-Gender Non- Conforming FBI LIST
  24. carnagenyc, Hater, https://flic.kr/p/93xwdV Ageism Ableism Anti-LGBTQ+ Conspiracy Theories Misogyny Nativism

    Racism Religious Bigotry Xenophobia White Supremacy Hate Groups Individuals Institutions Governments Media Who What
  25. Janusz Kaliszczak, 2018-07-14-AntiFascist-005, https://flic.kr/p/M161hm Why Bullied Lashing Out Fear Victim

    of Hate Looking for and Finding Community Religious Doctrine Radicalized Suffered Abuse Rejection Blaming Others Pain Masquerading as Hate Protection Violent Childhood Mistrust of Other Belief in Stereotypes Learned Cultural Biases Feeling Threatened War
  26. Biases & Hate Ideologies Perpetrators

  27. Why are people targeted? London, England, United Kingdom - July

    7th, 2019: Revelers pictured atop the Pride Bus in London by Avel Shah from Noun Project
  28. Photo by Crawford Jolly on Unsplash Racist Hate Speech Death

    Threats Sexist Hate Speech Discrimination Dehumanization Doxxing Insults Blocking Play Othering Rape Threats Harassment Identity Impersonation Non-consensual Pornography Deepfakes Swatting Bullying Throwing the Game Mobbing Overt Negativity Digital Voyeurism Cyber Stalking Race Religion Gender Gender Identity Sexual Orientation Age National Origin Ethnicity Ability Profession Why How
  29. super awesome, riot, https://flic.kr/p/9fNVGx Outcomes Apathy Discrimination Suicides Ethnic Cleansing

    Silenced Voices Lost Reputation Lost Income Undermined Elections Radicalization Offline Attacks Escalation & Retaliation
  30. Targets Behaviors Aga inst Targets

  31. Woman taking selfie with her friend for her social media

    post by Jacob Lund from Noun Project Where is this happening?
  32. Photo by Thais Do Rio on Unsplash Arcade Games Role

    Playing Games Collaborative Games Action & Adventure Games 1st Person Shooter Games Sports Massively Multiplayer Simulations Photo Sharing Videos Live Streaming Podcasting Blogging Chat Messaging Email Message Boards Groups Collecting Events Professional Traditional Media Online Radio Online Games Social Platforms & Apps Dating Apps Media Where Types
  33. Three friends hanging out and playing video games by Jacob

    Lund from Noun Project Activities Live Streams Text Chats Voice Chats Comments Connections Conversations Comments Hashtags Text Photos
  34. Amplifiers Activities That Am plify Hate Speech

  35. Photo by Sushil Nash on Unsplash Who is working to

    stop online hate?
  36. Community Guidelines Terms of Service Moderators Best Practices Artificial Intelligence

    Algorithms Product Goals Ethics Principles Designers Journalists Editors Publishers Researchers Academics Authors Activists Youth Groups Schools Teachers & Parents Civil Rights Feminist Policy Immigration Human Rights Discrimination International Federal State Local Social Platforms Online Games Dating Apps Media Individuals NGOs & Non-Profits Governments Who How Photo by Matt Popovich on Unsplash
  37. Outputs Rewarding Desired Behavior User Reputation Light Friction Built-in User

    Tools Algorithmic Filtering Moderator Tools Celebrity Influence Articles Research Papers Books Fact Checking Counter Speech Market Reform Campaigns Workshops Policies Laws Photo by Anna Sullivan on Unsplash
  38. Mitigators Mitigator Tactics & Tools

  39. Mitigator Tactics & Tools OUTP UTS Activities That Am plify

    Hate Speech ACTIV ITIES Biases & Hate Ideologies W H Y Behaviors Aga inst Targets OUTC OM ES Hate Groups Media Governments Individuals Institutions Race Death Threats Racist Hate Speech Anti-LGBTQ+ Racism Misogyny Sexist Hate Speech Insults Blocking Play Othering Dehumanization Doxing Rape Threats Harassm ent Identity Im personation Non-consensual Pornography Deepfakes Threats of Violence Swatting Bullying Throwing the Game Mobbing Overt Negativity Digital Voyeurism Stereotyping Gender National Origin Ethnicity Ability Social Platforms & Games Individuals NGOs & Non-Profits Governments Media Social Platform s & Social Apps Online Games Nativism Xenophobia Ageism Religious Bigotry Conspiracy Theories Ableism Religion Traditional Media Blogs Podcasts Online Radio VLogs Academ ics Activists Journalists Civil Rights Feminist Policy Immigration Human Rights Discrimination International Federal State Local Community Guidelines Best Practices Moderators Terms of Service Researchers Artifical Intelligence Algorithms Cyber Stalking Discrimination Photo Sharing Videos Podcasting Live Streaming Blogging Collecting Dating Professional Events M essage Boards M essaging Em ail Groups Chat Perpetrators M itigators Targets A m plifiers Live Streams Text Chats Voice Chats Comments Role Playing Arcade Collaborative Action & Adventure 1st Person Shooter Sports Massively Multiplayer Simulation Youth Groups Schools Teachers & Parents Conversations Photos Com m ents Hashtags Audio Video Text Connections Designers Product Goals Age Market Reform Campaigns User Tools Algorithmic Filtering User Reputation Light Friction Built-in Rewarding Desired Behavior Moderator Tools Counter Speech Celebrity Influence Laws Policies Workshops Research Papers Fact Checking Silenced Voices Radicalization Undermined Elections Ethnic Cleansing Suicides Offline Attacks Escalation & Retaliation Lost Reputation Lost Income Radicalized Suffered Abuse Rejection Bullied Looking for and Finding Community Lashing Out Protection Mistrust of Other Violent Childhood Belief in Stereotypes Learned Cultural Biases Feeling Threatened Victim of Hate Religious Doctrine Pain m asquerading as Hate War Fear Blam ing Others Discrimination Apathy Online Hate Media Editors Authors Publishers Gender Identity Sexual Orientation Profession Articles White Supremacy
  40. Understanding the relationships ONLINE HATE ECOSYSTEM Photo by Diego Lozano

    on Unsplash
  41. ONLINE HATE ECOSYSTEM - THE RELATIONSHIPS What are the relationships

    between the parts? What paths can we see connecting them? Where do we see perpetrators in the system? Where do we see mitigators the system? How do these all interconnect? Photo by Nitish Meena on Unsplash
  42. None
  43. None
  44. None
  45. None
  46. None
  47. None
  48. None
  49. None
  50. None
  51. None
  52. None
  53. None
  54. Follow the money ONLINE HATE ECOSYSTEM Photo by Diego Lozano

    on Unsplash
  55. None
  56. None
  57. None
  58. Anthony Crider, CC BY 2.0 , via Wikimedia Commons Escalation

    path A POTENTIAL
  59. ONLINE HATE ECOSYSTEM - A POTENTIAL ESCALATION PATH What is

    a single person’s experience? Where do they get immersed? Where do bad actors target them? Where might we find opportunities for intervention? Where can platforms intervene or suppress hate content? Who should be intervening and disrupting the cycle? How can design intervene? Photo by Clay Banks on Unsplash
  60. A straight path of negative possibility

  61. and consumes which in fluences and continues Influences, amplification &

    confirmation biases reinforce the path
  62. Interventions & places of disruption invite discussion around programs, education

    and other initiatives. and consumes which in fluences and continues
  63. Platforms can interrupt & suppress before hate is posted and

    consumes which in fluences and continues
  64. Research HOW DID I GET HERE Photo by Gayatri Malhotra

    on Unsplash
  65. UNDERSTANDING BY THE NUMBERS - PRIMARY & SECONDARY RESEARCH carnagenyc,

    Hater, https://flic.kr/p/93xwdV Mainstream articles, NGO reports, academic papers & expert authored books read 300+ NGO and Non-profit websites, reports, articles and missions reviewed 50+ Left-leaning & mainstream online magazines, news sites & blogs 19+ & a handful of Alt-Right media sites
  66. UNDERSTANDING BY THE NUMBERS - PRIMARY & SECONDARY RESEARCH Janusz

    Kaliszczak, 2018-07-14-AntiFascist-005, https://flic.kr/p/M161hm External feedback and validation reviewer sessions 17+ Companies, disciplines and expertises represented in reviewer & feedback sessions 15 ADL Stakeholder & Academic interviews to start the research process 17+
  67. Questions STILL TO ANSWER Photo by Gayatri Malhotra on Unsplash

  68. How do games differ than social platforms? How should our

    approach differ? QUESTIONS
  69. How can we manipulate “engagement” and “virality” for good (kinder)

    outcomes? QUESTIONS
  70. How can we continue to encourage connections but design better

    guardrails? 
 How much is too much? QUESTIONS
  71. How do we know if our design intentions make positive

    change? QUESTIONS
  72. How can understanding the ecosystem help “design” improve in small

    places for big impact? QUESTIONS
  73. Where are the most salient opportunities for change? Where are

    there time delays? QUESTIONS
  74. What policies, laws & legislation should I understand in my

    role as a designer? QUESTIONS
  75. What is our responsibility for pushing back — for keeping

    kindness & community at the forefront over profits? QUESTIONS
  76. Thank You ▸ t: @emalone ▸ ig: erin_8pawpress ▸ www.experiencemattersdesign.com

    Photo by Gayatri Malhotra on Unsplash Diagrams: https://www.adl.org/online-hate-ecosystem-primer
  77. Questions? Group of protestors with their fists raised up in

    the air by Jacob Lund from Noun Project
  78. REFERENCES Royalty Free & Creative Commons Images from: ▸ London,

    England, United Kingdom - June 6th, 2020: Group of people at BLM protest in London by Avel Shah from Noun Project ▸ Anti-government protest by the social activist group by Jacob Lund from Noun Project ▸ Group of men and women on anti-government protest by Jacob Lund from Noun Project ▸ Large group of people at BLM protest at the Lincoln Memorial, Washington, D.C. by Carol M. Highsmith from Noun Project ▸ Three friends hanging out and playing video games by Jacob Lund from Noun Project ▸ Woman taking selfie with her friend for her social media post by Jacob Lund from Noun Project ▸ London, England, United Kingdom - July 7th, 2019: Revelers pictured atop the Pride Bus in London by Avel Shah from Noun Project ▸ Robert Thivierge, Aryan Guard in Kensington 2, https://flic.kr/p/2UMZUh ▸ Janusz Kaliszczak, 2018-07-14-AntiFascist-005, https://flic.kr/p/M161hm ▸ carnagenyc, Hater, https://flic.kr/p/93xwdV ▸ super awesome, riot, https://flic.kr/p/9fNVGx ▸ Newspaper boxes, photo by Matt Popovich on Unsplash ▸ Courthouse, Photo by Anna Sullivan on Unsplash ▸ Black Lives Matter Signs, Photo by Diego Lozano on Unsplash ▸ Black Lives Matters protesters, Photo by Sushil Nash on Unsplash ▸ Stalking mannequins, Photo by Mika Baumeister on Unsplash ▸ Southern confederate protesters, Anthony Crider, CC BY 2.0 , via Wikimedia Commons ▸ White Pride, Robert Thivierge, Aryan Guard in Kensington 2, https://flic.kr/p/2UMZUh ▸ KKK Rally in Georgia, Craig O'Neal from Florida, CC BY 2.0 , via Wikimedia Commons ▸ Anton Fredrikson / Keski-Suomen museo, Public domain, via Wikimedia Commons ▸ Activists, Photo by Nitish Meena on Unsplash ▸ Black Lives Matters protesters & activists, Photo by Clay Banks on Unsplash ▸ Black Lives Matters protesters, Photo by Gayatri Malhotra on Unsplash ▸ Group of protestors with their fists raised up in the air by Jacob Lund from Noun Project Anton Fredrikson / Keski-Suomen museo, Public domain, via Wikimedia Commons Typography - Font design: www.vocaltype.co - Headings Martin, Subheads/body VTC Carrie, Body text Avenir Next Condensed