Consensual Software - QCon London 2017

Online safety has become a huge problem in the world of oversharing. Real name policies, automatic geolocation tracking, and photo tagging increase user adoption rates, but these features can be quickly abused by bad actors. This talk will discuss how to apply a "consent filter" to product decisions to make a safer user experience and to help protect your most vulnerable users from harm.
https://qconlondon.com/london-2017/speakers/danielle-leong

Transcript

1. @tsunamino ! " Consensual Software: Prioritizing Trust & Safety
2. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 2 @tsunamino http://www.wocintechchat.com/blog/wocintechphotos
3. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 3 @tsunamino http://www.wocintechchat.com/blog/wocintechphotos
4. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 4 @tsunamino http://www.wocintechchat.com/blog/wocintechphotos
5. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 5 @tsunamino http://www.wocintechchat.com/blog/wocintechphotos
6. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 6 @tsunamino http://www.wocintechchat.com/blog/wocintechphotos
7. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 7 @tsunamino
8. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 8 @tsunamino http://www.wocintechchat.com/blog/wocintechphotos
9. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 9 @tsunamino
10. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 10 @tsunamino http://www.wocintechchat.com/blog/wocintechphotos
11. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 11 @tsunamino http://www.wocintechchat.com/blog/wocintechphotos
12. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 12
13. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 13 @tsunamino http://www.wocintechchat.com/blog/wocintechphotos
14. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 14 @tsunamino
15. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 15
16. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 16 Danielle Leong Engineer on GitHub’s Community & Safety team @tsunamino
17. Title Text @tsunamino
18. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 18 GitHub is a web-based version-control and collaboration platform for software developers. @tsunamino
19. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 19 Collaboration in text form is hard.
20. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 20 Build systems that empower inclusive and healthy communities and encourage good online citizenship. Discourage behavior that is destructive or threatens personal safety.
21. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 21 Some things we’ve built • Repository Invitations • User blocking API • Organizations have the ability to block users • Contributor badges • Community Health management tools • Transparency around comment edit history
22. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 22 On any platform, any time you have user-to-user interactions you have the possibility of harassment.
23. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 23 Types of online harassment • Posting personal info (doxing) • Hate speech • Sexual or pornographic content • Impersonation • SWATing • Spam • Identity theft • Exclusionary behavior • Death and rape threats • Dog piling • Personal attacks • Threatening and inappropriate images • So many more
24. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 24 Racist repository names led to repository invitations
25. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 25
26. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 26 Women, women of color, and trans women experience harassment daily. We are experts in online safety. Not to say that men don’t experience harassment, but women tend to get the worst of it
27. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 27 70% women 40% women of color 30% trans women 30% white men 4 engineers 1 designer 1 product manager 1 qa 1 year of work to address 7 years of backlog
28. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 28 Some tactics women use to avoid online harassment • Not using full names on social media sites • Not using real photos for avatars • Turning off location tracking data • Regularly Googling name, address, & phone numbers • Using a burner phone number service & PO boxes • Turning off direct messages • Not live posting on social media during vacation
29. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 29 @tsunamino http://www.wocintechchat.com/blog/wocintechphotos
30. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 30 @tsunamino
31. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 31 It’s our job to write software that protects vulnerable people.
32. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 32 http://www.huffingtonpost.com/aran-khanna/stalking-your-friends-on-facebook-messenger-_b_7452912.html
33. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 33 http://www.huffingtonpost.com/aran-khanna/stalking-your-friends-on-facebook-messenger-_b_7452912.html
34. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 34 http://www.huffingtonpost.com/aran-khanna/venmo-money_b_8418130.html
35. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 35 " " " " BUT FREE SPEECH BUT FREE SPEECH https://www.flickr.com/photos/epw/348448681/in/photolist-wMTDT-eMnoZK-ahqLSQ-dWv9vQ-dK4Hnd-eLPF8d-DKNYyX-jnCGNf-eNgckU-388Mus-eKGUs3- eR26Ui-58nGwR-idg5CJ-9Jwyxb-639vTn-FmvThd-4e5Fff-ah1Dse-eiTkAT-9Jwyzh-pvMWjr-ovCHpM-dWtVVp-i2aAxc-5mDSDM-DrXkrE-eciQvg-c9811Y-4t5Tcr-dbzN7K- r1WnoK-qKrNny-qKrueE-dWtJ4T-hYCB6D-hDr4y9-bq3RAJ-92ee1-dTuCwi-Ji54Ed-eMa6Du-9yqLWZ-7PoVXq-b2Ev8R-aho1T2-9TR1s8-9E6usg-nzmYLL-dNEzjC
36. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 36 “Free speech” does not mean you get to say whatever you want without consequences
37. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 37 I mean, you can, but it’ll cost you a bunch of money http://www.businessinsider.com/twitter-trolls-caused-salesforce-to-walk-away-from-deal-2016-10 http://sfist.com/2016/10/18/disney_dropped_twitter_bid_because.php https://www.bloomberg.com/news/articles/2016-10-17/disney-said-to-have-dropped-twitter-pursuit-partly-over-image http://money.cnn.com/2016/10/14/technology/salesforce-not-buying-twitter/
38. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 38 https://www.theguardian.com/commentisfree/2017/jan/03/ive-left-twitter-unusable-anyone-but-trolls-robots-dictators-lindy-west
39. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 39 More than half of app users have uninstalled or decided to not install an app due to concerns about personal information Pew Research Center http://www.pewinternet.org/2012/09/05/privacy-and-data-management-on-mobile-devices/
40. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 40 Safety features shouldn’t be an afterthought. This leads to years of technical debt.
41. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 41 70% women 40% women of color 30% trans women 30% white men 4 engineers 1 designer 1 product manager 1 qa 1 year of work to address 7 years of backlog
42. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 42 Design software with trust & safety in mind before it becomes a problem.
43. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 43 Have a definition of harassment.
44. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 44 GitHub’s definition of harassment is destructive behavior that undermines productivity.
45. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 45 While it is not forbidden to broach topics such as age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation, we do not tolerate speech that attacks a person or group of people on the basis of who they are.
46. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 46 https://mobile.twitter.com/TweetAnnaMarie/status/789957313649967104
47. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 47 Build Trust & Safety into new features • Ensure explicit consent • Keep an audit log trail • Strive to minimize abuse https://githubengineering.com/community-and-safety-feature-reviews/
48. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 48 Build consensual software.
49. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 49
50. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 50 Consent means all parties are aware and able to agree to an interaction.
51. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 51 https://www.youtube.com/watch?v=oQbei5JGiT8
52. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 52
53. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 53
54. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 54 Consensual software means all users are aware and explicitly consenting to every interaction and disclosure of their personal data on your site or service.
55. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 55
56. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 56 More than half of app users have uninstalled or decided to not install an app due to concerns about personal information Pew Research Center http://www.pewinternet.org/2012/09/05/privacy-and-data-management-on-mobile-devices/
57. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 57 % &
58. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 58 Ensure explicit consent
59. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 59
60. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 60 %• Does Alice want to be in this interaction? • Is it safe to block Bob? • How much of Alice’s information is revealed if she reports Bob?
61. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 61
62. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 62 https://www.vg247.com/2015/06/03/the-sad-saga-of-lego-universes-dong-detection/
63. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 63 https://techcrunch.com/2016/03/24/microsoft-silences-its-new-a-i-bot-tay-after-twitter-users-teach-it-racism/
64. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 64 https://qz.com/818151/virtual-assistant-bots-like-siri-alexa-and-cortana-spend-much-of-their-time-fending-off-sexual-harassment/
65. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 65 • Is it easy to create sock puppet accounts? • Can you introduce friction to discourage destructive behavior? • Is it easy to learn from mistakes? &
66. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 66 Keep an audit log trail
67. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 67 )• Is there an audit log for your support folks? • If you can edit content, do you know who did what change? • How many people have reported Bob?
68. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 68 Strive to minimize abuse
69. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 69 *• Are you limiting 0-day account activity? • Are you rate limiting potentially abusive accounts? • If you have a reputation score, how is it calculated?
70. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 70 https://githubengineering.com/community-and-safety- feature-reviews/
71. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 71 Smart Girl’s Guide to Privacy by Violet Blue HATERS: Harassment, Abuse, and Violence Online by Bailey Poland The Internet of Garbage by Sarah Jeong
72. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 72 Have a definition of harassment.
73. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 73 Trust & Safety guidelines for new features • Ensure explicit consent • Keep an audit log trail • Strive to minimize abuse https://githubengineering.com/community-and-safety-feature-reviews/
74. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 74 Make sure your users are aware and explicitly consenting to every interaction and disclosure of their personal data on your site or service.
75. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 75 Hire more women of color and trans women. We are experts in avoiding online harassment.
76. Consensual Software: Prioritizing Trust & Safety ! @tsunamino 76 Questions?
77. How people build software ! " Thanks!