Practical Ethics

How do you collect data and run experiments on users in an ethical way?

Presented as a keynote at O'Reilly Velocity NYC 2018.

Transcript

1. Practical Ethics
   Laura Thomson
   laura@mozilla.com
   @lxt
   

   View Slide

2. View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. –Anonymous commenter
   “Do you think other browser makers collect this type of data?”
   

   View Slide

8. Not an ethicist
   

   View Slide

9. How To Be Perfect
   

   View Slide

10. How To Be Perfect
    

    View Slide

11. Practical Ethics
    

    View Slide

12. Standard Disclaimers
    This is what we do. It’s not perfect.
    This approach is open source so you can steal it and make it better.
    Give us your feedback so we can make it better too.
    

    View Slide

13. Lean Data
    Collect only what you need
    Keep it for the minimum amount of time
    Don’t violate user expectations
    

    View Slide

14. Classes of Data
    

    View Slide

15. Category 1: Technical Data
    Examples: OS, available memory, version number
    Generally okay to collect, opt-out
    

    View Slide

16. Category 2: Interaction Data
    Examples: # of tabs, session length, config settings, feature use
    Generally okay to collect, opt-out.
    

    View Slide

17. Category 3: Web Activity Data
    Example: browsing history
    Stickier. Usually no, but may be possible with mitigation.
    

    View Slide

18. Category 4: Highly Sensitive Data
    Examples: email, username, identifiers
    Assume no. Maybe opt-in with advance notice, user consent, and secondary opt-out.
    

    View Slide

19. Collecting data is simple
    1. Request for collection
    2. Review by data steward
    https://github.com/mozilla/data-review
    

    View Slide

20. What is a Data Steward?
    

    View Slide

21. “Case Law”
    Precedent
    Allows reasoning about data collection
    Suggests alternatives
    

    View Slide

22. Privacy Preserving Data Collection
    

    View Slide

23. Experiments
    

    View Slide

24. –Rebecca Weiss, Director of Data Science
    ‘By not performing A/B tests before we release new features and
    products, we are guilty of administering massive uncontrolled
    experiments upon our users.
    The only outcome measure that we can observe as a result of
    these experiments is “how many users have we driven away
    since we released that feature?”’
    

    View Slide

25. View Slide

26. View Slide

27. View Slide

28. Case Studies
    

    View Slide

29. View Slide

30. How’d that happen?
    Good intentions, road to hell, etc
    No data collected
    No one felt empowered to say no
    

    View Slide

31. What did we learn?
    More formal process
    Definition of red flags
    Deeper engineering review
    Documented escalation paths
    

    View Slide

32. View Slide

33. “Burn it all. Burn it to the ground.”
    

    View Slide

34. Fin
    We can all do better.
    Learn from your mistakes.
    Steal these ideas.
    Steward your users’ data wisely.
    Come ask questions.
    

    View Slide

35. References
    • https://wiki.mozilla.org/Firefox/Data_Collection
    • https://github.com/mozilla/data-review
    • https://wiki.mozilla.org/Firefox/Shield/PHD
    • https://testpilot.firefox.com/
    • https://addons.mozilla.org/en-US/firefox/addon/firefox-pioneer/
    

    View Slide