Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Ethics

Practical Ethics

How do you collect data and run experiments on users in an ethical way?

Presented as a keynote at O'Reilly Velocity NYC 2018.

Laura Thomson

October 03, 2018

More Decks by Laura Thomson

Other Decks in Technology


  1. Standard Disclaimers This is what we do. It’s not perfect.

    This approach is open source so you can steal it and make it better. Give us your feedback so we can make it better too.
  2. Lean Data Collect only what you need Keep it for

    the minimum amount of time Don’t violate user expectations
  3. Category 2: Interaction Data Examples: # of tabs, session length,

    config settings, feature use Generally okay to collect, opt-out.
  4. Category 4: Highly Sensitive Data Examples: email, username, identifiers Assume

    no. Maybe opt-in with advance notice, user consent, and secondary opt-out.
  5. Collecting data is simple 1. Request for collection 2. Review

    by data steward https://github.com/mozilla/data-review
  6. –Rebecca Weiss, Director of Data Science ‘By not performing A/B

    tests before we release new features and products, we are guilty of administering massive uncontrolled experiments upon our users. The only outcome measure that we can observe as a result of these experiments is “how many users have we driven away since we released that feature?”’
  7. How’d that happen? Good intentions, road to hell, etc No

    data collected No one felt empowered to say no
  8. What did we learn? More formal process Definition of red

    flags Deeper engineering review Documented escalation paths
  9. Fin We can all do better. Learn from your mistakes.

    Steal these ideas. Steward your users’ data wisely. Come ask questions.