Science! Science Harder!

How we reinvented ourselves to be data literate, experiment-driven, and ship faster.

Presented by Laura Thomson and Rebecca Weiss at Monitorama 2017

Transcript

1. Science! Science Harder! How we reinvented ourselves to be data literate, experiment-driven, and ship faster Rebecca Weiss & Laura Thomson (@lxt) {rweiss, laura}@mozilla.com
2. Science! Science Harder! How we are reinventeding ourselves to be data literate, experiment-driven, and ship faster Rebecca Weiss & Laura Thomson (@lxt) {rweiss, laura}@mozilla.com
3. In the beginning...
4. Data is bad. It’s bad. And you’re bad for wanting it. You carry only 3 things as you enter the wilderness: 1. Downloads 2. Blocklist pings 3. Crash data (kept in a monastery, guarded by ninjas)
6. Decision-making, without evidence (also bad)
7. ! Did this code change affect retention? ! Are we regressing on Firefox performance? ! Is crashiness getting worse or better? ! How often do our users experience janky behavior? ! How many active users do we have, anyway?
8. Browser = app? Not quite.
9. Browser = service? Sort of.
10. =
11. Failure to abstract Build system to answer question. Answer the question! Yay! Ask more questions. Fail. Build new measurement system.
12. ! Blocklist ping ◦ Disables badness (default on, for your protection) ! Telemetry (v1) ◦ Aggregate performance measurement (default off) ! Firefox Health Report ◦ Very limited individual-level longitudinal measurement (default on) ◦ Extended individual-level longitudinal measurement (default off) ! Crash stats ◦ Crash details (default off) Measuring by accident, reluctantly
15. Need one data collection system, not four* *there are more than four. perhaps a lot more.
16. Also, privacy
18. Meanwhile, in the real world... From http://flickr.com/photo/44124323641@N01/246805948, CC-Generic 2.0 with attribution
21. Enabling a data culture Based on scientific principles Embrace data consistency and reliability over precision Prioritize self-service over personalized tools Build a badass experimental platform ASAP
22. A single data collection system
23. Unified Telemetry One infrastructure, many types of probes and pings Many types of events to measure One good system vs many half baked Unify mental models One set of infrastructure quirks One set of errors By Mike Wutzler AKA Darth Mike (Own work) [GFDL (http://www.gnu.org/ copyleft/fdl.html), CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/ 3.0/) or FAL], via Wikimedia Commons
24. Choose self-service services
25. Before
26. !re:dash: automatic dashboarding for simple requests !Airflow: monitoring and job scheduling for dataset creation !Spark and Parquet: standardized analytical model and data format !Presto: SQL access to many data sets !And more... Now
27. Reproducible knowledge generation Experimented Tabulated Version controlled Productionized Democratized
28. Transparency = citation needed Results must be reproducible as a URL Audit a number all the way down to the code Enforce transparent model of work Open science, open methodology https://xkcd.com/285/ Transparency = [citation needed]
29. Success smells like https://sql.telemetry.mozilla.org/dashboard/re-dash-health 623 total users so far
30. Public reports you can use https://metrics.mozilla.com/firefox-hardware-report/
31. Experimenting with experiments
34. Previously: Funnelcakes Telemetry experiments Test Pilot (Mark I) Weird science By The original uploader was Lorax at English Wikipedia (Taken by user Lorax and released under the GNU FDL) [GFDL (http://www.gnu. copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], via Wikimedia Commons
35. These kind of sucked: High barriers to entry Inconsistent measurement approaches Not production-y Science needs to be open
36. Part of core Firefox, modularized into add- ons Build/test against existing Firefox builds Update via xpi, independent of Fx Update up to daily (for now) on any release channel (Not really an experimental mechanism) System add-ons Photo by Francesco Lodolo, l10n team.
37. Mark II Test Pilot Users opt-in to install and try whole new features with extended measurement Successful features graduate to Firefox - e.g. Firefox Screenshots More: https://testpilot.firefox.com/
38. Real sampling! Flip a preference, install a new feature, collect data Multiple experiments in flight at once SHIELD Studies
39. Run Firefox Run
41. Data will shape the Web. What kind of Web do you want? laura@mozilla.com / @lxt rweiss@mozilla.com / @rweiss