Firefox Privacy Settings Breakage Study

From July-August 2018, we conducted a study to learn privacy settings effects' on website breakage.

Transcript

1. Privacy Settings Breakage Study August 2017
2. Brought to you by ... Jacqueline Savory Interaction Designer Luke Crouch PrivSec Engineer Peter Dolanjski Product Manager
3. Brought to you by ... Ryan Harter Browser Measurement Ilana Segall Data Science Years of awesome Privacy engineers
4. Existing Claims Tracking Protection breaks websites Broken websites make users leave Firefox Some existing prefs could protect users with minimal breakage
5. Opt-in page
6. On-boarding
7. Report: “page problem” “page works”
8. Breakage type
9. Notes
10. Thank you!
11. Disable Study
12. The numbers https://sql.telemetry.mozilla.org/dashboard/shield-study-improve-privacy-settings
13. 19,000+ users 9 branches https://github.com/mozilla/shield-study-privacy
14. 2,100+ users in each branch https://sql.telemetry.mozilla.org/queries/17837#73402
15. Up to 8,500 active users/day https://sql.telemetry.mozilla.org/queries/23123#60018
16. About those claims ... Tracking Protection breaks websites Broken websites make users leave Firefox Some existing prefs could protect users with minimal breakage
17. Does Tracking Protection break websites?
18. Avg. problems reported per user looks lower for trackingProtection ... https://sql.telemetry.mozilla.org/queries/23721#61701
19. Avg. problems reported per user looks lower for trackingProtection ... https://sql.telemetry.mozilla.org/queries/23721#61701 WTF?
20. Some control users’ problems ... “Something* on the page is slowing down the loading speed significantly.” *Spoiler Alert: it’s the trackers “not responsive”, “slow, freezing”, “Took longer than usual for page to load”, “Connection appears slower than usual”, “Pages are scrolling slowly”, “very slow to load”, “long wait for anything to occur”, “the fire fox not always responding”, “page is very slow to load”, “tremendous lag , page loads very slowly”, “page was laggy and didn't respond”, “Sending mail in Gmail is very slow since installation of this study”, “really slow to load”, “video doesn't load fast”, ...
21. Tracking Protection may actually fix websites by blocking tracking elements that break/slow them down!
22. Do broken websites make users leave Firefox?
23. Do broken websites make users leave this study?
24. Some common site breakages ... resistFingerprinting causes Facebook problems firstPartyIsolation causes YouTube problems https://sql.telemetry.mozilla.org/queries/18276#61772
25. What % of users leave the study? After reporting breakage on certain popular sites https://sql.telemetry.mozilla.org/queries/27989#73748 Don’t break SUMO: 72% of users left Don’t break Google sites: 6 out of top 25 Don’t break email: Yahoo Mail, Gmail, Outlook Live in top 20
26. What % of users leave the study? After reporting breakage on longer tail sites https://sql.telemetry.mozilla.org/queries/27989#73748 Don’t break dev sites? Atlassian, GitHub show up Don’t break porn sites? xvideos show up
27. Breaking workflow sites (search, accounts, email, support, development) makes users leave this study
28. Do certain kinds of breakage make users leave this study?
29. “other” is most common breakage* ... https://sql.telemetry.mozilla.org/queries/19634#50162 * new claim: we’re not good at predicting kinds of breakage?
30. Some common breakages firstPartyIsolation causes login failures resistFingerprinting causes flash problems https://sql.telemetry.mozilla.org/queries/19634#61483
31. What % of users with certain breakage disable the study? https://sql.telemetry.mozilla.org/queries/20097#51471 94% of users reporting screen breakage disable study 84% of users reporting flash breakage disable study 82% of users reporting login-failure breakage disable study 64% of users reporting payment breakage disable study
32. What % of users with certain breakage disable the study? https://sql.telemetry.mozilla.org/queries/20097#51471 screen and flash are only in resistFingerprinting payment is in control & 3DP cookies branches login-failure in control, 3DP cookies, first-party isolation, & referer branches
33. Across all branches, breaking logins and payments makes users leave this study
34. Are there existing prefs that could protect users with minimal breakage?
35. 14% of control users report breakage 18% of firstPartyIsolationOpenerAccess users: the max recorded in the study https://sql.telemetry.mozilla.org/queries/23644#61485 6 settings are within margin of error of control
36. .21 avg. problems per control user .25 thirdPartyCookiesOnlyFromVisited .19 trackingProtection https://sql.telemetry.mozilla.org/queries/23721#61701 4 settings are within margin of error of control
37. 5.1% of control users disable study 8.5% of firstPartyIsolation users 4.7% of originOnlyToThirdParties users https://sql.telemetry.mozilla.org/queries/19633#50159 5 settings are within margin of error of control
38. How can we compare “overall breakage”?
39. “Composite Breakage Score” An index of web breakage % of users who report breakage Average breakage reported by each user % of users who disable the protection (presumably because of breakage) * *
40. “Composite Breakage Scores” https://docs.google.com/spreadsheets/d/1m7XEXh93Sa-lu9jZClf-CQYuRoN3zg7rI5naVKtHWOA/edit#gid=0
41. “Composite Breakage Scores” https://docs.google.com/spreadsheets/d/1m7XEXh93Sa-lu9jZClf-CQYuRoN3zg7rI5naVKtHWOA/edit#gid=0 WTF?
42. .24 noThirdPartyCookies .24 sessionOnlyThirdPartyCookies .27 thirdPartyCookiesOnlyFromVisited https://sql.telemetry.mozilla.org/queries/23721#61701 WTF?
43. Some thirdPartyCookiesOnlyFromVisited users’ problems ... “The message tells me that my cookies are blocked even though my settings are to accept cookies.” “Got this message ... Cookies are blocked. … your browser doesn’t allow cookies. ...change your browser settings.”, “Cannot access on onlyine bill pay because it thinks cookies are blocked. I checked an my options say to allow cookies. what is going on?”
44. Users don’t understand Accept third-party cookies: From Visited ?
45. Most promising prefs Based on “Composite Breakage Score” originOnlyReferer ToThirdParties trackingProtection sessionOnly ThirdPartyCookies
46. User values originOnlyRefererToThirdParties Reduces detail sent to trackers Very few login failures Very little mail breakage
47. Ecosystem values originOnlyRefererToThirdParties Does not block ads Referers are used to guarantee ad policies
48. User values trackingProtection Blocks known trackers completely Speed boost Very little mail breakage Triggers ad-blocker-blocker walls
49. Ecosystem values trackingProtection Blocks ads
50. User values sessionOnlyThirdPartyCookies Limits duration of tracking Very little mail breakage Some login and “unexpected signout” failures
51. Ecosystem values sessionOnlyThirdPartyCookies Does not blocks ads
52. Key take-aways Tracking Protection doesn’t seem to “break” websites as much as we feared Breaking workflow sites makes users disable the study Yes! Some existing prefs could protect users with minimal breakage
53. More: post-study user surveys https://data.surveygizmo.com/r/28049_59b7e980008742.80492645
54. Next ... Q3-Q4 2017
55. Next Q3-Q4 Tracking Protection opt-out study measuring user engagement & retention “Creepy Ads” Experiment|Study to identify trackers involved with creepiest online ads Your idea? Join us in #privacy