Tweet
Tweet

Slide 1

Slide 1 text

Privacy Settings Breakage Study August 2017

Slide 2

Slide 2 text

Brought to you by ... Jacqueline Savory Interaction Designer Luke Crouch PrivSec Engineer Peter Dolanjski Product Manager

Slide 3

Slide 3 text

Brought to you by ... Ryan Harter Browser Measurement Ilana Segall Data Science Years of awesome Privacy engineers

Slide 4

Slide 4 text

Existing Claims Tracking Protection breaks websites Broken websites make users leave Firefox Some existing prefs could protect users with minimal breakage

Slide 5

Slide 5 text

Opt-in page

Slide 6

Slide 6 text

On-boarding

Slide 7

Slide 7 text

Report: “page problem” “page works”

Slide 8

Slide 8 text

Breakage type

Slide 9

Slide 9 text

Notes

Slide 10

Slide 10 text

Thank you!

Slide 11

Slide 11 text

Disable Study

Slide 12

Slide 12 text

The numbers https://sql.telemetry.mozilla.org/dashboard/shield-study-improve-privacy-settings

Slide 13

Slide 13 text

19,000+ users 9 branches https://github.com/mozilla/shield-study-privacy

Slide 14

Slide 14 text

2,100+ users in each branch https://sql.telemetry.mozilla.org/queries/17837#73402

Slide 15

Slide 15 text

Up to 8,500 active users/day https://sql.telemetry.mozilla.org/queries/23123#60018

Slide 16

Slide 16 text

About those claims ... Tracking Protection breaks websites Broken websites make users leave Firefox Some existing prefs could protect users with minimal breakage

Slide 17

Slide 17 text

Does Tracking Protection break websites?

Slide 18

Slide 18 text

Avg. problems reported per user looks lower for trackingProtection ... https://sql.telemetry.mozilla.org/queries/23721#61701

Slide 19

Slide 19 text

Avg. problems reported per user looks lower for trackingProtection ... https://sql.telemetry.mozilla.org/queries/23721#61701 WTF?

Slide 20

Slide 20 text

Some control users’ problems ... “Something* on the page is slowing down the loading speed significantly.” *Spoiler Alert: it’s the trackers “not responsive”, “slow, freezing”, “Took longer than usual for page to load”, “Connection appears slower than usual”, “Pages are scrolling slowly”, “very slow to load”, “long wait for anything to occur”, “the fire fox not always responding”, “page is very slow to load”, “tremendous lag , page loads very slowly”, “page was laggy and didn't respond”, “Sending mail in Gmail is very slow since installation of this study”, “really slow to load”, “video doesn't load fast”, ...

Slide 21

Slide 21 text

Tracking Protection may actually fix websites by blocking tracking elements that break/slow them down!

Slide 22

Slide 22 text

Do broken websites make users leave Firefox?

Slide 23

Slide 23 text

Do broken websites make users leave this study?

Slide 24

Slide 24 text

Some common site breakages ... resistFingerprinting causes Facebook problems firstPartyIsolation causes YouTube problems https://sql.telemetry.mozilla.org/queries/18276#61772

Slide 25

Slide 25 text

What % of users leave the study? After reporting breakage on certain popular sites https://sql.telemetry.mozilla.org/queries/27989#73748 Don’t break SUMO: 72% of users left Don’t break Google sites: 6 out of top 25 Don’t break email: Yahoo Mail, Gmail, Outlook Live in top 20

Slide 26

Slide 26 text

What % of users leave the study? After reporting breakage on longer tail sites https://sql.telemetry.mozilla.org/queries/27989#73748 Don’t break dev sites? Atlassian, GitHub show up Don’t break porn sites? xvideos show up

Slide 27

Slide 27 text

Breaking workflow sites (search, accounts, email, support, development) makes users leave this study

Slide 28

Slide 28 text

Do certain kinds of breakage make users leave this study?

Slide 29

Slide 29 text

“other” is most common breakage* ... https://sql.telemetry.mozilla.org/queries/19634#50162 * new claim: we’re not good at predicting kinds of breakage?

Slide 30

Slide 30 text

Some common breakages firstPartyIsolation causes login failures resistFingerprinting causes flash problems https://sql.telemetry.mozilla.org/queries/19634#61483

Slide 31

Slide 31 text

What % of users with certain breakage disable the study? https://sql.telemetry.mozilla.org/queries/20097#51471 94% of users reporting screen breakage disable study 84% of users reporting flash breakage disable study 82% of users reporting login-failure breakage disable study 64% of users reporting payment breakage disable study

Slide 32

Slide 32 text

What % of users with certain breakage disable the study? https://sql.telemetry.mozilla.org/queries/20097#51471 screen and flash are only in resistFingerprinting payment is in control & 3DP cookies branches login-failure in control, 3DP cookies, first-party isolation, & referer branches

Slide 33

Slide 33 text

Across all branches, breaking logins and payments makes users leave this study

Slide 34

Slide 34 text

Are there existing prefs that could protect users with minimal breakage?

Slide 35

Slide 35 text

14% of control users report breakage 18% of firstPartyIsolationOpenerAccess users: the max recorded in the study https://sql.telemetry.mozilla.org/queries/23644#61485 6 settings are within margin of error of control

Slide 36

Slide 36 text

.21 avg. problems per control user .25 thirdPartyCookiesOnlyFromVisited .19 trackingProtection https://sql.telemetry.mozilla.org/queries/23721#61701 4 settings are within margin of error of control

Slide 37

Slide 37 text

5.1% of control users disable study 8.5% of firstPartyIsolation users 4.7% of originOnlyToThirdParties users https://sql.telemetry.mozilla.org/queries/19633#50159 5 settings are within margin of error of control

Slide 38

Slide 38 text

How can we compare “overall breakage”?

Slide 39

Slide 39 text

“Composite Breakage Score” An index of web breakage % of users who report breakage Average breakage reported by each user % of users who disable the protection (presumably because of breakage) * *

Slide 40

Slide 40 text

“Composite Breakage Scores” https://docs.google.com/spreadsheets/d/1m7XEXh93Sa-lu9jZClf-CQYuRoN3zg7rI5naVKtHWOA/edit#gid=0

Slide 41

Slide 41 text

“Composite Breakage Scores” https://docs.google.com/spreadsheets/d/1m7XEXh93Sa-lu9jZClf-CQYuRoN3zg7rI5naVKtHWOA/edit#gid=0 WTF?

Slide 42

Slide 42 text

.24 noThirdPartyCookies .24 sessionOnlyThirdPartyCookies .27 thirdPartyCookiesOnlyFromVisited https://sql.telemetry.mozilla.org/queries/23721#61701 WTF?

Slide 43

Slide 43 text

Some thirdPartyCookiesOnlyFromVisited users’ problems ... “The message tells me that my cookies are blocked even though my settings are to accept cookies.” “Got this message ... Cookies are blocked. … your browser doesn’t allow cookies. ...change your browser settings.”, “Cannot access on onlyine bill pay because it thinks cookies are blocked. I checked an my options say to allow cookies. what is going on?”

Slide 44

Slide 44 text

Users don’t understand Accept third-party cookies: From Visited ?

Slide 45

Slide 45 text

Most promising prefs Based on “Composite Breakage Score” originOnlyReferer ToThirdParties trackingProtection sessionOnly ThirdPartyCookies

Slide 46

Slide 46 text

User values originOnlyRefererToThirdParties Reduces detail sent to trackers Very few login failures Very little mail breakage

Slide 47

Slide 47 text

Ecosystem values originOnlyRefererToThirdParties Does not block ads Referers are used to guarantee ad policies

Slide 48

Slide 48 text

User values trackingProtection Blocks known trackers completely Speed boost Very little mail breakage Triggers ad-blocker-blocker walls

Slide 49

Slide 49 text

Ecosystem values trackingProtection Blocks ads

Slide 50

Slide 50 text

User values sessionOnlyThirdPartyCookies Limits duration of tracking Very little mail breakage Some login and “unexpected signout” failures

Slide 51

Slide 51 text

Ecosystem values sessionOnlyThirdPartyCookies Does not blocks ads

Slide 52

Slide 52 text

Key take-aways Tracking Protection doesn’t seem to “break” websites as much as we feared Breaking workflow sites makes users disable the study Yes! Some existing prefs could protect users with minimal breakage

Slide 53

Slide 53 text

More: post-study user surveys https://data.surveygizmo.com/r/28049_59b7e980008742.80492645

Slide 54

Slide 54 text

Next ... Q3-Q4 2017

Slide 55

Slide 55 text

Next Q3-Q4 Tracking Protection opt-out study measuring user engagement & retention “Creepy Ads” Experiment|Study to identify trackers involved with creepiest online ads Your idea? Join us in #privacy