Prediction and Control: Watching Algorithms Jennifer Helsby (@redshiftzero) Postdoctoral Fellow Computation Institute University of Chicago 32c3 December 29, 2015 

• Algorithmic landscape • Advantages of intelligent systems • Current applications • The nightmare case • Properties of algorithmic systems • Fairness • Transparency • Accountability • Paths forward Outline 

Technology is neither good not bad; nor is it neutral. - Kranzberg’s First Law of Technology 

government collection social media IoT devices, sensors medical history voice location genetic information biometrics browsing profiles wearables emails and messaging logs 

• slow • biased • not transparent • difficult to audit • fast • unbiased? • transparent? • can audit? human decision algorithmic decision 

Data Inference Action 

Algorithmic decision making is expanding from industry into new domains. 

No content

No content

No content

VOTE VOTE VOTE VOTE VOTE VOTE VOTE VOTE 

data-driven city 

Advantages • Automate rote tasks • Enable us to distill huge volumes of data down to the critical pieces of information • Optimize resource allocation • Optimize actions to produce desired outcomes 

Algorithms can have serious implications. 

Chinese Social Credit System ranks each citizen based on their behavior to be used by all citizens by 2020 • financial record • criminal history • drivers license history • some medical information • purchase history • social media monitoring • social network analysis • … “carry forward sincerity and traditional virtues” 

What do we want? 

Privacy Fairness Transparency Accountability What do we want? 

I. Fairness 

Predictive Policing 

Predictive Policing individual level geographic level 

No content

No content

Individual Fairness similar people treated similarly Group Fairness protected groups treated similarly Fairness in Errors errors should not be concentrated in protected groups Zemel et al. 2013, Dwork et al. 2011 Fairness Concerns 

Training Data Issues • crime databases contain only crimes detected by police • effect of biased policing, e.g. black people arrested at rates 3.73x that of whites for marijuana crimes [ACLU] 

• Race and location are correlated in the US: machine learning systems can learn sensitive features • No opt-out: citizens need more information about these systems 

II. Transparency 

What are these systems doing? ? 

Complexity Algorithms must be human interpretable 

Influence 

Alerting users to the fact manipulation might be occurring did not decrease the amplitude of the effect. Conservative Conservative Liberal Liberal Control Control Manipulate Manipulate Epstein and Robertson 2015 “it is a relatively simple matter to mask the bias in search rankings so that it is undetectable to virtually every user” Influencing Voting Introduced 20% shift in voter preferences 

behaviouralinsights.co.uk 

behaviouralinsights.co.uk 

No content

Optimization Industry: • Time spent on website • Click through rate • Likes • Profit Politics: • Votes for the desired candidate Policy: • Better use of government services • Voter registration • Health outcomes • Education outcomes • Compliance 

The invisible barbed wire of big data limits our lives to a space that might look quiet and enticing enough, but is not of our own choosing and that we cannot rebuild or expand. The worst part is that we do not see it as such. Because we believe that we are free to go anywhere, the barbed wire remains invisible. -Morozov 

III. Accountability 

oversight 

No content

Greene vs. San Francisco • San Francisco ALPR gets (false) hit on car • Traffic stop of 47-year old woman • Officers conducted pat-down, search, held at gunpoint 

Auditing 

? Inputs: Test accounts Real accounts (collaborative approach) Outputs: Was one output shown to one user and not another? 

? Browsing profile Ad 1 Ad 2 … Ad n 

Datta et al 2015 Ad Network Browsing Profile Browsing Profile Browsing Profile Browsing Profile Treatment 2 Treatment 1 Ad 1 … Ad n Ad 1 … Ad n Ad 1 … Ad n Ad 1 … Ad n 

No content

Who is in control? 

Audit tools Sunlight: https://columbia.github.io/sunlight/ AdFisher: https://github.com/tadatitam/info-flow-experiments/ Anonymity systems (Tor) View the landscape Obfuscate by injecting noise (Adnauseam.io) Technology 

Policy Regulation Independent ethics review 3rd party audits 

Closing thoughts Algorithmic systems need to have appropriate oversight in order to be controlled Hacking and privacy advocacy community has an important role to play in this fight Thanks! Jennifer Helsby @redshiftzero Postdoctoral Fellow Computation Institute University of Chicago 

References ACLU, “The War on Marijuana”, https://www.aclu.org/files/assets/aclu- thewaronmarijuana-rel2.pdf Bond, Farris, Jones, Kramer, Marlow, Settle and Fowler, “A 61-milion-person experiment in social influence and political mobilization”, Nature 489, 295–298 , 2012 Datta, Tschantz, and Datta, “Automated Experiments on Ad Privacy Settings”, Proceedings on Privacy Enhancing Technologies 2015; 2015 (1):92-112 Dwork, Hardt, Pitassi, Reingold, and Zemel, “Fairness through Awareness”, ITCS '12 Proceedings of the 3rd Innovations in Theoretical Computer Science Conference Epstein and Robertson, “The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections”, PNAS 2015, vol. 112 no. 33 Zemel, Wu, Swersky, Pitassi, and Dwork. “Learning Fair Representations.”, ICML (3), volume 28 of JMLR Proceedings, page 325-333 Chinese Social Credit System: volkskrant.nl/buitenland/china-rates-its-own-citizens-including-online- behaviour~a3979668/ bbc.com/news/world-asia-china-34592186 newyorker.com/news/daily-comment/how-china-wants-to-rate-its-citizens Predictive Policing: azavea.com/blogs/newsletter/v8i5/hunchlab-2-0-defining-the-future-of-predictive- policing/ popsci.com/want-to-prevent-another-ferguson predpol.com Political Manipulation: newstatesman.com/politics/2014/06/facebook-could-decide-election-without- anyone-ever-finding-out politico.com/magazine/story/2015/08/how-google-could-rig-the-2016- election-121548 Twins: wbay.com/2015/10/23/twins-denied-drivers-permit-because-dmv-cant-tell-them- apart/ ALPR: Greene v. San Francisco, 9th Cir. Court Image credit: Freddy Martinez