Data Mining 101

How data collection and
aggregation shapes our world

Transcript

1. Data Mining 101 How data collection and aggregation shapes our world @anton612
2. Case Study: Nice Ride
3. 2011 Nice Ride Data • Subscriber ID • Starting Location • Starting Date/Time • Ending Location • Ending Date/Time • Subscriber Type
4. 2011 Nice Ride Data
5. 2011 Nice Ride Data • Subscriber ID • Starting Location • Starting Date/Time • Ending Location • Ending Date/Time • Subscriber Type • Gender • ZIP Code • Date of Birth
6. Nice Ride Privacy Policy
7. Nice Ride Privacy Policy "How is my personal information used or shared?"
8. Nice Ride Privacy Policy "How is my personal information used or shared?" "We may share aggregated demographic information (data that cannot identify any individual person) with our partners and sponsors." -Nice Ride Privacy Policy, (effective as of 03/17/2010)
9. What information is "Personally Identifiable"?
10. What information is "Personally Identifiable"? https://www.eff.org/deeplinks/2009/09/what-information-personally-identifiable
11. example
12. Data Aggregation: Easy Mode
13. Data Aggregation: Easy Mode
14. Data Aggregation: Easy Mode R.T. Rybak Born: 11/12/1955 Gender: M ZIP: 55409
15. Data Aggregation: Easy Mode R.T. Rybak Born: 11/12/1955 Gender: M ZIP: 55409 Nothing!
16. Data Aggregation: Easy Mode R.T. Rybak Born: 11/12/1955 Gender: M ZIP: 55409 55419 Unique ID: PV9KSC9BSY
17. Data Aggregation: Easy Mode What can we learn with this Subscriber ID? • Beginning and ending location and times for all rides • Frequency of use • Subscriber IDs of “friends” who check out and return bikes at the same time as target Most importantly, we can combine the above data with other data sources
18. Data Aggregation: Hard Mode • Subscriber ID • Starting Location • Starting Date/Time • Ending Location • Ending Date/Time • Subscriber Type • Gender • ZIP Code • Date of Birth
19. Data Aggregation: Hard Mode What are some other sources of data?
20. Data Aggregation: Hard Mode What are some other sources of data?
21. Data Aggregation: Hard Mode What are some other sources of data?
22. Data Aggregation: Hard Mode
23. Data Aggregation: Hard Mode
24. Data Aggregation: Hard Mode How many users rode 3 or more times on September 7, 2011?
25. Data Aggregation: Hard Mode How many users rode 3 or more times on September 7, 2011? A: 160
26. Data Aggregation: Hard Mode How many users rode 3 or more times on September 7, 2011? A: 160 How could we further narrow this data down?
27. Data Aggregation: Hard Mode Combine with other tweets which reveal info:
28. Data Aggregation: Hard Mode Combine with other tweets which reveal info: Confirm/deny potential IDs via public info: - RT probably checked out a bike the day for this event
29. Data Aggregation: Hard Mode
30. Data Aggregation: Hard Mode How many users stopped near City Hall in the morning (before noon)?
31. Data Aggregation: Hard Mode How many users stopped near City Hall in the morning (before noon)? How many users stopped near the East Harriet neighborhood in the evening?
32. Data Aggregation: Hard Mode How many users stopped near City Hall in the morning (before noon)? 21 How many users stopped near the East Harriet neighborhood in the evening? 6
33. Data Aggregation: Hard Mode How many users stopped near City Hall in the morning AND near East Harriet in evening? Subscriber ID: PV9KSC9BSY
35. Privacy problem? • Wrote concise blog post, using data to tell a story • Emailed Nice Ride: Response: “We see no problem publishing that” • Tweeted:
36. Privacy problem? • Contacted advocacy organizations (ACLU, EFF) • Contacted media: ◦ City Pages
37. Data Aggregation Targeted surveillance: Focuses on a single target
38. Data Aggregation Collected data itself can identify patterns
39. Data Aggregation example
40. Data Aggregation How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did
41. Data Aggregation How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did 25+ items make up a “pregnancy prediction” score
42. Data Aggregation How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did 25+ items make up a “pregnancy prediction” score “One Target employee I spoke to provided a hypothetical example. Take a fictional Target shopper named Jenny Ward, who is 23, lives in Atlanta and in March bought cocoa-butter lotion, a purse large enough to double as a diaper bag, zinc and magnesium supplements and a bright blue rug. There’s, say, an 87 percent chance that she’s pregnant and that her delivery date is sometime in late August.”
43. Data Aggregation Collected data itself identifies patterns
44. Data Aggregation Collected data itself identifies patterns
45. Data Aggregation Collected data itself identifies patterns
46. Data Aggregation Collected data itself identifies patterns
47. Data Aggregation Data collection and information asymmetry
48. Data Aggregation Collected data itself determines targets $8/day
49. Data Aggregation Collected data itself determines targets $8/day Transaction info + ID - name, DOB, address, height, weight, etc.
50. Data Aggregation More than just financial data
51. Data Aggregation More than just financial data
52. Data Aggregation Collected data itself identifies patterns $ $ valuable data
53. Data Aggregation Collected data itself identifies patterns
54. Data Aggregation New data possibilities: Auto Insurance
55. Data Aggregation New data possibilities: Auto Insurance Make/model/year ZIP Code Age of driver Accident history Credit score
56. Data Aggregation New data possibilities: Auto Insurance GPS (where you go) Rate of speed Time of day you drive Acceleration/deceleration rate
57. Data accountability
58. Data accountability Payment rates of friends determines creditworthiness
59. Data accountability Payment rates of friends determines creditworthiness Do you read the terms? Using a VPN?
60. Data accountability Payment rates of friends determines creditworthiness Do you read the terms? Using a VPN? Different search results based on browser
61. A data accountability model
62. A data accountability model Why?
63. A data accountability model Why? People are entitled to some degree of privacy Institutions should be held accountable for what data they collect and how they use it
64. Privacy
65. Privacy
66. Personal Privacy I'm not doing anything wrong, so I have nothing to hide! "The real danger is the gradual erosion of individual liberties through automation, integration, and interconnection of many small, separate record-keeping systems, each of which alone may seem innocuous, even benevolent, and wholly justifiable." -Anon., U. S. Privacy Study Commission, 1977
68. We’ve been here before
69. We’ve been here before
70. We’ve been here before Access to data collected about yourself Ability to fix or dispute data Some insight into how data is used
71. Institutional Transparency Publicly available data can be easily found online: Minneapolis Open Data Portal Open Twin Cities Data Portal • Crime • GIS • Property Value / Taxes • Snow Emergency Routes • Neighborhood Boundaries • Local Government Finances
72. Institutional Transparency Publicly available data can be easily found online: Minneapolis Open Data Portal Open Twin Cities Data Portal State of Minnesota • Census / Demographic • Air & Water Quality • State Government Finances • Economic Data • Traffic Data (crashes) • Health Data (birth/death, immunizations)
73. Institutional Transparency Publicly available data can be easily found online: Minneapolis Open Data Portal Open Twin Cities Data Portal State of Minnesota United States (data.gov) • Hospital Charge Data • Consumer Complaint Database • Climate Data • Federal Student Loan Data • Small Business Lending Data • Airline On-Time Performance • Over 100K datasets available
74. Institutional Transparency Non-government data can be easily found online: Quandl /r/datasets /r/opendata Amazon DataCite Internet Archive
75. Institutional Transparency Much publicly available data can *NOT* be easily found online:
76. Where to find data? Much publicly available data can *NOT* be easily found online: How? Freedom of Information Act requests (templates via Muckrock) Minnesota Government Data Practices Act requests Court Records Minneapolis day-to-day police reports Minnesota Campaign Finance Disclosure Board
77. Where to find data? Lots of publicly available data can *NOT* be easily found online: This gives you, the amateur data scientist, the advantage if you’re willing to seek data out! Towing Data: Metro Transit Bus Shelter Data:
78. Questions? @anton612