DST4L 2015 - Ch1

Transcript

1. Blogging, Data-Driven Questions, and The Quantified Self James R. A. Davenport #DST4L (Feb 6, 2015)
2. What do I want to cover today? • About me, my blog, my varied interests • The two methods for coming up with blog posts • Visualization & it’s role in asking questions • What is the Quantified Self? • Hands-on examples of my analysis • Discussion: the role of a library for the modern data-driven researcher?
3. Introduction
4. ifweassume.com @jradavenport
5. ifweassume.com @jradavenport
8. Kepler: Stellar Flare Machine • Long continuous light curves
 (up to ~4years) • Very precise photometry
 (~0.01%) • Enormous sample
 (>100,000 solar-type stars) • Look for outliers (super-flares) • Complete samples! 8
9. For research, make lots and lots and lots of graphs…
11. My Blog An outlet for (nerdy) creativity
12. ifweassume.com @jradavenport
15. 597 AD
16. A bot to Tweet the entirety of “The Adventures of Tom Sawyer” over a year
17. $25 RaspberryPi computer
22. The US population - by longitude
24. Made possible by openly sharing data (and using sensible standards)
25. My application made available 
 thanks to open tools
26. data questions Data Science
27. 2 methods of idea generation • 1: have Q, look for data to answer it • more “scientific” • 2: have interesting data, look for a Q to ask
 or story to tell using it • I find this more common in practice now
28. 1. have Q, need data data questions Data Science
29. Q: When is the best time to find an apartment in Seattle?
30. Answer: ummm….
31. Q: What are the most common first names for lawyers vs scientists?
32. Q: What are the most common first names for lawyers vs scientists? Tough to get data on this! (have to steal it)
33. Scrape data… Scientists: AAAS, and AAS Lawyers: Bar Exam Pass Lists
34. Scrape data… can literally be COPY/PASTE or can be more programatic
38. Another source of first names: The Social Security Administration’s 
 “Baby Names” dataset Every first name (and gender) for SSA applicants 1890’s - Present
40. Q: How long do Popes usually serve?
43. Q: Are students using the UW libraries?
44. short answer: yes Gate Counts per Month
45. Gate Counts per Month short answer: yes
46. But, this question is less simple than it seems…
47. All Items Checkouts
48. But, enrollment has been increasing rapidly
50. Q: Where are the worst (in terms of hygiene) restaurants in Seattle?
53. There are more health inspections in the spring/fall, fewer in the summer
54. Q: Just how covered is the UW Seattle campus in coffee shops, anyhow?
55. Ingredients… • a UW Seattle campus map • locations of all coffee shops • average human walking speed
58. Coffee:! Always ! within a ! 2 minute! walk! 2 min ! 1 min ! 5! Coffee: Always within a 2 minute walk
59. ifweassume.com @jradavenport
60. United States of Starbucks ifweassume.com @jradavenport
61. furthest point from a Starbucks ifweassume.com @jradavenport
62. data from 2010 US Census ifweassume.com @jradavenport
63. 80% of US lives within 20 miles of a Starbucks ifweassume.com @jradavenport
64. Visualization makes technical ideas accessible to everyone ifweassume.com @jradavenport #thinkeyes2014
65. All FM radio station transmitters in the US
66. Neat map, but what else can we do with it?
67. Stations monitored by Nielsen
68. Stations monitored by Nielsen
69. Stations monitored by Nielsen
70. Stations monitored by Nielsen
71. 2. have data, need Q data questions Data Science
72. Airports of the World data from ourairports.com airports, runways, helipads, landing strips ifweassume.com @jradavenport
73. Population Density ifweassume.com @jradavenport
74. Population Density Wealth
 Inequality? ifweassume.com @jradavenport
75. ifweassume.com @jradavenport
76. DEW Line ifweassume.com
77. ifweassume.com @jradavenport
78. Bermuda Cuba Haiti Dominican Republic Puerto Rico Jamaica Cayman Islands Bahamas Cancún Sable Island Nat’l Park ifweassume.com @jradavenport
79. People engage with dynamic range & detail ifweassume.com @jradavenport
80. People engage with dynamic range & detail Encourage viewer to discover for themselves ifweassume.com @jradavenport
81. ifweassume.com @jradavenport
82. Spark conversations about sample bias, correlation, causation… ifweassume.com @jradavenport
83. Sometimes best question I can ask is: “what does it look like?” data questions Data Science
84. Neat data set: Chris Erdmann compiled data on NSF grants & Astronomy papers
86. (average)
87. $249k (median)
90. Is this an OK thing to publish?
92. Often I find a data source so interesting that I 
 look hard for a “story”
94. Problems with reading CSV files (comma separated values) a short aside
95. Problems with reading CSV files (comma separated values) a, b, c, d e, f, g, h 1, 2, , 4 5, 6, 7, 8 red, blue, green, black up, down, left, right a, b, b, a Easy to read:
96. Problems with reading CSV files (comma separated values) a, b, c, d e, f, g, h 1, 2, “Bob, or Mary”, 4 5, 6, 7, 8 red, blue, green, black up, down, left, right a, b, b, a Hard to read:
97. Problems with reading CSV files (comma separated values) a, b, c, d e, f, g, h 1, 2, “Bob, or Mary”, 4 5, 6, 7, 8 red, blue, green, black up, down, left, right a, b, b, a Hard to read:
99. The Dimensions of Art 65,000 pieces of art from the Tate Modern Width Height Width ifweassume.com @jradavenport
100. ifweassume.com @jradavenport The Dimensions of Art 65,000 pieces of art from the Tate Modern
101. Need to balance complexity & beauty ifweassume.com @jradavenport
102. Let’s try playing with this data. On to the IPython Notebook!
103. One last example of this Data Science data
106. 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Top 10 Best-Seller Book Covers Over Time data from USA Today ifweassume.com @jradavenport
107. 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2005 2006 2007 2008 2009 2004 2003 20 200 ifweassume.com @jradavenport
108. Feb 2008 Aug 2008 ifweassume.com @jradavenport
109. Feb 2008 Aug 2008 ifweassume.com @jradavenport
110. Feb 2008 Aug 2008 ifweassume.com @jradavenport
111. Feb 2008 Aug 2008 ifweassume.com @jradavenport
112. Feb 2008 Aug 2008 ifweassume.com @jradavenport
113. Feb 2008 Aug 2008 ifweassume.com @jradavenport
114. Feb 2008 ifweassume.com @jradavenport
115. 2008 2009 ifweassume.com @jradavenport
116. 2012 ifweassume.com @jradavenport
117. The Average Top-10 Best-Selling Book Cover *From 2000-2011 ifweassume.com @jradavenport
118. Best-Seller Brown ifweassume.com @jradavenport
119. ifweassume.com @jradavenport The Top-10 of the Top-10
120. ifweassume.com @jradavenport
121. ifweassume.com @jradavenport
122. Cool visualizations, but now have more questions than when I started! e.g. genre, average color vs rank, average color over time… ifweassume.com @jradavenport