Survey Research in the Digital Age

Abstract: In the past several years, we have witnessed the birth and rapid spread of social media, smart phones, and numerous other digital marvels. In addition to changing how we live, these tools enable us to collect and process data about human behavior on a scale never before imaginable. In this talk, I’ll describe how survey research fits into this new data landscape. Further, I’ll use specific examples to illustrate how survey researchers can harness the tools of the digital age to collect data in new ways. Throughout the talk I will emphasize ways that big data sources and surveys can serve as compliments rather than substitutes.

These slides are from a webinar organized by the American Association of Public Opinion Research (AAPOR). More information: https://www.aapor.org/Education-Resources/Online-Education/Webinar-Details.aspx?webinar=WEB0219

Transcript

1. Survey research in the digital age Matthew J. Salganik Department of Sociology Princeton University msalganik AAPOR Webinar February 21, 2019
3. Isn’t “big data” a fad?
4. https://commons.wikimedia.org/wiki/File:Gartner_Hype_Cycle.svg
5. Key abstraction is research design
6. Four main research designs:
7. Four main research designs: Observing behavior
8. Four main research designs: Observing behavior Asking questions
9. Four main research designs: Observing behavior Asking questions Running experiments
10. Four main research designs: Observing behavior Asking questions Running experiments Creating mass collaboration
11. Four main research designs: Observing behavior Asking questions Running experiments Creating mass collaboration
12. Social Scientists ←→ Data Scientists
14. Readymades
15. Readymades
16. Readymades Custommades
17. + Readymades Custommades https://commons.wikimedia.org/wiki/File:Duchamp_Fountaine.jpg https://commons.wikimedia.org/wiki/File:%27David%27_by_Michelangelo_JBU0001.JPG
25. Blumenstock et al. (2015), Figure 2
26. Readymade + Custommade
27. Readymade + Custommade Custommade
28. Readymade + Custommade Custommade 10 times faster 50 times cheaper Blumenstock et al. (2015), Figure 3
29. + Readymades Custommades https://commons.wikimedia.org/wiki/File:Duchamp_Fountaine.jpg https://commons.wikimedia.org/wiki/File:%27David%27_by_Michelangelo_JBU0001.JPG
30. Created by Kedron Rhodes
31. Why should I care about surveys in the age of big data?
32. We will always need to ask limitations of big data (fubu vs. nufu-nubu)
33. We will always need to ask limitations of big data (fubu vs. nufu-nubu) internal states vs. external states
34. We will always need to ask limitations of big data (fubu vs. nufu-nubu) internal states vs. external states inaccessibility of big data
35. We will always need to ask limitations of big data (fubu vs. nufu-nubu) internal states vs. external states inaccessibility of big data But how we are going to ask is going to change
36. Sampling Interviews 1st era Area probability Face-to-face
37. Sampling Interviews 1st era Area probability Face-to-face 2nd era Random digital dial probability Telephone
38. Sampling Interviews 1st era Area probability Face-to-face 2nd era Random digital dial probability Telephone 3rd era
39. Sampling Interviews 1st era Area probability Face-to-face 2nd era Random digital dial probability Telephone 3rd era Non-probability Computer-administered
40. Sampling Interviews Data environment 1st era Area probability Face-to-face Stand-alone 2nd era Random digital dial probability Telephone Stand-alone 3rd era Non-probability Computer-administered Linked
41. Sampling Interviews Data environment 1st era Area probability Face-to-face Stand-alone 2nd era Random digital dial probability Telephone Stand-alone 3rd era Non-probability Computer-administered Linked
42. Sampling Interviews Data environment 1st era Area probability Face-to-face Stand-alone 2nd era Random digital dial probability Telephone Stand-alone 3rd era Non-probability Computer-administered Linked
43. https://www.chicagotribune.com/news/opinion/commentary/ ct-truman-defeats-dewey-1948-flashback-perspec-1113-md-20161111-story.html
47. Sampling Interviews Data environment 1st era Area probability Face-to-face Stand-alone 2nd era Random digital dial probability Telephone Stand-alone 3rd era Non-probability Computer-administered Linked
48. Human-administered → computer-administered enables change requires change
49. https://doi.org/10.1371/journal.pone.0123483
50. http://kittenwar.com
51. http://kittenwar.com
52. http://kittenwar.com
55. quantification or openness
56. quantification + openness = wiki surveys
58. General principles of wiki surveys: greedy
59. Good web-based systems use the fat-head and the long-tail Zero Lots Information Contributed Contributors (sorted by rank) Contributes Most Contributes Least
60. Surveys don’t use the fat-head or the long-tail Zero Lots Information Contributed Contributors (sorted by rank) Contributes Most Contributes Least
61. General principles of wiki surveys: greedy
62. General principles of wiki surveys: greedy collaborative
63. General principles of wiki surveys: greedy collaborative adaptive
66. Which do you think is a better idea for creating a greener, greater New York City? Seeded the wiki survey with 25 ideas: Require all big buildings to make certain energy efficiency upgrades Increase targeted tree plantings in neighborhoods with high asthma rates Establish a New York City Energy Planning Board
71. What are we trying to estimate? Data Vote Session Prompt 1 1 item 4 item 1 2 1 item 3 item 1 3 1 item 4 item 3 4 2 item 3 item 4 5 2 item 4 item 2 . . . . . . . . . . . . Opinion matrix      θ1,1 θ1,2 . . . θ1,K θ2,1 θ2,2 . . . θ2,K . . . . . . ... . . . θJ,1 θJ,2 . . . θJ,K      θj,k: how much respondent j likes item k
72. Which do you think is a better idea for creating a greener, greater New York City? Seeded the wiki survey with 25 ideas: Require all big buildings to make certain energy efficiency upgrades Increase targeted tree plantings in neighborhoods with high asthma rates Establish a New York City Energy Planning Board
73. Recruited participants through Twitter, Facebook, blogs, etc. This is not a random sample, but random samples are possible
74. 31,893 responses 464 ideas uploaded q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q Which do you think is a better idea for creating a greener, greater New York City? Rank of session Responses 1 500 1000 1 200 400 600 800 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 1 40 80 120 160 200 240 1 10 20 30 40 50 Rank of session Contributed ideas
75. 60 65 70 75 80 85 90 Which do you think is a better idea for creating a greener, greater New York City? Score, si ^ q Provide better transit service outside of Manhattan q Create a network of protected bike paths throughout the entire city q Support and protect community gardens and create mechanisms to create new gardens and open space q Implement congestion pricing in lower Manhattan q Require all big buildings to make certain energy efficiency upgrades q Create more year−round Greenmarkets in under−served communities. q Continue enhancing bike lane network, to finally connect separated bike lane systems to each other across all five boroughs. q Plug ships into electricity grid so they don't idle in port − reducing emissions equivalent to 12000 cars per ship. q Invest in multiple modes of transportation and provide both improved infrastructure and improved safety q Keep NYC's drinking water clean by banning fracking in NYC's watershed.
76. Alternative framings: “Keep NYC’s drinking water clean by banning fracking in NYC’s watershed”
77. Alternative framings: “Keep NYC’s drinking water clean by banning fracking in NYC’s watershed” Novel information: “Plug ships into electricity grid so they don’t idle in port - reducing emissions equivalent to 12000 cars per ship.”
78. q q q q q q q q q q q q q q q q q q q q q q q q q Seed ideas Score, si ^ Rank 1 25 10 20 30 40 50 60 70 80 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q User−contributed ideas Rank 1 50 100 150 200 244 10 20 30 40 50 60 70 80
79. q q q q q q q q q q q q q q q q q q q q q q q q q Seed ideas Score, si ^ Rank 1 25 10 20 30 40 50 60 70 80 q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q User−contributed ideas Rank 1 50 100 150 200 244 10 20 30 40 50 60 70 80 variance + volume → extreme cases
80. Currently hosting: 14,000 wiki surveys with 700,000 ideas and 21 million votes
81. improved allourideas.org users research
84. Sampling Interviews Data environment 1st era Area probability Face-to-face Stand-alone 2nd era Random digital dial probability Telephone Stand-alone 3rd era Non-probability Computer-administered Linked
85. Sampling Interviews Data environment 1st era Area probability Face-to-face Stand-alone 2nd era Random digital dial probability Telephone Stand-alone 3rd era Non-probability Computer-administered Linked
86. Will big data kill surveys?
87. http://schlitterblog.com/wp-content/uploads/2014/05/peanutbutterlover.jpg
88. http://schlitterblog.com/wp-content/uploads/2014/05/peanutbutterlover.jpg
89. Note the different role of the big data in each case
90. http://dx.doi.org/10.1126/science.aac4420
94. The beginning is not the end . . . .
96. http://dx.doi.org/10.1126/science.aaf7894 https://motherboard.vice.com/en_us/article/artificial-intelligence-is-predicting-human-poverty-from-space
97. Daytime satellite images are available, but most researchers had been using night lights https://www.nasa.gov/multimedia/imagegallery/image_feature_2480.html
98. Prior research: Nightlights + survey data → estimates of wealth in places without surveys
99. Jean et al. (2016): Day pictures + Nightlights + survey data → estimates of wealth in places without surveys
100. Start with CNN pretrained on ImageNet
101. Start with CNN pretrained on ImageNet Train CNN to predict nightlights from day pictures (lots of training data)
102. Start with CNN pretrained on ImageNet (e.g. hampsters and weasels) Train CNN to predict nightlights from day pictures (lots of training data) Take features from CNN and train ridge regression to predict cluster mean survey response http://dx.doi.org/10.1126/science.aah5217
103. http://dx.doi.org/10.1126/science.aaf7894
104. Two patterns: Performance decreases when train on one country and test on another Performance varies by the quantity being estimated (assets seems easier to estimate than consumption expenditures) http://dx.doi.org/10.1126/science.aaf7894
105. https://github.com/nealjean/predicting-poverty
106. Sampling Interviews Data environment 1st era Area probability Face-to-face Stand-alone 2nd era Random digital dial probability Telephone Stand-alone 3rd era Non-probability Computer-administered Linked
107. Sampling Interviews Data environment 1st era Area probability Face-to-face Stand-alone 2nd era Random digital dial probability Telephone Stand-alone 3rd era Non-probability Computer-administered Linked
108. https://commons.wikimedia.org/wiki/File:Gartner_Hype_Cycle.svg
109. Read: http://www.bitbybitbook.com Teach: http://www.bitbybitbook.com/en/teaching/