Make better surveys

Good surveys are deceptively hard to get right, here's some tips to make yours better. This was a talk I presented at @uxbri.

Transcript

1. Make better surveys Jeremy Rosenberg @jeremy74
2. @jeremy74 Surveys are hard ̣ Face to face conversations are rich interactions ̣ We seek feedback and modulate our responses ̣ We have a good idea when there is shared understanding ̣ Surveys are rigid and prone to misunderstanding
3. @jeremy74 Be clear about the purpose ̣ What will you do with the information ̣ Clear about what you want to find out ̣ Hypotheses
4. @jeremy74 Put boring questions at the end ̣ Improves completion rates ̣ Completion tendency/sunk cost ̣ Still get partial info for drop outs Roberson, M.T. & Sundstrom, E. (1990). Questionnaire design, return rates, and response favorableness in an employee attitude questionnaire. Journal of Applied Psychology, 75, 354-357. Martin, J.D. & McConnell, J.P. (1970). Mail questionnaire response induction: the effect of four variables on the response of a random sample to a difficult questionnaire. Social Science Quarterly, 51, 409-414.
5. @jeremy74 Be clear/specific ̣ “How often do you cook?”
6. @jeremy74 Be clear/specific ̣ “How often do you cook?” ̣ “How often do you prepare a meal?”
7. @jeremy74 Be clear/specific ̣ “How often do you cook?” ̣ “How often do you prepare a meal?” ̣ “How often do you prepare a meal, excluding ready-meals”
8. @jeremy74 Be clear/specific ̣ “How often do you cook?” ̣ “How often do you prepare a meal?” ̣ “How often do you prepare a meal, excluding ready-meals” ̣ “How often do you prepare an evening meal, excluding ready meals?”
9. @jeremy74 Be clear/specific ̣ “How many times did you prepare an evening meal, excluding ready-meals in the last 7 days?”
10. @jeremy74 Be clear/specific ̣ “How many times did you prepare an evening meal, excluding ready-meals in the last 7 days?” ̣ “Did you prepare any evening meals, excluding ready-meals, in the last 7 days?” then, if so, ̣ “How many did you prepare?”
11. @jeremy74 Subjective questions are unreliable ̣ Like usability testing, behaviour is more reliable than attitudes/preferences Bertrand, M. & Mullainathan, S. (2001). Do people mean what they say? Implications for subjective survey data. American Economic Review, 91, 67-72.
12. @jeremy74 Anticipate biases ̣ “Is sustainable seafood important to you” versus ̣ “Are you interested in any of the following?”
13. @jeremy74 Other things to get right ̣ Avoid leading questions ̣ Avoid double barrelled questions ̣ Avoid absolutes ̣ Make the questions answerable e.g. include ‘other’ as an answer option ̣ Target the appropriate audience ̣ Reduce reliance on memory ̣ Avoid questions about the future
14. @jeremy74 Other things to get right ̣ Consider cost/benefit of open ended questions ̣ Consider a pilot study ̣ Randomise question order, and answer options, where possible ̣ Don’t waste Q’s on ones better answered elsewhere, like analytics ̣ Map it out if it’s complicated
15. @jeremy74
16. @jeremy74 Thanks! Will Tweet link to slides @jeremy74