Break It Down: A Comparison of Macro- and Microtasks

Presented at CHI 2015.

A large, seemingly overwhelming task can sometimes be transformed into a set of smaller, more manageable microtasks that can each be accomplished independently. In crowdsourcing systems, microtasking enables unskilled workers with limited commitment to work together to complete tasks they would not be able to do individually. We explore the costs and benefits of decomposing macrotasks into microtasks for three task categories: arithmetic, sorting, and transcription. We find that breaking these tasks into microtasks results in longer overall task completion times, but higher quality outcomes and a better experience that may be more resilient to interruptions. These results suggest that microtasks can help people complete high quality work in interruption-driven environments.

Transcript

1. Justin Cheng jcccf@cs.stanford.edu 1 A Comparison of Macro- and Microtasks Jaime Teevan teevan@microsoft.com Shamsi Iqbal shamsi@microsoft.com Michael Bernstein msb@cs.stanford.edu Break it Down
2. A Comparison of Macro- and Microtasks
3. A Comparison of Macro- and Microtasks Picking the best photos for an album
4. A Comparison of Macro- and Microtasks Large
5. A Comparison of Macro- and Microtasks Monolithic
6. A Comparison of Macro- and Microtasks Overwhelming
7. A Comparison of Macro- and Microtasks Pairwise comparisons of photos
8. A Comparison of Macro- and Microtasks Smaller
9. A Comparison of Macro- and Microtasks Done in seconds
10. A Comparison of Macro- and Microtasks Done independently
11. A Comparison of Macro- and Microtasks Transcribing a speech Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.
12. A Comparison of Macro- and Microtasks Transcribing individual sentences Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit. Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet. Transcribing a speech
13. Are macrotasks or microtasks better?
14. How do macrotasks and microtasks differ?
15. People already break tasks down.
16. Tasks are perceived in segments. Zacks, J. M., Tversky, B., and Iyer, G. Perceiving,remembering, and communicating structure in events. In Exp. Psychol. Gen. 2001. Wickens, C. D. Multiple resources and performance prediction. In Theor. Issues Ergon. 2002. =
17. Many tasks are done in short bursts. Gonzalez, V. M., and Mark, G. Constant, constant,multi-tasking craziness: managing multiple working spheres. In CHI 2004.
18. They are constantly interrupted. Gonzalez, V. M., and Mark, G. Constant, constant,multi-tasking craziness: managing multiple working spheres. In CHI 2004. Hey, quick question… Have a minute?
19. Microtasking is prevalent in crowdsourcing.
20. Microtasking can help us be productive in spare moments of time. Vaish, R., et al. Twitch crowdsourcing: crowd contributions in short bursts of time. In CHI 2014.
21. How do macrotasks and microtasks differ in terms of…?
22. How do macrotasks and microtasks differ in terms of time taken?
23. How do macrotasks and microtasks differ in terms of work quality?
24. How do macrotasks and microtasks differ in terms of interruptibility?
25. How do macrotasks and microtasks differ in terms of worker preference?
26. × Task Format (Macrotask/Microtask) Presence of Interruptions (Interruptions/No Interruptions) Arithmetic Sorting Transcription 3 Tasks 110 Participants on Clickworker (+MTurk)
27. Arithmetic Macrotask and Microtask
28. Sorting Macrotask and Microtask
29. Sorting Macrotask and Microtask
30. Transcription Macrotask and Microtask
31. How do macrotasks and microtasks differ in terms of time taken?
32. Microtasks generally take longer Task Time (s) 0 50 100 150 200 Arithmetic Sorting Transcription Macrotask Microtask p<0.001
33. How do macrotasks and microtasks differ in terms of work quality?
34. Microtasks result in fewer errors Error Rate 0 0.1 0.2 0.3 0.4 Arithmetic Sorting Transcription Macrotask Microtask p<0.001
35. take more time fewer errors take less time more errors Macrotasks Microtasks … …
36. How do macrotasks and microtasks differ in terms of interruptibility?
37. Macrotasks more susceptible to interruptions Task Time w/o Interruptions 0 20 40 60 80 100 120 140 Uninterrupted Interrupted Macrotask Microtask p<0.10
38. Microtasks less susceptible to interruptions Task Time w/o Interruptions 0 20 40 60 80 100 120 140 Uninterrupted Interrupted Macrotask Microtask p<0.10
39. How do macrotasks and microtasks differ in terms of worker preference?
40. 77% of workers preferred microtasks to macrotasks.
41. take more time fewer errors take less time more errors unaffected by interruptions preferred more slowed by interruptions preferred less Macrotasks Microtasks
42. Information workers can benefit from microtasking. (Can we write research papers on our phones?)
43. Information workers can benefit from microtasking. (Can turkers benefit from macrotasking?)
44. 44 Break it Down A Comparison of Macro- and Microtasks Justin Cheng jcccf@cs.stanford.edu Jaime Teevan teevan@microsoft.com Shamsi Iqbal shamsi@microsoft.com Michael Bernstein msb@cs.stanford.edu