Break It Down: A Comparison of Macro- and Microtasks

Break It Down: A Comparison of Macro- and Microtasks

Presented at CHI 2015.

A large, seemingly overwhelming task can sometimes be transformed into a set of smaller, more manageable microtasks that can each be accomplished independently. In crowdsourcing systems, microtasking enables unskilled workers with limited commitment to work together to complete tasks they would not be able to do individually. We explore the costs and benefits of decomposing macrotasks into microtasks for three task categories: arithmetic, sorting, and transcription. We find that breaking these tasks into microtasks results in longer overall task completion times, but higher quality outcomes and a better experience that may be more resilient to interruptions. These results suggest that microtasks can help people complete high quality work in interruption-driven environments.

8480b47e733a040fba07c32da414b0e0?s=128

Justin Cheng

April 23, 2015
Tweet

Transcript

  1. 1.

    Justin Cheng jcccf@cs.stanford.edu 1 A Comparison of Macro- and Microtasks

    Jaime Teevan teevan@microsoft.com Shamsi Iqbal shamsi@microsoft.com Michael Bernstein msb@cs.stanford.edu Break it Down
  2. 11.

    A Comparison of Macro- and Microtasks Transcribing a speech Lorem

    ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.
  3. 12.

    A Comparison of Macro- and Microtasks Transcribing individual sentences Lorem

    ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit. Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet. Transcribing a speech
  4. 16.

    Tasks are perceived in segments. Zacks, J. M., Tversky, B.,

    and Iyer, G. Perceiving,remembering, and communicating structure in events. In Exp. Psychol. Gen. 2001. Wickens, C. D. Multiple resources and performance prediction. In Theor. Issues Ergon. 2002. =
  5. 17.

    Many tasks are done in short bursts. Gonzalez, V. M.,

    and Mark, G. Constant, constant,multi-tasking craziness: managing multiple working spheres. In CHI 2004.
  6. 18.

    They are constantly interrupted. Gonzalez, V. M., and Mark, G.

    Constant, constant,multi-tasking craziness: managing multiple working spheres. In CHI 2004. Hey, quick question… Have a minute?
  7. 20.

    Microtasking can help us be productive in spare moments of

    time. Vaish, R., et al. Twitch crowdsourcing: crowd contributions in short bursts of time. In CHI 2014.
  8. 26.

    × Task Format (Macrotask/Microtask) Presence of Interruptions (Interruptions/No Interruptions) Arithmetic

    Sorting Transcription 3 Tasks 110 Participants on Clickworker (+MTurk)
  9. 32.

    Microtasks generally take longer Task Time (s) 0 50 100

    150 200 Arithmetic Sorting Transcription Macrotask Microtask p<0.001
  10. 34.

    Microtasks result in fewer errors Error Rate 0 0.1 0.2

    0.3 0.4 Arithmetic Sorting Transcription Macrotask Microtask p<0.001
  11. 35.
  12. 37.

    Macrotasks more susceptible to interruptions Task Time w/o Interruptions 0

    20 40 60 80 100 120 140 Uninterrupted Interrupted Macrotask Microtask p<0.10
  13. 38.

    Microtasks less susceptible to interruptions Task Time w/o Interruptions 0

    20 40 60 80 100 120 140 Uninterrupted Interrupted Macrotask Microtask p<0.10
  14. 41.

    take more time fewer errors take less time more errors

    unaffected by interruptions preferred more slowed by interruptions preferred less Macrotasks Microtasks
  15. 44.

    44 Break it Down A Comparison of Macro- and Microtasks

    Justin Cheng jcccf@cs.stanford.edu Jaime Teevan teevan@microsoft.com Shamsi Iqbal shamsi@microsoft.com Michael Bernstein msb@cs.stanford.edu