Break It Down: A Comparison of Macro- and Microtasks

Break It Down: A Comparison of Macro- and Microtasks

Presented at CHI 2015.

A large, seemingly overwhelming task can sometimes be transformed into a set of smaller, more manageable microtasks that can each be accomplished independently. In crowdsourcing systems, microtasking enables unskilled workers with limited commitment to work together to complete tasks they would not be able to do individually. We explore the costs and benefits of decomposing macrotasks into microtasks for three task categories: arithmetic, sorting, and transcription. We find that breaking these tasks into microtasks results in longer overall task completion times, but higher quality outcomes and a better experience that may be more resilient to interruptions. These results suggest that microtasks can help people complete high quality work in interruption-driven environments.

8480b47e733a040fba07c32da414b0e0?s=128

Justin Cheng

April 23, 2015
Tweet

Transcript

  1. Justin Cheng jcccf@cs.stanford.edu 1 A Comparison of Macro- and Microtasks

    Jaime Teevan teevan@microsoft.com Shamsi Iqbal shamsi@microsoft.com Michael Bernstein msb@cs.stanford.edu Break it Down
  2. A Comparison of Macro- and Microtasks

  3. A Comparison of Macro- and Microtasks Picking the best photos

    for an album
  4. A Comparison of Macro- and Microtasks Large

  5. A Comparison of Macro- and Microtasks Monolithic

  6. A Comparison of Macro- and Microtasks Overwhelming

  7. A Comparison of Macro- and Microtasks Pairwise comparisons of photos

  8. A Comparison of Macro- and Microtasks Smaller

  9. A Comparison of Macro- and Microtasks Done in seconds

  10. A Comparison of Macro- and Microtasks Done independently

  11. A Comparison of Macro- and Microtasks Transcribing a speech Lorem

    ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.
  12. A Comparison of Macro- and Microtasks Transcribing individual sentences Lorem

    ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit.Lorem ipsum dolor sit amet, consectetur adipiscing elit. Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet. Transcribing a speech
  13. Are macrotasks or microtasks better?

  14. How do macrotasks and microtasks differ?

  15. People already break tasks down.

  16. Tasks are perceived in segments. Zacks, J. M., Tversky, B.,

    and Iyer, G. Perceiving,remembering, and communicating structure in events. In Exp. Psychol. Gen. 2001. Wickens, C. D. Multiple resources and performance prediction. In Theor. Issues Ergon. 2002. =
  17. Many tasks are done in short bursts. Gonzalez, V. M.,

    and Mark, G. Constant, constant,multi-tasking craziness: managing multiple working spheres. In CHI 2004.
  18. They are constantly interrupted. Gonzalez, V. M., and Mark, G.

    Constant, constant,multi-tasking craziness: managing multiple working spheres. In CHI 2004. Hey, quick question… Have a minute?
  19. Microtasking is prevalent in crowdsourcing.

  20. Microtasking can help us be productive in spare moments of

    time. Vaish, R., et al. Twitch crowdsourcing: crowd contributions in short bursts of time. In CHI 2014.
  21. How do macrotasks and microtasks differ in terms of…?

  22. How do macrotasks and microtasks differ in terms of time

    taken?
  23. How do macrotasks and microtasks differ in terms of work

    quality?
  24. How do macrotasks and microtasks differ in terms of interruptibility?

  25. How do macrotasks and microtasks differ in terms of worker

    preference?
  26. × Task Format (Macrotask/Microtask) Presence of Interruptions (Interruptions/No Interruptions) Arithmetic

    Sorting Transcription 3 Tasks 110 Participants on Clickworker (+MTurk)
  27. Arithmetic Macrotask and Microtask

  28. Sorting Macrotask and Microtask

  29. Sorting Macrotask and Microtask

  30. Transcription Macrotask and Microtask

  31. How do macrotasks and microtasks differ in terms of time

    taken?
  32. Microtasks generally take longer Task Time (s) 0 50 100

    150 200 Arithmetic Sorting Transcription Macrotask Microtask p<0.001
  33. How do macrotasks and microtasks differ in terms of work

    quality?
  34. Microtasks result in fewer errors Error Rate 0 0.1 0.2

    0.3 0.4 Arithmetic Sorting Transcription Macrotask Microtask p<0.001
  35. take more time fewer errors take less time more errors

    Macrotasks Microtasks … …
  36. How do macrotasks and microtasks differ in terms of interruptibility?

  37. Macrotasks more susceptible to interruptions Task Time w/o Interruptions 0

    20 40 60 80 100 120 140 Uninterrupted Interrupted Macrotask Microtask p<0.10
  38. Microtasks less susceptible to interruptions Task Time w/o Interruptions 0

    20 40 60 80 100 120 140 Uninterrupted Interrupted Macrotask Microtask p<0.10
  39. How do macrotasks and microtasks differ in terms of worker

    preference?
  40. 77% of workers preferred microtasks to macrotasks.

  41. take more time fewer errors take less time more errors

    unaffected by interruptions preferred more slowed by interruptions preferred less Macrotasks Microtasks
  42. Information workers can benefit from microtasking. (Can we write research

    papers on our phones?)
  43. Information workers can benefit from microtasking. (Can turkers benefit from

    macrotasking?)
  44. 44 Break it Down A Comparison of Macro- and Microtasks

    Justin Cheng jcccf@cs.stanford.edu Jaime Teevan teevan@microsoft.com Shamsi Iqbal shamsi@microsoft.com Michael Bernstein msb@cs.stanford.edu