Break It Down: A Comparison of Macro- and Microtasks

Presented at CHI 2015.

A large, seemingly overwhelming task can sometimes be transformed into a set of smaller, more manageable microtasks that can each be accomplished independently. In crowdsourcing systems, microtasking enables unskilled workers with limited commitment to work together to complete tasks they would not be able to do individually. We explore the costs and benefits of decomposing macrotasks into microtasks for three task categories: arithmetic, sorting, and transcription. We find that breaking these tasks into microtasks results in longer overall task completion times, but higher quality outcomes and a better experience that may be more resilient to interruptions. These results suggest that microtasks can help people complete high quality work in interruption-driven environments.

Transcript

1. Justin Cheng
   jcccf@cs.stanford.edu
   1
   A Comparison of Macro- and Microtasks
   Jaime Teevan
   teevan@microsoft.com
   Shamsi Iqbal
   shamsi@microsoft.com
   Michael Bernstein
   msb@cs.stanford.edu
   Break it
   Down
   

   View Slide

2. A Comparison of Macro- and Microtasks
   

   View Slide

3. A Comparison of Macro- and Microtasks
   Picking the best photos for an album
   

   View Slide

4. A Comparison of Macro- and Microtasks
   Large
   

   View Slide

5. A Comparison of Macro- and Microtasks
   Monolithic
   

   View Slide

6. A Comparison of Macro- and Microtasks
   Overwhelming
   

   View Slide

7. A Comparison of Macro- and Microtasks
   Pairwise comparisons of photos
   

   View Slide

8. A Comparison of Macro- and Microtasks
   Smaller
   

   View Slide

9. A Comparison of Macro- and Microtasks
   Done in seconds
   

   View Slide

10. A Comparison of Macro- and Microtasks
    Done independently
    

    View Slide

11. A Comparison of Macro- and Microtasks
    Transcribing a speech
    Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.Lorem ipsum dolor
    sit amet, consectetur adipiscing
    elit.Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.Lorem ipsum dolor
    sit amet, consectetur adipiscing
    elit.Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.
    

    View Slide

12. A Comparison of Macro- and Microtasks
    Transcribing individual sentences
    Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.Lorem ipsum dolor
    sit amet, consectetur adipiscing
    elit.Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.Lorem ipsum dolor
    sit amet, consectetur adipiscing
    elit.Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.
    Lorem
    ipsum dolor
    sit amet.
    Lorem
    ipsum dolor
    sit amet.
    Lorem
    ipsum dolor
    sit amet.
    Transcribing a speech
    

    View Slide

13. Are macrotasks or microtasks better?
    

    View Slide

14. How do macrotasks and microtasks differ?
    

    View Slide

15. People already break tasks down.
    

    View Slide

16. Tasks are perceived in segments.
    Zacks, J. M., Tversky, B., and Iyer, G. Perceiving,remembering, and communicating structure in events. In Exp. Psychol. Gen. 2001.
    Wickens, C. D. Multiple resources and performance prediction. In Theor. Issues Ergon. 2002.
    =
    

    View Slide

17. Many tasks are done in short bursts.
    Gonzalez, V. M., and Mark, G. Constant, constant,multi-tasking craziness: managing multiple working spheres. In CHI 2004.
    

    View Slide

18. They are constantly interrupted.
    Gonzalez, V. M., and Mark, G. Constant, constant,multi-tasking craziness: managing multiple working spheres. In CHI 2004.
    Hey, quick
    question…
    Have a minute?
    

    View Slide

19. Microtasking is prevalent in
    crowdsourcing.
    

    View Slide

20. Microtasking can help us be productive
    in spare moments of time.
    Vaish, R., et al. Twitch crowdsourcing: crowd contributions in short bursts of time. In CHI 2014.
    

    View Slide

21. How do macrotasks and microtasks
    differ in terms of…?
    

    View Slide

22. How do macrotasks and microtasks
    differ in terms of time taken?
    

    View Slide

23. How do macrotasks and microtasks
    differ in terms of work quality?
    

    View Slide

24. How do macrotasks and microtasks
    differ in terms of interruptibility?
    

    View Slide

25. How do macrotasks and microtasks
    differ in terms of worker preference?
    

    View Slide

26. ×
    Task Format
    (Macrotask/Microtask)
    Presence of Interruptions
    (Interruptions/No Interruptions)
    Arithmetic Sorting Transcription
    3 Tasks
    110 Participants on Clickworker (+MTurk)
    

    View Slide

27. Arithmetic Macrotask and Microtask
    

    View Slide

28. Sorting Macrotask and Microtask
    

    View Slide

29. Sorting Macrotask and Microtask
    

    View Slide

30. Transcription Macrotask and Microtask
    

    View Slide

31. How do macrotasks and microtasks differ
    in terms of time taken?
    

    View Slide

32. Microtasks generally take longer
    Task Time (s)
    0
    50
    100
    150
    200
    Arithmetic Sorting Transcription
    Macrotask
    Microtask
    p<0.001
    

    View Slide

33. How do macrotasks and microtasks differ
    in terms of work quality?
    

    View Slide

34. Microtasks result in fewer errors
    Error Rate
    0
    0.1
    0.2
    0.3
    0.4
    Arithmetic Sorting Transcription
    Macrotask
    Microtask
    p<0.001
    

    View Slide

35. take more time
    fewer errors
    take less time
    more errors
    Macrotasks
    Microtasks
    …
    …
    

    View Slide

36. How do macrotasks and microtasks differ
    in terms of interruptibility?
    

    View Slide

37. Macrotasks more susceptible to interruptions
    Task Time w/o Interruptions
    0
    20
    40
    60
    80
    100
    120
    140
    Uninterrupted Interrupted
    Macrotask
    Microtask
    p<0.10
    

    View Slide

38. Microtasks less susceptible to interruptions
    Task Time w/o Interruptions
    0
    20
    40
    60
    80
    100
    120
    140
    Uninterrupted Interrupted
    Macrotask
    Microtask
    p<0.10
    

    View Slide

39. How do macrotasks and microtasks differ
    in terms of worker preference?
    

    View Slide

40. 77% of workers
    preferred microtasks to macrotasks.
    

    View Slide

41. take more time
    fewer errors
    take less time
    more errors
    unaffected by interruptions
    preferred more
    slowed by interruptions
    preferred less
    Macrotasks
    Microtasks
    

    View Slide

42. Information workers can
    benefit from microtasking.
    (Can we write research papers on our phones?)
    

    View Slide

43. Information workers can
    benefit from microtasking.
    (Can turkers benefit from macrotasking?)
    

    View Slide

44. 44
    Break it
    Down
    A Comparison of Macro- and Microtasks
    Justin Cheng
    jcccf@cs.stanford.edu
    Jaime Teevan
    teevan@microsoft.com
    Shamsi Iqbal
    shamsi@microsoft.com
    Michael Bernstein
    msb@cs.stanford.edu
    

    View Slide