Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Break It Down: A Comparison of Macro- and Microtasks

Break It Down: A Comparison of Macro- and Microtasks

Presented at CHI 2015.

A large, seemingly overwhelming task can sometimes be transformed into a set of smaller, more manageable microtasks that can each be accomplished independently. In crowdsourcing systems, microtasking enables unskilled workers with limited commitment to work together to complete tasks they would not be able to do individually. We explore the costs and benefits of decomposing macrotasks into microtasks for three task categories: arithmetic, sorting, and transcription. We find that breaking these tasks into microtasks results in longer overall task completion times, but higher quality outcomes and a better experience that may be more resilient to interruptions. These results suggest that microtasks can help people complete high quality work in interruption-driven environments.

Justin Cheng

April 23, 2015
Tweet

More Decks by Justin Cheng

Other Decks in Research

Transcript

  1. Justin Cheng
    [email protected]
    1
    A Comparison of Macro- and Microtasks
    Jaime Teevan
    [email protected]
    Shamsi Iqbal
    [email protected]
    Michael Bernstein
    [email protected]
    Break it
    Down

    View Slide

  2. A Comparison of Macro- and Microtasks

    View Slide

  3. A Comparison of Macro- and Microtasks
    Picking the best photos for an album

    View Slide

  4. A Comparison of Macro- and Microtasks
    Large

    View Slide

  5. A Comparison of Macro- and Microtasks
    Monolithic

    View Slide

  6. A Comparison of Macro- and Microtasks
    Overwhelming

    View Slide

  7. A Comparison of Macro- and Microtasks
    Pairwise comparisons of photos

    View Slide

  8. A Comparison of Macro- and Microtasks
    Smaller

    View Slide

  9. A Comparison of Macro- and Microtasks
    Done in seconds

    View Slide

  10. A Comparison of Macro- and Microtasks
    Done independently

    View Slide

  11. A Comparison of Macro- and Microtasks
    Transcribing a speech
    Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.Lorem ipsum dolor
    sit amet, consectetur adipiscing
    elit.Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.Lorem ipsum dolor
    sit amet, consectetur adipiscing
    elit.Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.

    View Slide

  12. A Comparison of Macro- and Microtasks
    Transcribing individual sentences
    Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.Lorem ipsum dolor
    sit amet, consectetur adipiscing
    elit.Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.Lorem ipsum dolor
    sit amet, consectetur adipiscing
    elit.Lorem ipsum dolor sit amet,
    consectetur adipiscing elit.Lorem
    ipsum dolor sit amet, consectetur
    adipiscing elit.
    Lorem
    ipsum dolor
    sit amet.
    Lorem
    ipsum dolor
    sit amet.
    Lorem
    ipsum dolor
    sit amet.
    Transcribing a speech

    View Slide

  13. Are macrotasks or microtasks better?

    View Slide

  14. How do macrotasks and microtasks differ?

    View Slide

  15. People already break tasks down.

    View Slide

  16. Tasks are perceived in segments.
    Zacks, J. M., Tversky, B., and Iyer, G. Perceiving,remembering, and communicating structure in events. In Exp. Psychol. Gen. 2001.
    Wickens, C. D. Multiple resources and performance prediction. In Theor. Issues Ergon. 2002.
    =

    View Slide

  17. Many tasks are done in short bursts.
    Gonzalez, V. M., and Mark, G. Constant, constant,multi-tasking craziness: managing multiple working spheres. In CHI 2004.

    View Slide

  18. They are constantly interrupted.
    Gonzalez, V. M., and Mark, G. Constant, constant,multi-tasking craziness: managing multiple working spheres. In CHI 2004.
    Hey, quick
    question…
    Have a minute?

    View Slide

  19. Microtasking is prevalent in
    crowdsourcing.

    View Slide

  20. Microtasking can help us be productive
    in spare moments of time.
    Vaish, R., et al. Twitch crowdsourcing: crowd contributions in short bursts of time. In CHI 2014.

    View Slide

  21. How do macrotasks and microtasks
    differ in terms of…?

    View Slide

  22. How do macrotasks and microtasks
    differ in terms of time taken?

    View Slide

  23. How do macrotasks and microtasks
    differ in terms of work quality?

    View Slide

  24. How do macrotasks and microtasks
    differ in terms of interruptibility?

    View Slide

  25. How do macrotasks and microtasks
    differ in terms of worker preference?

    View Slide

  26. ×
    Task Format
    (Macrotask/Microtask)
    Presence of Interruptions
    (Interruptions/No Interruptions)
    Arithmetic Sorting Transcription
    3 Tasks
    110 Participants on Clickworker (+MTurk)

    View Slide

  27. Arithmetic Macrotask and Microtask

    View Slide

  28. Sorting Macrotask and Microtask

    View Slide

  29. Sorting Macrotask and Microtask

    View Slide

  30. Transcription Macrotask and Microtask

    View Slide

  31. How do macrotasks and microtasks differ
    in terms of time taken?

    View Slide

  32. Microtasks generally take longer
    Task Time (s)
    0
    50
    100
    150
    200
    Arithmetic Sorting Transcription
    Macrotask
    Microtask
    p<0.001

    View Slide

  33. How do macrotasks and microtasks differ
    in terms of work quality?

    View Slide

  34. Microtasks result in fewer errors
    Error Rate
    0
    0.1
    0.2
    0.3
    0.4
    Arithmetic Sorting Transcription
    Macrotask
    Microtask
    p<0.001

    View Slide

  35. take more time
    fewer errors
    take less time
    more errors
    Macrotasks
    Microtasks


    View Slide

  36. How do macrotasks and microtasks differ
    in terms of interruptibility?

    View Slide

  37. Macrotasks more susceptible to interruptions
    Task Time w/o Interruptions
    0
    20
    40
    60
    80
    100
    120
    140
    Uninterrupted Interrupted
    Macrotask
    Microtask
    p<0.10

    View Slide

  38. Microtasks less susceptible to interruptions
    Task Time w/o Interruptions
    0
    20
    40
    60
    80
    100
    120
    140
    Uninterrupted Interrupted
    Macrotask
    Microtask
    p<0.10

    View Slide

  39. How do macrotasks and microtasks differ
    in terms of worker preference?

    View Slide

  40. 77% of workers
    preferred microtasks to macrotasks.

    View Slide

  41. take more time
    fewer errors
    take less time
    more errors
    unaffected by interruptions
    preferred more
    slowed by interruptions
    preferred less
    Macrotasks
    Microtasks

    View Slide

  42. Information workers can
    benefit from microtasking.
    (Can we write research papers on our phones?)

    View Slide

  43. Information workers can
    benefit from microtasking.
    (Can turkers benefit from macrotasking?)

    View Slide

  44. 44
    Break it
    Down
    A Comparison of Macro- and Microtasks
    Justin Cheng
    [email protected]
    Jaime Teevan
    [email protected]
    Shamsi Iqbal
    [email protected]
    Michael Bernstein
    [email protected]

    View Slide