How Annotation Styles Influence Content

Presented at HT 2013.

Transcript

1. How
   ANNOTATION STYLES
   influence
   CONTENT & PREFERENCES
   Justin Cheng, Stanford University
   Dan Cosley, Cornell University
    
   

   View Slide

2. Why do people tag?
   (Ames & Naaman 2007)
   2  
   

   View Slide

3. View Slide

4. Annotation Styles
   savannah circle of life
   Everything the light
   touches is our
   kingdom.
   Single-word Tag
   (SWT)
   Multi-word Tag
   (MWT)
   Comment
   (Comment)
   4  
   

   View Slide

5. View Slide

6. 6  
   http://www.flickr.com/photos/culturesubculture/375190432/
   

   View Slide

7. 7  
   http://www.flickr.com/photos/culturesubculture/375190432/
   

   View Slide

8. How do different annotation
   styles motivate different
   uses?
   8  
   

   View Slide

9. What we found
   1.  Annotation styles differ in objectivity,
   descriptiveness and interestingness
   2.  Producers and consumers of annotation
   assess these styles differently
   9  
   

   View Slide

10. Experiment 1:
    How do annotation styles differ?
    10  
    

    View Slide

11. •  In a within-subjects experiments,
    21 participants annotated 30 Flickr images
    •  We evaluated annotation on differences in
    objectivity, and word categories
    11  
    Experiment 1:
    How do annotation styles differ?
    

    View Slide

12. Subjectivity in Annotation
    12
    25
    48
    0
    10
    20
    30
    40
    50
    60
    SWTs MWTs Comments
    % Subjective
    Coding
    12  
    

    View Slide

13. Annotation styles also
    differed in the types of
    words used.
    13  
    LIWC
    

    View Slide

14. SWTs less likely than
    MWTs, comments to
    indicate time or
    location
    14  
    “Bricks”, “Ruins” vs.
    “Sitting in the dirt”
    LIWC Relative
    

    View Slide

15. MWTs more
    descriptive than
    SWTs, comments for
    sensory perception
    15  
    “Golden crispy fries”
    LIWC Perceptual
    

    View Slide

16. Comments more
    expressive for
    thought and
    judgment
    16  
    “X had 3 extra turns and
    still couldn’t pull out the
    victory; O is a crafty player.”
    LIWC Cognitive
    

    View Slide

17. Experiment 2:
    How are different styles evaluated?
    17  
    

    View Slide

18. •  29 participants evaluated annotations from
    Experiment 1 on the same images
    •  These evaluations on accuracy, discovery, and
    interestingness were compared across SWTs,
    MWTs and comments
    18  
    Experiment 2:
    How are different styles evaluated?
    

    View Slide

19. How Accurate?
    •  MWTs most
    descriptive
    •  Comments
    opinionated,
    contained
    unnecessary words
    Likert
    19  
    3.98 4.09
    3.73
    0
    1
    2
    3
    4
    SWTs MWTs Comments
    

    View Slide

20. 3.62
    3.88
    3.37
    1
    1.5
    2
    2.5
    3
    3.5
    4
    SWTs MWTs Comments
    How Discoverable?
    •  Correlates with
    accuracy
    •  MWTs and SWTs
    more like keywords
    Likert
    20  
    =  
    

    View Slide

21. 3
    3.25 3.32
    1
    1.5
    2
    2.5
    3
    3.5
    4
    SWTs MWTs Comments
    How Interesting?
    •  Comments provided
    interesting
    interpretations
    •  SWTs simply stated
    what an image was
    about
    Likert
    21  
    

    View Slide

22. Different styles support
    different goals.
    SWTs are for search.
    Comments are for discourse.
    Use MWTs for both!
    22  
    

    View Slide

23. Producer Consumer
    Producers and Consumers are
    Intentioned
    23  
    

    View Slide

24. •  SWTs were quick and easy to
    think of
    •  With comments, I could get
    across the exact message I
    wanted
    Producer 24  
    

    View Slide

25. •  MWTs provide more
    description than SWTs.
    •  Comments are personal,
    irrelevant, opinionated.
    Consumer 25  
    

    View Slide

26. Roles and Effort Matter.
    While consumers prefer
    MWTs (41%), producers
    prefer SWTs (43%).
    26  
    

    View Slide

27. The Case for More Words in Tags
    •  Multi-word tags achieved a balance between
    single-word tags and comments
    •  More descriptive than SWTs, and more
    succinct than comments
    •  High accuracy, discoverability, interestingness
    27  
    

    View Slide

28. 1.  Annotation styles differ in objectivity,
    descriptiveness and interestingness
    2.  Producers and consumers of annotation
    assess these styles differently
    The
    End
    28  
    Justin Cheng
    jcccf@cs.stanford.edu  
    Dan Cosley
    danco@cs.cornell.edu  
    

    View Slide