Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Annotation Styles Influence Content

How Annotation Styles Influence Content

Presented at HT 2013.

8480b47e733a040fba07c32da414b0e0?s=128

Justin Cheng

May 29, 2014
Tweet

Transcript

  1. How ANNOTATION STYLES influence CONTENT & PREFERENCES Justin Cheng, Stanford

    University Dan Cosley, Cornell University  
  2. Why do people tag? (Ames & Naaman 2007) 2  

  3. 3  

  4. Annotation Styles savannah circle of life Everything the light touches

    is our kingdom. Single-word Tag (SWT) Multi-word Tag (MWT) Comment (Comment) 4  
  5. 5  

  6. 6   http://www.flickr.com/photos/culturesubculture/375190432/

  7. 7   http://www.flickr.com/photos/culturesubculture/375190432/

  8. How do different annotation styles motivate different uses? 8  

  9. What we found 1.  Annotation styles differ in objectivity, descriptiveness

    and interestingness 2.  Producers and consumers of annotation assess these styles differently 9  
  10. Experiment 1: How do annotation styles differ? 10  

  11. •  In a within-subjects experiments, 21 participants annotated 30 Flickr

    images •  We evaluated annotation on differences in objectivity, and word categories 11   Experiment 1: How do annotation styles differ?
  12. Subjectivity in Annotation 12 25 48 0 10 20 30

    40 50 60 SWTs MWTs Comments % Subjective Coding 12  
  13. Annotation styles also differed in the types of words used.

    13   LIWC
  14. SWTs less likely than MWTs, comments to indicate time or

    location 14   “Bricks”, “Ruins” vs. “Sitting in the dirt” LIWC Relative
  15. MWTs more descriptive than SWTs, comments for sensory perception 15

      “Golden crispy fries” LIWC Perceptual
  16. Comments more expressive for thought and judgment 16   “X

    had 3 extra turns and still couldn’t pull out the victory; O is a crafty player.” LIWC Cognitive
  17. Experiment 2: How are different styles evaluated? 17  

  18. •  29 participants evaluated annotations from Experiment 1 on the

    same images •  These evaluations on accuracy, discovery, and interestingness were compared across SWTs, MWTs and comments 18   Experiment 2: How are different styles evaluated?
  19. How Accurate? •  MWTs most descriptive •  Comments opinionated, contained

    unnecessary words Likert 19   3.98 4.09 3.73 0 1 2 3 4 SWTs MWTs Comments
  20. 3.62 3.88 3.37 1 1.5 2 2.5 3 3.5 4

    SWTs MWTs Comments How Discoverable? •  Correlates with accuracy •  MWTs and SWTs more like keywords Likert 20   =  
  21. 3 3.25 3.32 1 1.5 2 2.5 3 3.5 4

    SWTs MWTs Comments How Interesting? •  Comments provided interesting interpretations •  SWTs simply stated what an image was about Likert 21  
  22. Different styles support different goals. SWTs are for search. Comments

    are for discourse. Use MWTs for both! 22  
  23. Producer Consumer Producers and Consumers are Intentioned 23  

  24. •  SWTs were quick and easy to think of • 

    With comments, I could get across the exact message I wanted Producer 24  
  25. •  MWTs provide more description than SWTs. •  Comments are

    personal, irrelevant, opinionated. Consumer 25  
  26. Roles and Effort Matter. While consumers prefer MWTs (41%), producers

    prefer SWTs (43%). 26  
  27. The Case for More Words in Tags •  Multi-word tags

    achieved a balance between single-word tags and comments •  More descriptive than SWTs, and more succinct than comments •  High accuracy, discoverability, interestingness 27  
  28. 1.  Annotation styles differ in objectivity, descriptiveness and interestingness 2. 

    Producers and consumers of annotation assess these styles differently The End 28   Justin Cheng jcccf@cs.stanford.edu   Dan Cosley danco@cs.cornell.edu