Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Annotation Styles Influence Content

How Annotation Styles Influence Content

Presented at HT 2013.

Justin Cheng

May 29, 2014
Tweet

More Decks by Justin Cheng

Other Decks in Research

Transcript

  1. Annotation Styles savannah circle of life Everything the light touches

    is our kingdom. Single-word Tag (SWT) Multi-word Tag (MWT) Comment (Comment) 4  
  2. What we found 1.  Annotation styles differ in objectivity, descriptiveness

    and interestingness 2.  Producers and consumers of annotation assess these styles differently 9  
  3. •  In a within-subjects experiments, 21 participants annotated 30 Flickr

    images •  We evaluated annotation on differences in objectivity, and word categories 11   Experiment 1: How do annotation styles differ?
  4. Subjectivity in Annotation 12 25 48 0 10 20 30

    40 50 60 SWTs MWTs Comments % Subjective Coding 12  
  5. SWTs less likely than MWTs, comments to indicate time or

    location 14   “Bricks”, “Ruins” vs. “Sitting in the dirt” LIWC Relative
  6. MWTs more descriptive than SWTs, comments for sensory perception 15

      “Golden crispy fries” LIWC Perceptual
  7. Comments more expressive for thought and judgment 16   “X

    had 3 extra turns and still couldn’t pull out the victory; O is a crafty player.” LIWC Cognitive
  8. •  29 participants evaluated annotations from Experiment 1 on the

    same images •  These evaluations on accuracy, discovery, and interestingness were compared across SWTs, MWTs and comments 18   Experiment 2: How are different styles evaluated?
  9. How Accurate? •  MWTs most descriptive •  Comments opinionated, contained

    unnecessary words Likert 19   3.98 4.09 3.73 0 1 2 3 4 SWTs MWTs Comments
  10. 3.62 3.88 3.37 1 1.5 2 2.5 3 3.5 4

    SWTs MWTs Comments How Discoverable? •  Correlates with accuracy •  MWTs and SWTs more like keywords Likert 20   =  
  11. 3 3.25 3.32 1 1.5 2 2.5 3 3.5 4

    SWTs MWTs Comments How Interesting? •  Comments provided interesting interpretations •  SWTs simply stated what an image was about Likert 21  
  12. •  SWTs were quick and easy to think of • 

    With comments, I could get across the exact message I wanted Producer 24  
  13. •  MWTs provide more description than SWTs. •  Comments are

    personal, irrelevant, opinionated. Consumer 25  
  14. The Case for More Words in Tags •  Multi-word tags

    achieved a balance between single-word tags and comments •  More descriptive than SWTs, and more succinct than comments •  High accuracy, discoverability, interestingness 27  
  15. 1.  Annotation styles differ in objectivity, descriptiveness and interestingness 2. 

    Producers and consumers of annotation assess these styles differently The End 28   Justin Cheng [email protected]   Dan Cosley [email protected]