How Annotation Styles Influence Content

Presented at HT 2013.

Transcript

1. How ANNOTATION STYLES influence CONTENT & PREFERENCES Justin Cheng, Stanford University Dan Cosley, Cornell University  
2. Why do people tag? (Ames & Naaman 2007) 2  
4. Annotation Styles savannah circle of life Everything the light touches is our kingdom. Single-word Tag (SWT) Multi-word Tag (MWT) Comment (Comment) 4  
6. 6   http://www.flickr.com/photos/culturesubculture/375190432/
7. 7   http://www.flickr.com/photos/culturesubculture/375190432/
8. How do different annotation styles motivate different uses? 8  
9. What we found 1.  Annotation styles differ in objectivity, descriptiveness and interestingness 2.  Producers and consumers of annotation assess these styles differently 9  
10. Experiment 1: How do annotation styles differ? 10  
11. •  In a within-subjects experiments, 21 participants annotated 30 Flickr images •  We evaluated annotation on differences in objectivity, and word categories 11   Experiment 1: How do annotation styles differ?
12. Subjectivity in Annotation 12 25 48 0 10 20 30 40 50 60 SWTs MWTs Comments % Subjective Coding 12  
13. Annotation styles also differed in the types of words used. 13   LIWC
14. SWTs less likely than MWTs, comments to indicate time or location 14   “Bricks”, “Ruins” vs. “Sitting in the dirt” LIWC Relative
15. MWTs more descriptive than SWTs, comments for sensory perception 15   “Golden crispy fries” LIWC Perceptual
16. Comments more expressive for thought and judgment 16   “X had 3 extra turns and still couldn’t pull out the victory; O is a crafty player.” LIWC Cognitive
17. Experiment 2: How are different styles evaluated? 17  
18. •  29 participants evaluated annotations from Experiment 1 on the same images •  These evaluations on accuracy, discovery, and interestingness were compared across SWTs, MWTs and comments 18   Experiment 2: How are different styles evaluated?
19. How Accurate? •  MWTs most descriptive •  Comments opinionated, contained unnecessary words Likert 19   3.98 4.09 3.73 0 1 2 3 4 SWTs MWTs Comments
20. 3.62 3.88 3.37 1 1.5 2 2.5 3 3.5 4 SWTs MWTs Comments How Discoverable? •  Correlates with accuracy •  MWTs and SWTs more like keywords Likert 20   =  
21. 3 3.25 3.32 1 1.5 2 2.5 3 3.5 4 SWTs MWTs Comments How Interesting? •  Comments provided interesting interpretations •  SWTs simply stated what an image was about Likert 21  
22. Different styles support different goals. SWTs are for search. Comments are for discourse. Use MWTs for both! 22  
23. Producer Consumer Producers and Consumers are Intentioned 23  
24. •  SWTs were quick and easy to think of •  With comments, I could get across the exact message I wanted Producer 24  
25. •  MWTs provide more description than SWTs. •  Comments are personal, irrelevant, opinionated. Consumer 25  
26. Roles and Effort Matter. While consumers prefer MWTs (41%), producers prefer SWTs (43%). 26  
27. The Case for More Words in Tags •  Multi-word tags achieved a balance between single-word tags and comments •  More descriptive than SWTs, and more succinct than comments •  High accuracy, discoverability, interestingness 27  
28. 1.  Annotation styles differ in objectivity, descriptiveness and interestingness 2.  Producers and consumers of annotation assess these styles differently The End 28   Justin Cheng jcccf@cs.stanford.edu   Dan Cosley danco@cs.cornell.edu