How Annotation Styles Influence Content

Slide 1

Slide 1 text

How ANNOTATION STYLES influence CONTENT & PREFERENCES Justin Cheng, Stanford University Dan Cosley, Cornell University

Slide 2

Slide 2 text

Why do people tag? (Ames & Naaman 2007) 2

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Annotation Styles savannah circle of life Everything the light touches is our kingdom. Single-word Tag (SWT) Multi-word Tag (MWT) Comment (Comment) 4

Slide 5

Slide 5 text

Slide 6

Slide 6 text

6 http://www.flickr.com/photos/culturesubculture/375190432/

Slide 7

Slide 7 text

7 http://www.flickr.com/photos/culturesubculture/375190432/

Slide 8

Slide 8 text

How do different annotation styles motivate different uses? 8

Slide 9

Slide 9 text

What we found 1.  Annotation styles differ in objectivity, descriptiveness and interestingness 2.  Producers and consumers of annotation assess these styles differently 9

Slide 10

Slide 10 text

Experiment 1: How do annotation styles differ? 10

Slide 11

Slide 11 text

•  In a within-subjects experiments, 21 participants annotated 30 Flickr images •  We evaluated annotation on differences in objectivity, and word categories 11 Experiment 1: How do annotation styles differ?

Slide 12

Slide 12 text

Subjectivity in Annotation 12 25 48 0 10 20 30 40 50 60 SWTs MWTs Comments % Subjective Coding 12

Slide 13

Slide 13 text

Annotation styles also differed in the types of words used. 13 LIWC

Slide 14

Slide 14 text

SWTs less likely than MWTs, comments to indicate time or location 14 “Bricks”, “Ruins” vs. “Sitting in the dirt” LIWC Relative

Slide 15

Slide 15 text

MWTs more descriptive than SWTs, comments for sensory perception 15 “Golden crispy fries” LIWC Perceptual

Slide 16

Slide 16 text

Comments more expressive for thought and judgment 16 “X had 3 extra turns and still couldn’t pull out the victory; O is a crafty player.” LIWC Cognitive

Slide 17

Slide 17 text

Experiment 2: How are different styles evaluated? 17

Slide 18

Slide 18 text

•  29 participants evaluated annotations from Experiment 1 on the same images •  These evaluations on accuracy, discovery, and interestingness were compared across SWTs, MWTs and comments 18 Experiment 2: How are different styles evaluated?

Slide 19

Slide 19 text

How Accurate? •  MWTs most descriptive •  Comments opinionated, contained unnecessary words Likert 19 3.98 4.09 3.73 0 1 2 3 4 SWTs MWTs Comments

Slide 20

Slide 20 text

3.62 3.88 3.37 1 1.5 2 2.5 3 3.5 4 SWTs MWTs Comments How Discoverable? •  Correlates with accuracy •  MWTs and SWTs more like keywords Likert 20 =

Slide 21

Slide 21 text

3 3.25 3.32 1 1.5 2 2.5 3 3.5 4 SWTs MWTs Comments How Interesting? •  Comments provided interesting interpretations •  SWTs simply stated what an image was about Likert 21

Slide 22

Slide 22 text

Different styles support different goals. SWTs are for search. Comments are for discourse. Use MWTs for both! 22

Slide 23

Slide 23 text

Producer Consumer Producers and Consumers are Intentioned 23

Slide 24

Slide 24 text

•  SWTs were quick and easy to think of •  With comments, I could get across the exact message I wanted Producer 24

Slide 25

Slide 25 text

•  MWTs provide more description than SWTs. •  Comments are personal, irrelevant, opinionated. Consumer 25

Slide 26

Slide 26 text

Roles and Effort Matter. While consumers prefer MWTs (41%), producers prefer SWTs (43%). 26

Slide 27

Slide 27 text

The Case for More Words in Tags •  Multi-word tags achieved a balance between single-word tags and comments •  More descriptive than SWTs, and more succinct than comments •  High accuracy, discoverability, interestingness 27

Slide 28

Slide 28 text

1.  Annotation styles differ in objectivity, descriptiveness and interestingness 2.  Producers and consumers of annotation assess these styles differently The End 28 Justin Cheng [email protected] Dan Cosley [email protected]