montag_lab_meeting_11_182.pdf

Gender representations in book.

Transcript

1. Socialization of gender stereotypes through children’s books Molly Lewis University of Chicago University of Wisconsin-Madison Work with Matt Cooper-Borkenhagen, Gary Lupyan, and Mark Seidenberg 29 November 2018
2. Humans have social stereotypes. (salary, office, professional) (parents, children, home) (woman, she, sister) (man, he, brother) Explicit (Nosek et al., 2002): d = .5 Implicit (IAT): d = .72 e.g., Gender stereotypes
3. Language as causal in shaping social stereotypes Word choice matters: • Children infer novel skill is stereotypical of a gender if introduced with generic form (Cimpian & Markman, 2011) • [Boys are/There is a boy who is] really good at a game called “gorp.” Larger scale properties of language? • Caliskan, Bryson, & Narayanan (2017) trained word embedding models on large corpora of text • Strong correlation between gender association of concepts in language and psychological bias, as measured by the IAT
4. −0.06 −0.04 −0.02 0.00 0.02 0.04 Implicit psychological gender bias by country Implicit gender bias is variable (Project Implicit: Nosek, Banaji, & Greenwald, 2002)
5. …and implicit bias is correlated with language (Lewis & Lupyan, in prep.) ar de en es fa fi fr he id it ko ms nl no pl pt ro sv tr ar da de en es fa fi fr he hi hr id it ja ko ms nl no pl pt ro sv tl tr zh Subtitle Embeddings Wikipedia Embeddings −.5 (male− family) 0 .5 1 (male− career) −.5 (male− family) 0 .5 1 (male− career) −.075 (male− family) −.05 −.025 0 .025 .05 (male− career) Linguistic Gender Bias (effect size) Behavioral IAT Gender Bias (residualized) Behavioral and Linguistic Gender Biases
6. Exploring gender representations in children’s literature Hypothesis: Concepts of girls and boys are presented differently in narrative textual input to children. Conception of gender develop in childhood (e.g., Gelman, Taylor, Nguyen, Leaper, & Bigler, 2004) Children receive a lot of linguistic input from books Many public health campaigns advocating reading to children.
7. The plan 1. The corpus 2. Study 1: Estimating word valence 3. Study 2: Estimating book valence 4. Study 3: Predicting book valence 5. Study 4: Gender statistics from books 6. Next steps
8. The corpus Montag, Jones, & Smith corpus (2015): 100 picture books parents read to children between 0-60 months Added ~150 books based on top selling books from Amazon and the Time Magazine’s “100 Best Children’s Books of All Time” 0 50 100 0 5000 10000 15000 20000 Number of words (tokens) count N words per book median = 374 words
9. Study 1: Estimating word gender valence • Selected words that occurred frequently in corpus • Asked people to rate gender valence:
10. Study 1: Method • Words that occurred 15 or more times (n = 1,163) • Included both nouns and verbs • Disambiguated word senses (e.g. “cross (thing)”) • 3 attention check questions • 164 participants • Gender: 81 = females; 82 = males; 1 other • Mean age = 35.16 • Modal education: Undergraduate degree • Instructions:
11. 0 50 100 150 200 1 2 3 4 5 Mean gender rating (feminine−ness) count Mean ratings by item “sir” “men” “him” “girls” “her” “mommy” “chicken” “book” “six”
12. 0.0 0.3 0.6 0.9 1 2 3 4 5 Mean gender rating (feminine−ness) density gender male female Mean rating by participant gender Ratings by participant gender 0.0 0.3 0.6 0.9 1 2 3 4 5 Mean gender rating (feminine−ness) gender male female Mean rating by participant gender females < males females > males 1 2 3 4 5 1 2 3 4 5 male−participant feminine−ness rating female−participant feminine−ness rating females < males females > males 1 2 3 4 5 1 2 3 4 5 male−participant feminine−ness rating female−participant feminine−ness rating females < males absolutely all allowed almost along always anyone apple battle beach became bees believe below big bigger bird black blew block book boom bottom box boy brain breakfast brother bunny butter cap captain carry case chase choo choose climbed close clouds coat (clothes) cock color coming cookie corn crawled cross (action) curious dig dinner dinosaur dreams drive dry eggs empty every everybody eyes fall falling far farmer favorite fell fight filled finished fire first flowers foot forever free freight friend friendly frog give gone good grace gray had hair hang happily hear heard help hid his hurt ice jump kissed kisses lay leaves (on trees) leg letter library long lot makes mama mayor me met mind miss (action) mouse much myself news north not nothing orange (fruit) other our out owl part (..of something) part (as in leave) peter pickle pond puppy pushed quack (duck sound) quick quiet raised reach replied river robin rocks rope rules sandwich says secret sheep sick sit sleeping smile smiled snake snow softly somewhere song space spaghetti speak special stinky stole stood stop story straight string surprise table taken teeth tell terribly thin though thought throw time toes tonight track trash truck until upon us walked watched wet why wild wings wolf wonder wonderful wood write writing you females > males 1 2 3 4 5 1 2 3 4 5 male−participant feminine−ness rating female−participant feminine−ness rating
13. Are gendering ratings distinct from other word properties? Yes, but correlated. In regression, all measures predict independent variance, except dominance Feminine words are: 1. Learned earlier 2. Lower frequency 3. More positively valenced 4. Associate with lower arousal 5. Less concrete Log frequency Dominance Valence Arousal Concreteness AoA Word length (nchars)
14. • Word meaning = co-occurring words in a corpus (“Distributional semantics”; Firth, 1957) • Auto-encoding neural network algorithm (e.g. word2vec, Mikolov et al., 2013a) • Represents each word as an N-dimensional vector • From vectors, derive a measure of the semantic similarity between two words by taking the distance between their vectors (e.g., cosine distance). “cat”: [1 x 200 vector ] “dog”: [1 x 200 vector ] cat dog… Wikipedia pages word2vec ... Representing semantics with word embeddings
15. able above absolutely across afraid after afternoon again against ago ah ahead air alice all alligator allowed almost alone along already also always am and angry animals another answer answered any anymore anyone anything anyway anywhere apple apples arms around arrived ask asked asleep ate away awful baby back bad bag ball balloon band bang bark barn basket bat bath bats battle beach beans bear bears beautiful became because become bed bee been bees before began beginning behind being believe bell bells belly below berries best bet better between big bigger bike bird birds birthday bit bite black blew block blow blue boat book books boom born both bottle bottom bow box boy boys brain branch branches bread break breakfast breath bridge bright bring broke broken brother brothers brought brown brush build built bump bunny bus bush bushes busy but butter butterfly button buttons buy buzz bye cake call called calling came can cannot cans cap captain car care carried carrots carry cars case cat catch cats caught cause cave cellar certainly chair chairs change chase cheese chicken child children chin chocolate choo choose christmas circle city class clean clear clever climb climbed clock close closed clothes clouds coat cock cold color colors come comes coming cookie corn corner could count countrycourse covered cow cozy crawled crayon crazy cream cried cross crow cry cup curious cut dad daddy dance danced dancing dark day days dear decided deep delicious den department desk did different dig dinner dinosaur dinosaurs dirt discovered do doctor does dog dogs doing dolls done door doors doughnuts down downstairs dr dragons dream dreams dress dressed drink drive driver drives dropped drum dry duck dug dump during dust each ear early ears easy eat eaten eating egg eggs eight elephant elephants elevator elizabeth else empty end engine enough escape even evening ever every everybody everyone everything everywhere exactly except excuse explained extra eye eyes face fair fall fallen falling family far farmer fast faster fat father favorite fear feel feeling feels feet fell felt fence few field fight filled finally find fine finger fingers finished fire first fish fit five flew floor flower flowers fly flying folks follow followed food foot forest forever forget forgot forth found four fox free freight fresh friend friendly friends frightened frog front fruit full fun funny fur game garbage garden gate gave general george get gets getting giant girl girls give gives glad glass go goat god goes going gold gone goo good goodbye goodnight got grace grandma grass gray great green grew ground grow grown grows guess had hair half hall hand hands hang hanging happen happened happily happy hard hare has hat hate have having hay he head heads hear heard heart heavy held hello help helped hen henry her here herself hey hi hid hide high higher hill hills him himself his hold hole home honey hoo hook hooray hop hope horn horse hot hour house houses hug huge hundred hung hungry hurried hurry hurt hush ice idea if important indeed inside instead into its jack jam job join jump jumped jumping jungle just keep kept kid kids kind king kiss kissed kisses kitchen kitten kittens knees knew knock knocked know knows ladder lady ladybug land large last late later laughed laughing lay lazy lead least leave leaves led left leg legs let letter library life lifted light lights like liked likes line lion list listen listened little live lived lives living lonely long longer look looked looking looks lost lot lots loud love loved lovely loves low luck lucky lunch ma mad made magic mail make makes making mama man many mary master matter maybe mayor me mean means meant meet meeting men mess met middle might miles milk mind mine minute mirror miss mixed mom moment mommy money monkey monkeys monster moo moon more morning most mother mountain mountains mouse mouth move moved moving mr mrs much mud music must my myself name named nap near neck need nest never new news next nice night nine no nobody noise none north nose not note nothing now ocean oclock off office often oh okay old once one ones only onto open opened orange other others our out outside over owl own page pail pants papa paper park part party past pat paul paws peanut peep penguin penny people perfect perhaps person pet peter piano pick picked pickle picnic picture pictures pie piece pig pink pip place places plain plan plate play played playing please pocket poem pole pond poor pop pot pretty prince princess probably proud pull pulled puppies puppy purple pushed put quack queen quick quickly quiet quietly quite rabbit raced rain rained raised ran rather reach reached read ready real really red remember replied rescue rest returned ride riding right ring river road robin rocks rode rolled room rope rose round rules run running rushed sad safe said salt same sand sandwich sang sat saturday saw say saying says scared school scratch scratched sea seat second secret see seeds seem seemed seen sell send sent set seven shake shall share she sheep shiny ship shoe shoes shook shop short should shout shouted shovel show shut sick side sighed sight sign silly silver since sing singing sir sister sit sitting six sky sleep sleeping slept slide slow slowly small smart smell smile smiled smoke snake sneeze snow so socks soft softly some somebody someone something sometimes somewhere son song soon sorry sort sound sounds soup south space spaghetti speak special splash spoke spring square stairs stand standing star stars start started station stay stayed steam step stepped stick still stinky stole stone stood stop stopped store stories story straight strange strawberry street streets string stripes strong stuck such suddenly suit summer sun supper sure surprise surprised swamp sweet swim swish table tail take taken takes taking talk talking tall taste tea teacher teeth telephone tell ten tent terrible terribly than thank thanks their them then there these they thin thing things think thinking thinks third those though thought thousand three threw through throw thumb thump tickle tight till time times tiny tired today toes together told tomorrow tonight too took tooth top toward town toy toys track traffic train trains trash tree trees trick tricks tried trip trouble truck trucks try turn turned turtle turtles twenty two uh umbrella uncle under until up upon upstairs us use used very voice wait waited waiting wake walk walked walking wall want wanted wants warm wash watch watched watching water waved way we wear wearing wee week well went were wet whatever wheels which while whiskers whispered whistle white whole whoo whose why wide wife wild wind window windows wings winter wish wished without woke wolf woman wonder wondered wonderful wood woods woof word words work worked working world worried worry would wow wren write writing wrong wrote yard year years yelled yellow yes yet you young your yourself zoo Semantics of book words
16. thinking verbs communcation verbs precipitation clock/drive adverbs greeting positive emotions male words numbers face/body parts container male names egg onomatopoeia generic pronouns emotions wall/fence physical affection/positive emotion baby words misc music light verbs conjunctions come/go verbs light/dark ocean building types positive adjs first/second pronouns working preparation books misc children time quantifiers vegetation manual tasks cognitive verbs action verbs appearances/misc mind money conversational (e.g. ok, yes) ordinal life misc/prepositions distance metrics valence adjs (awful) emotion verbs motion verbs preposition non−mammal s adverbs geographic features thing misc motion verbs adverbial quantifiers action verbs sounds eating/drinking sky colors people pronouns locamative verbs cognitive vebrs quantifiers sweet language verbs garden food star body parts house parts games big vehicles misc adjs clothing temperature holiday/time action verbs helper verbs size adjs mammals prepositions fruit magic family fire/smoke misc furniture seasons/boat time space descripters watch/like/crazy desk ability/adverbs vehicles preposition/spatial eating location
17. thinking verbs communcation verbs precipitation clock/drive adverbs greeting positive emotions male words numbers face/body parts container male names egg onomatopoeia generic pronouns emotions wall/fence physical affection/positive emotion baby words misc music light verbs conjunctions come/go verbs light/dark ocean building types positive adjs first/second pronouns working preparation books misc children time quantifiers vegetation manual tasks cognitive verbs action verbs appearances/misc mind money conversational (e.g. ok, yes) ordinal life misc/prepositions distance metrics valence adjs (awful) emotion verbs motion verbs preposition non−mammal s adverbs geographic features thing misc motion verbs adverbial quantifiers action verbs sounds eating/drinking sky colors people pronouns locamative verbs cognitive vebrs quantifiers sweet language verbs garden food star body parts house parts games big vehicles misc adjs clothing temperature holiday/time action verbs helper verbs size adjs mammals prepositions fruit magic family fire/smoke misc furniture seasons/boat time space descripters watch/like/crazy desk ability/adverbs vehicles preposition/spatial eating location 1.5 2.0 2.5 3.0 3.5 mean female gender rating
18. Study 2: Estimating book valence 0 20 40 60 1 2 3 4 5 Overall Gender Rating (feminine−ness) count Gender ratings by book “Dear Zoo” “Where’s Spot” “Trashy Town” “Hug” “Olivia” “Time for Bed” “Numbers, Colors, and Shapes” “Pat the Bunny” “Show Dog” Girls books have words that are overall more abstract, more positively valenced, and lower frequency
19. 0.00 0.05 0.10 mean female gender rating Gender book valence and semantic space Clusters of book centroids
20. Study 1 and 2: Discussion • Words have gender associations that are distinct from other word properties. • Semantic clusters vary as a function of gender association. • Book have gender associations. • Books topics vary as a function of gender associations.
21. Study 3: Predicting book gender • Visual statistics • Book covers from Google Books API • For each book cover, characterize color in HSV space (Hue- Saturation- Value/Lightness) HSV space
23. 0.000 0.005 0.010 0.015 0.0 0.5 Overall book gender (feminine−ness) Cover saturation (hsv) Book cover saturation 0.000 0.005 0.010 0.015 0.0 0.5 Overall book gender (feminine−ness) Cover saturation (hsv) Book cover saturation r = -.17*
24. 0.000 0.025 0.050 0.075 0.100 Hue prop Overall color distribution in book covers
25. −0.2 0.0 0.2 0.4 Hue Pearsons r between gender and prop. color Book cover hue and gender −0.4 0.0 0.4 0.8 Hue Pearsons r between gender and prop. color Associated with girls Associated with boys Girls books have more blue, purple, pink; boy books more red.
26. Narrative structure VALENCE negative positive Issue #1: How to measure valence? Issue #2: Books have different lengths Study 3: Predicting book gender
27. Measuring narrative structure from text • Valence norms (Warriner, Kuperman, & Brysbaert, 2013) • “pleasantness of the stimulus” • 9-pt scale ranging from happy to unhappy • 13,915 English lemmas • 10,684 unique tokens in our corpus, on 37% represented in valence norms
28. • Five positive and 5 negative valenced words (Mohammad & Turney, 2013) • For each word, take cosine distance to the centroid of the positive words and the distance to the negative words • Measure of valence: positive – negative Deriving valence estimates from word embeddings
29. Human vs. embedding measure of valence −0.2 −0.1 0.0 0.1 0.5 1.0 1.5 2.0 Human judgement of valence (log) Embedding measure of valence r = .43*
30. Narrative structure VALENCE negative positive Issue #1: How to measure valence? Issue #2: Books have different lengths With embeddings.
31. Blue… …oops I... …plaza 45 words 3552 words Books have very different lengths 0 50 100 0 5000 10000 15000 20000 Number of words count Book length Normalize by taking mean valence of words by percentile:
32. −0.040 −0.038 −0.036 −0.034 −0.032 0 50 100 150 200 250 Word percentile Embedding measure of valence (negativity) girl_tile 1 2 girl books boy books Narrative structure by gender Boy books have larger narrative arc; more conflict
33. Study 4: Semantics from books Evidence that books are gendered, based on adult judgements of gender But can gendered semantics derived directly from statistics of books? Train word2vec model on corpus of children’s books Examine semantics of girls versus boys.
34. Study 4: Method • Train on corpus of children’s text • Nearest neighbor analysis • For gender pairs (”he” vs. she”) identified N closest neighbors in semantic space. • Compared to pairs might expect to have overlap in semantic space: antonyms. (“hot” vs. “cold”) • Get effect size for each word pair type by comparing to two random words. • Asked how how distinct male-gender associated cluster of words was from female-associated cluster of words • Distinct clusters -> male and female clusters have different associations.
35. able above absolutely across afraid after afternoon again against ago ah ahead air alice all alligator allowed almost alone along already also always am and angry animals another answer answered any anymore anyone anything anyway anywhere apple apples arms around arrived ask asked asleep ate away awful baby back bad bag ball balloon band bang bark barn basket bat bath bats battle beach beans bear bears beautiful became because become bed bee been bees before began beginning behind being believe bell bells belly below berries best bet better between big bigger bike bird birds birthday bit bite black blew block blow blue boat book books boom born both bottle bottom bow box boy boys brain branch branches bread break breakfast breath bridge bright bring broke broken brother brothers brought brown brush build built bump bunny bus bush bushes busy but butter butterfly button buttons buy buzz bye cake call called calling came can cannot cans cap captain car care carried carrots carry cars case cat catch cats caught cause cave cellar certainly chair chairs change chase cheese chicken child children chin chocolate choo choose christmas circle city class clean clear clever climb climbed clock close closed clothes clouds coat cock cold color colors come comes coming cookie corn corner could count countrycourse covered cow cozy crawled crayon crazy cream cried cross crow cry cup curious cut dad daddy dance danced dancing dark day days dear decided deep delicious den department desk did different dig dinner dinosaur dinosaurs dirt discovered do doctor does dog dogs doing dolls done door doors doughnuts down downstairs dr dragons dream dreams dress dressed drink drive driver drives dropped drum dry duck dug dump during dust each ear early ears easy eat eaten eating egg eggs eight elephant elephants elevator elizabeth else empty end engine enough escape even evening ever every everybody everyone everything everywhere exactly except excuse explained extra eye eyes face fair fall fallen falling family far farmer fast faster fat father favorite fear feel feeling feels feet fell felt fence few field fight filled finally find fine finger fingers finished fire first fish fit five flew floor flower flowers fly flying folks follow followed food foot forest forever forget forgot forth found four fox free freight fresh friend friendly friends frightened frog front fruit full fun funny fur game garbage garden gate gave general george get gets getting giant girl girls give gives glad glass go goat god goes going gold gone goo good goodbye goodnight got grace grandma grass gray great green grew ground grow grown grows guess had hair half hall hand hands hang hanging happen happened happily happy hard hare has hat hate have having hay he head heads hear heard heart heavy held hello help helped hen henry her here herself hey hi hid hide high higher hill hills him himself his hold hole home honey hoo hook hooray hop hope horn horse hot hour house houses hug huge hundred hung hungry hurried hurry hurt hush ice idea if important indeed inside instead into its jack jam job join jump jumped jumping jungle just keep kept kid kids kind king kiss kissed kisses kitchen kitten kittens knees knew knock knocked know knows ladder lady ladybug land large last late later laughed laughing lay lazy lead least leave leaves led left leg legs let letter library life lifted light lights like liked likes line lion list listen listened little live lived lives living lonely long longer look looked looking looks lost lot lots loud love loved lovely loves low luck lucky lunch ma mad made magic mail make makes making mama man many mary master matter maybe mayor me mean means meant meet meeting men mess met middle might miles milk mind mine minute mirror miss mixed mom moment mommy money monkey monkeys monster moo moon more morning most mother mountain mountains mouse mouth move moved moving mr mrs much mud music must my myself name named nap near neck need nest never new news next nice night nine no nobody noise none north nose not note nothing now ocean oclock off office often oh okay old once one ones only onto open opened orange other others our out outside over owl own page pail pants papa paper park part party past pat paul paws peanut peep penguin penny people perfect perhaps person pet peter piano pick picked pickle picnic picture pictures pie piece pig pink pip place places plain plan plate play played playing please pocket poem pole pond poor pop pot pretty prince princess probably proud pull pulled puppies puppy purple pushed put quack queen quick quickly quiet quietly quite rabbit raced rain rained raised ran rather reach reached read ready real really red remember replied rescue rest returned ride riding right ring river road robin rocks rode rolled room rope rose round rules run running rushed sad safe said salt same sand sandwich sang sat saturday saw say saying says scared school scratch scratched sea seat second secret see seeds seem seemed seen sell send sent set seven shake shall share she sheep shiny ship shoe shoes shook shop short should shout shouted shovel show shut sick side sighed sight sign silly silver since sing singing sir sister sit sitting six sky sleep sleeping slept slide slow slowly small smart smell smile smiled smoke snake sneeze snow so socks soft softly some somebody someone something sometimes somewhere son song soon sorry sort sound sounds soup south space spaghetti speak special splash spoke spring square stairs stand standing star stars start started station stay stayed steam step stepped stick still stinky stole stone stood stop stopped store stories story straight strange strawberry street streets string stripes strong stuck such suddenly suit summer sun supper sure surprise surprised swamp sweet swim swish table tail take taken takes taking talk talking tall taste tea teacher teeth telephone tell ten tent terrible terribly than thank thanks their them then there these they thin thing things think thinking thinks third those though thought thousand three threw through throw thumb thump tickle tight till time times tiny tired today toes together told tomorrow tonight too took tooth top toward town toy toys track traffic train trains trash tree trees trick tricks tried trip trouble truck trucks try turn turned turtle turtles twenty two uh umbrella uncle under until up upon upstairs us use used very voice wait waited waiting wake walk walked walking wall want wanted wants warm wash watch watched watching water waved way we wear wearing wee week well went were wet whatever wheels which while whiskers whispered whistle white whole whoo whose why wide wife wild wind window windows wings winter wish wished without woke wolf woman wonder wondered wonderful wood woods woof word words work worked working world worried worry would wow wren write writing wrong wrote yard year years yelled yellow yes yet you young your yourself zoo Semantics of book words SHE HOT HE COLD coherence = between ((within1 + within2 )/2) DISTINCTIVENESS:
36. −0.5 0.0 0.5 1.0 10 20 30 40 windowsize effect size (amount of overlap) word_type antonyms gender_pronoun corpus adult_full kid Overlap > random Overlap < random Overlap = random Study 4: Results Words associated with girls and boys non- overlapping in children’s book semantics!
37. Future Directions • Estimating gender of who books are being read to • Exploring semantic space in girl vs. boy books more directly • Comparing gender bias to spoken input (CHILDES) • Semantic space as a function of development • Intervention study?
38. Conclusion • Children’s books vary as a function of their gender association • Books associated with girls vs. boys have different semantic associations • Different visual statistics • Different narrative statistics • Linguistic statistics from books are biased • Suggest that, if children are tracking statistics in children’s books, children’s books could play a role into socializing gender stereotypes.
39. Thanks!
41. Method - Word meaning = co-occurring words in a corpus (“Distributional semantics”) - Auto-encoding neural network algorithm, Doc2vec (Le & Mikolov, 2014) Essay 1: [1 x 200 vector ] Essay 2: [1 x 200 vector ] “cat”: [1 x 200 vector ] “dog”: [1 x 200 vector ] Essay 1 (French) Essay 2 (French) Essay 3 (Korean) cat dog… Doc2vec Essay 3: [1 x 200 vector ] ...
42. Word-Embedding Association Test (WEAT) Replicate IAT findings with word embeddings in an analog measure Method: Pre-trained GloVe embeddings distributed by its authors “Common Crawl” corpus (840 billion tokens) 300-d vector, 10 word window Replicate with word2vec
43. Measuring psychological biases with word embeddings Given target words (X, Y) and attribute words (A, B), measure difference between X and Y in terms of their relative similarity to A and B Effect size
44. Results Replicates all behavioral IAT findings tested on the WEAT IAT WEAT X vs. Y A vs. B
45. Do embeddings also encode veridical data? Word-Embedding Factual Association Test (WEFAT) (U.S. Bureau of Labor Statistics) Occupation ~ female-ness Androgynous name ~ female-ness (1990 U.S. census)
46. Language • Caliskan paper • True in lots of languages (Lewis & Lupyan, in prep.)
47. • Motivation: • Kids quickly acquire gender stereotypes • Also, parents encouraged to read lots of books
48. The plan [1] Estimating gender valence of a book via gender valence of words • Words - Distinct from other stuff but correlated • Books – there’s a distribution (histogram) [2] Predictors of book-level gender valence • Books are becoming more girl associated over time • Target kid age of book: trend for books to get less girl- associated across development • Girl books have different colors in book covers • Girl books have different sentiment trajectories [3] Semantics in boy versus girl books
49. able above absolutely across afraid after afternoon again against ago ah ahead air alice all alligator allowed almost alone along already also always am and angry animals another answer answered any anymore anyone anything anyway anywhere apple apples arms around arrived ask asked asleep ate away awful baby back bad bag ball balloon band bang bark barn basket bat bath bats battle beach beans bear bears beautiful became because become bed bee been bees before began beginning behind being believe bell bells belly below berries best bet better between big bigger bike bird birds birthday bit bite black blew block blow blue boat book books boom born both bottle bottom bow box boy boys brain branch branches bread break breakfast breath bridge bright bring broke broken brother brothers brought brown brush build built bump bunny bus bush bushes busy but butter butterfly button buttons buy buzz bye cake call called calling came can cannot cans cap captain car care carried carrots carry cars case cat catch cats caught cause cave cellar certainly chair chairs change chase cheese chicken child children chin chocolate choo choose christmas circle city class clean clear clever climb climbed clock close closed clothes clouds coat cock cold color colors come comes coming cookie corn corner could count countrycourse covered cow cozy crawled crayon crazy cream cried cross crow cry cup curious cut dad daddy dance danced dancing dark day days dear decided deep delicious den department desk did different dig dinner dinosaur dinosaurs dirt discovered do doctor does dog dogs doing dolls done door doors doughnuts down downstairs dr dragons dream dreams dress dressed drink drive driver drives dropped drum dry duck dug dump during dust each ear early ears easy eat eaten eating egg eggs eight elephant elephants elevator elizabeth else empty end engine enough escape even evening ever every everybody everyone everything everywhere exactly except excuse explained extra eye eyes face fair fall fallen falling family far farmer fast faster fat father favorite fear feel feeling feels feet fell felt fence few field fight filled finally find fine finger fingers finished fire first fish fit five flew floor flower flowers fly flying folks follow followed food foot forest forever forget forgot forth found four fox free freight fresh friend friendly friends frightened frog front fruit full fun funny fur game garbage garden gate gave general george get gets getting giant girl girls give gives glad glass go goat god goes going gold gone goo good goodbye goodnight got grace grandma grass gray great green grew ground grow grown grows guess had hair half hall hand hands hang hanging happen happened happily happy hard hare has hat hate have having hay he head heads hear heard heart heavy held hello help helped hen henry her here herself hey hi hid hide high higher hill hills him himself his hold hole home honey hoo hook hooray hop hope horn horse hot hour house houses hug huge hundred hung hungry hurried hurry hurt hush ice idea if important indeed inside instead into its jack jam job join jump jumped jumping jungle just keep kept kid kids kind king kiss kissed kisses kitchen kitten kittens knees knew knock knocked know knows ladder lady ladybug land large last late later laughed laughing lay lazy lead least leave leaves led left leg legs let letter library life lifted light lights like liked likes line lion list listen listened little live lived lives living lonely long longer look looked looking looks lost lot lots loud love loved lovely loves low luck lucky lunch ma mad made magic mail make makes making mama man many mary master matter maybe mayor me mean means meant meet meeting men mess met middle might miles milk mind mine minute mirror miss mixed mom moment mommy money monkey monkeys monster moo moon more morning most mother mountain mountains mouse mouth move moved moving mr mrs much mud music must my myself name named nap near neck need nest never new news next nice night nine no nobody noise none north nose not note nothing now ocean oclock off office often oh okay old once one ones only onto open opened orange other others our out outside over owl own page pail pants papa paper park part party past pat paul paws peanut peep penguin penny people perfect perhaps person pet peter piano pick picked pickle picnic picture pictures pie piece pig pink pip place places plain plan plate play played playing please pocket poem pole pond poor pop pot pretty prince princess probably proud pull pulled puppies puppy purple pushed put quack queen quick quickly quiet quietly quite rabbit raced rain rained raised ran rather reach reached read ready real really red remember replied rescue rest returned ride riding right ring river road robin rocks rode rolled room rope rose round rules run running rushed sad safe said salt same sand sandwich sang sat saturday saw say saying says scared school scratch scratched sea seat second secret see seeds seem seemed seen sell send sent set seven shake shall share she sheep shiny ship shoe shoes shook shop short should shout shouted shovel show shut sick side sighed sight sign silly silver since sing singing sir sister sit sitting six sky sleep sleeping slept slide slow slowly small smart smell smile smiled smoke snake sneeze snow so socks soft softly some somebody someone something sometimes somewhere son song soon sorry sort sound sounds soup south space spaghetti speak special splash spoke spring square stairs stand standing star stars start started station stay stayed steam step stepped stick still stinky stole stone stood stop stopped store stories story straight strange strawberry street streets string stripes strong stuck such suddenly suit summer sun supper sure surprise surprised swamp sweet swim swish table tail take taken takes taking talk talking tall taste tea teacher teeth telephone tell ten tent terrible terribly than thank thanks their them then there these they thin thing things think thinking thinks third those though thought thousand three threw through throw thumb thump tickle tight till time times tiny tired today toes together told tomorrow tonight too took tooth top toward town toy toys track traffic train trains trash tree trees trick tricks tried trip trouble truck trucks try turn turned turtle turtles twenty two uh umbrella uncle under until up upon upstairs us use used very voice wait waited waiting wake walk walked walking wall want wanted wants warm wash watch watched watching water waved way we wear wearing wee week well went were wet whatever wheels which while whiskers whispered whistle white whole whoo whose why wide wife wild wind window windows wings winter wish wished without woke wolf woman wonder wondered wonderful wood woods woof word words work worked working world worried worry would wow wren write writing wrong wrote yard year years yelled yellow yes yet you young your yourself zoo
50. thinking verbs communcation verbs precipitation clock/drive adverbs greeting positive emotions male words numbers face/body parts container male names egg onomatopoeia generic pronouns emotions wall/fence physical affection/positive emotion baby words misc music light verbs conjunctions come/go verbs light/dark ocean building types positive adjs first/second pronouns working preparation books misc children time quantifiers vegetation manual tasks cognitive verbs action verbs appearances/misc mind money conversational (e.g. ok, yes) ordinal life misc/prepositions distance metrics valence adjs (awful) emotion verbs motion verbs preposition non−mammal s adverbs geographic features thing misc motion verbs adverbial quantifiers action verbs sounds eating/drinking sky colors people pronouns locamative verbs cognitive vebrs quantifiers sweet language verbs garden food star body parts house parts games big vehicles misc adjs clothing temperature holiday/time action verbs helper verbs size adjs mammals prepositions fruit magic family fire/smoke misc furniture seasons/boat time space descripters watch/like/crazy desk ability/adverbs vehicles preposition/spatial eating location
51. thinking verbs communcation verbs precipitation clock/drive adverbs greeting positive emotions male words numbers face/body parts container male names egg onomatopoeia generic pronouns emotions wall/fence physical affection/positive emotion baby words misc music light verbs conjunctions come/go verbs light/dark ocean building types positive adjs first/second pronouns working preparation books misc children time quantifiers vegetation manual tasks cognitive verbs action verbs appearances/misc mind money conversational (e.g. ok, yes) ordinal life misc/prepositions distance metrics valence adjs (awful) emotion verbs motion verbs preposition non−mammal s adverbs geographic features thing misc motion verbs adverbial quantifiers action verbs sounds eating/drinking sky colors people pronouns locamative verbs cognitive vebrs quantifiers sweet language verbs garden food star body parts house parts games big vehicles misc adjs clothing temperature holiday/time action verbs helper verbs size adjs mammals prepositions fruit magic family fire/smoke misc furniture seasons/boat time space descripters watch/like/crazy desk ability/adverbs vehicles preposition/spatial eating location 1.5 2.0 2.5 3.0 3.5 mean female gender rating
52. Implicit Association Test (IAT) Categories X = {marigold, poppy, azalea, crocus, iris} Y = {ant, caterpillar, flea, bedbug, spider} Attributes A = {cheer, freedom, health, love, peace} B = {abuse, crash, filth, murder, sickness} Participants slower for incongruent mapping (right), suggesting bias to associate flowers with pleasantness https://implicit.harvard.edu/implicit/takeatest.html marigold cheer ant abuse compare reaction time marigold cheer ant abuse
53. Predicting gender valence • Book covers • Books are becoming more girl associated over time • Target kid age of book: trend for books to get less girl-associated across development • Sentiment trajectory
54. fear negative sadness anger disgust positive trust surprise joy anticipation Deriving word-level measures of valence from word embeddings
55. Deriving word-level measures of valence from word embeddings • Large-scale distributional semantics models derived from auto- encoding neural networks trained on large text corpora. • Assumption: meaning of a word can be described by the words it tends to co-occur with—words occurring in similar contexts, tend to have similar meanings (Firth, 1957). • E.g , “dog” and “hound” more similar than “dog” and “banana” because “dog” co-occurs with words more in common with “hound” than “banana.” • Recent developments in machine learning allow the idea of distributional semantics • to be implemented in a way that takes into account many features of language structure while remaining computationally tractable. The best known of these word embedding models is word2vec (Mikolov, Chen, Corrado, & Dean, 2013). The model
56. BABY TOUCH AND FEEL: ANIMALS, BARNYARD DANCE BLUE HAT GREEN HAT, BUT NOT THE HIPPOPOTAMUS DOGS, EVERYONE POOPS HAPPY BIRTHDAY BABY!, HIPPOS GO BERSERK HORNS TO TOES AND IN BETWEEN, HOW DO DINOSAURS SAY GOOD NIGHT? HOW DO DINOSAURS SAY HAPPY BIRTHDAY?, I AM A BIG SISTER! I AM A BUNNY, MAISY GOES CAMPING MOO BAA LA LA LA!, NIGHT−NIGHT LITTLE POOKIE OH MY OH MY OH MY DINOSAURS!, PAJAMA TIME! THE CARROT SEED, THE GOING TO BED BOOK CAPS FOR SALE, GRANDFATHER'S JOURNEY HAROLD AND THE PURPLE CRAYON, KATY AND THE BIG SNOW LOVE YOU FOREVER, THE COLOR KITTENS THE FANTASTIC FLYING BOOKS OF MR. MORRIS LESSMORE, THE KEEPING QUILT THE STORY OF FERDINAND, TRAIN WINTER DAYS IN THE BIG WOODS ALEXANDER AND THE TERRIBLE HORRIBLE NO GOOD VERY BAD DAY, ARE YOU MY MOTHER? ARTHUR WRITES A STORY, FOX IN SOCKS GEORGE AND MARTHA, GOLDILICIOUS HOW TO BABYSIT A GRANDMA, HOW TO TRAIN A TRAIN I LOVE YOU STINKY FACE, MVP: MOST VALUABLE PUPPY THE GIVING TREE, THE GROUCHY LADYBUG THE GRUFFALO, THE OTHER SIDE THE TRUE STORY OF THE 3 LITTLE PIGS!, THOMAS AND FRIENDS: GO TRAIN GO! BOB BOOKS SET : BEGINNING READERS, DR. SEUSS'S ABC PETE THE CAT: THE WHEELS ON THE BUS, THE WHEELS ON THE BUS ALLIGATOR PIE, ARE YOU A COW? DEAR ZOO, DON...T LET THE PIGEON DRIVE THE BUS GOOD DOG CARL, GOOD NIGHT GORILLA HUG, I LOVE MY MOMMY BECAUSE... I LOVE YOU THROUGH AND THROUGH, LLAMA LLAMA NIGHTY−NIGHT NO DAVID!, POTTY THAT IS NOT A GOOD IDEA!, TODAY I...LL BE A PRINCESS TUESDAY, WHERE IS BABY'S BELLY BUTTON WHERE'S SPOT? CHUGGA−CHUGGA CHOO−CHOO, DOGGIES EACH PEACH PEAR PLUM, ELEPHANT AND PIGGY: TODAY I WILL FLY! GOODNIGHT MOON, GRANDMA WISHES JAMBERRY, JUST ME AND MY DAD NUMBERS COLORS SHAPES, RAIN MAKES APPLESAUCE SNUGGLE PUPPY, SOMETIMES I LIKE TO CURL UP IN A BALL TEN LITTLE LADYBUGS, THE PIGEON FINDS A HOT DOG! THE WONDERFUL THINGS YOU WILL BE, THIS IS NOT MY HAT TICKLE TIME, WELCOME LITTLE ONE ARNIE THE DOUGHNUT, BREAD AND JAM FOR FRANCES CARS AND TRUCKS AND THINGS THAT GO, CHRYSANTHEMUM ELOISE, FROG AND TOAD GREEN EGGS AND HAM, HORTON HEARS A WHO! KINDERGARTEN HERE I COME!, OH THE PLACES YOU...LL GO ONE FISH TWO FISH RED FISH BLUE FISH, STELLALUNA THE BERENSTAIN BEARS FORGET THEIR MANNERS, THE BIG HONEY HUNT THE CAT IN THE HAT, THE DAY THE CRAYONS QUIT THE GARDENER, THE LITTLE ENGINE THAT COULD THE LORAX, THE STINKY CHEESE MAN AND OTHER FAIRLY STUPID TALES THOMAS THE TANK ENGINE: THE VERY FIRST STORIES, WHEN DINOSAURS CAME WITH EVERYTHING A BAD CASE OF STRIPES, AMELIA BEDELIA BRAVE IRENE, CURIOUS GEORGE TAKES A JOB LIBRARY LION, LILLY...S PURPLE PLASTIC PURSE MIKE MULLIGAN AND HIS STEAM SHOVEL, MILLIONS OF CATS MISS RUMPHIUS, SAYONARA MRS. KACKLEMAN SHE PERSISTED: 3 AMERICAN WOMEN WHO CHANGED THE WORLD, SPAGHETTI IN A HOT DOG BUN: HAVING THE COURAGE TO BE WHO YOU ARE SYLVESTER AND THE MAGIC PEBBLE, THE BERENSTAIN BEARS AND THE GREEN−EYED MONSTER THE POLAR EXPRESS, THE RED BALLOON THE STORY OF BABAR, THE THREE QUESTIONS YERTLE THE TURTLE ARTHUR...S NOSE, B IS FOR BEAR BABY BELUGA, CHICKA CHICKA −2−3 CHICKA CHICKA BOOM BOOM, HOP ON POP KNUFFLE BUNNY, MAISY GOES TO THE LIBRARY PAT THE BUNNY, SLOW LORIS TEN LITTLE FINGERS AND TEN LITTLE TOES, TIME FOR BED LITTLE BLUE TRUCK, MR. BROWN CAN MOO! CAN YOU? TRASHY TOWN, WE'RE GOING ON A BEAR HUNT BABY BEAR BABY BEAR WHAT DO YOU SEE?, BIG SHARK LITTLE SHARK BROWN BEAR BROWN BEAR WHAT DO YOU SEE?, FIRST WORDS FROM HEAD TO TOE, LLAMA LLAMA HOME WITH MAMA LLAMA LLAMA RED PAJAMA, POLAR BEAR POLAR BEAR WHAT DO YOU HEAR? THE FOOT BOOK ANGELINA ICE SKATES, CORDUROY EXTRA YARN, FROGGY GOES TO BED GOODNIGHT GOODNIGHT CONSTRUCTION SITE, HOW ROCKET LEARNED TO READ MIGHTY MONSTER MACHINES (BLAZE AND THE MONSTER MACHINES), MISS NELSON IS MISSING OWN MOON, PUPPY BIRTHDAY TO YOU! (PAW PATROL) THE GRUFFALO'S CHILD, THE PAPER BAG PRINCESS BELLY BUTTON BOOK, CLIFFORD THE BIG RED DOG GOD BLESS YOU AND GOOD NIGHT, GRUMPY BIRD HELLO ROCK, I...M A BIG SISTER IF YOU EVER WANT TO BRING A PIANO TO THE BEACH DON'T!, IF YOU GIVE A MOOSE A MUFFIN IF YOU GIVE A MOUSE A COOKIE, OLIVIA OLIVIA. . . AND THE MISSING TOY, ON THE NIGHT YOU WERE BORN P IS FOR POTTY!, PETE THE CAT: ROCKING IN MY SCHOOL SHOES PRESS HERE, THERE'S A WOCKET IN MY POCKET! THERE...S AN ALLIGATOR UNDER MY BED, TRIANGLE (SHAPE TRILOGY) WHEREVER YOU ARE MY LOVE WILL FIND YOU ANIMALIA, BARK GEORGE DINOSAUR RESCUE, HAND HAND FINGERS THUMB I WISH YOU MORE, MAMA DON'T ALLOW THE VERY HUNGRY CATERPILLAR, WHERE IS THE GREEN SHEEP? JENNY AND THE CAT CLUB, THE REAL MOTHER GOOSE WHERE THE SIDEWALK ENDS, WINNIE THE POOH A SICK DAY FOR AMOS MCGEE, BUNNY PARTY CLIFFORD AT THE CIRCUS, GIRAFFES CAN'T DANCE GO DOG. GO!, HARRY THE DIRTY DOG HOW TO CATCH A STAR, LOST AND FOUND MADELINE, SHOW DOG THE HAT, THE NIGHT BEFORE MOTHER'S DAY THE POUT−POUT FISH, WHISTLE FOR WILLIE CHARLIE AND THE NEW BABY, GUESS HOW MUCH I LOVE YOU I WANT MY HAT BACK, LITTLE BEAR: WHAT WILL LITTLE BEAR WEAR? MORTIMER, OH THE THINKS YOU CAN THINK! SQUARE, THE DUCKLING GETS A COOKIE!? THE IMPORTANT BOOK, THE MONSTER AT THE END OF THIS BOOK THE RUNAWAY BUNNY, YOU GET WHAT YOU GET BIG RED BARN, LITTLE BLUE TRUCK LEADS THE WAY LITTLE OWL...S NIGHT, THE NAPPING HOUSE BEAR SNORES ON, BEAR WANTS MORE CLICK CLACK MOO COWS THAT TYPE, DIARY OF A WOMBAT DRAGONS LOVE TACOS, DUCK ON A BIKE IF ANIMALS KISSED GOOD NIGHT, IN THE NIGHT KITCHEN THE SNOWY DAY, WHERE DO DIGGERS SLEEP AT NIGHT? WHERE THE WILD THINGS ARE BLUEBERRIES FOR SAL, CLOUDY WITH A CHANCE OF MEATBALLS CURIOUS GEORGE, LADYBUG GIRL AT THE BEACH LON PO PO: A RED RIDING HOOD STORY FROM CHINA, MAKE WAY FOR DUCKLINGS THE GARDEN OF ABDUL GASAZI, THE LITTLE HOUSE THE POKY LITTLE PUPPY, THE TALE OF PETER RABBIT TIKKI TIKKI TEMBO 0.00 0.05 0.10 mean female gender rating 0.00 0.05 0.10 mean female gender rating Gender book valence and semantic space
58. 0.00 0.05 0.10 mean female gender rating Gender book valence and semantic space Clusters of book centroids