Upgrade to Pro — share decks privately, control downloads, hide ads and more …

montag_lab_meeting_11_182.pdf

mllewis
November 30, 2018
130

 montag_lab_meeting_11_182.pdf

Gender representations in book.

mllewis

November 30, 2018
Tweet

Transcript

  1. Socialization of gender stereotypes
    through children’s books
    Molly Lewis
    University of Chicago
    University of Wisconsin-Madison
    Work with Matt Cooper-Borkenhagen,
    Gary Lupyan, and Mark Seidenberg
    29 November 2018

    View Slide

  2. Humans have social stereotypes.
    (salary, office, professional)
    (parents, children, home)
    (woman, she, sister)
    (man, he, brother)
    Explicit (Nosek et al., 2002): d = .5
    Implicit (IAT): d = .72
    e.g., Gender stereotypes

    View Slide

  3. Language as causal in shaping social
    stereotypes
    Word choice matters:
    • Children infer novel skill is stereotypical of a gender if introduced
    with generic form (Cimpian & Markman, 2011)
    • [Boys are/There is a boy who is] really good at a game called “gorp.”
    Larger scale properties of language?
    • Caliskan, Bryson, & Narayanan (2017)
    trained word embedding models on
    large corpora of text
    • Strong correlation between gender
    association of concepts in language
    and psychological bias, as measured
    by the IAT

    View Slide

  4. −0.06
    −0.04
    −0.02
    0.00
    0.02
    0.04
    Implicit psychological gender bias by country
    Implicit gender bias is variable
    (Project Implicit: Nosek, Banaji, & Greenwald, 2002)

    View Slide

  5. …and implicit bias is correlated with
    language (Lewis & Lupyan, in prep.)
    ar
    de
    en
    es fa
    fi
    fr
    he
    id
    it
    ko
    ms
    nl
    no
    pl
    pt
    ro
    sv
    tr
    ar
    da
    de
    en
    es fa
    fi
    fr
    he
    hi
    hr
    id
    it
    ja
    ko
    ms
    nl
    no
    pl
    pt
    ro
    sv
    tl
    tr
    zh
    Subtitle Embeddings Wikipedia Embeddings
    −.5
    (male−
    family)
    0 .5 1
    (male−
    career)
    −.5
    (male−
    family)
    0 .5 1
    (male−
    career)
    −.075
    (male−
    family)
    −.05
    −.025
    0
    .025
    .05
    (male−
    career)
    Linguistic Gender Bias
    (effect size)
    Behavioral IAT Gender Bias (residualized)
    Behavioral and Linguistic Gender Biases

    View Slide

  6. Exploring gender representations in
    children’s literature
    Hypothesis: Concepts of girls and boys are presented
    differently in narrative textual input to children.
    Conception of gender
    develop in childhood (e.g.,
    Gelman, Taylor, Nguyen, Leaper, &
    Bigler, 2004)
    Children receive a lot of
    linguistic input from books
    Many public health
    campaigns advocating
    reading to children.

    View Slide

  7. The plan
    1. The corpus
    2. Study 1: Estimating word valence
    3. Study 2: Estimating book valence
    4. Study 3: Predicting book valence
    5. Study 4: Gender statistics from books
    6. Next steps

    View Slide

  8. The corpus
    Montag, Jones, &
    Smith corpus (2015):
    100 picture books
    parents read to children
    between 0-60 months
    Added ~150 books
    based on top selling
    books from Amazon
    and the Time
    Magazine’s “100 Best
    Children’s Books of All
    Time”
    0
    50
    100
    0 5000 10000 15000 20000
    Number of words (tokens)
    count
    N words per book
    median =
    374 words

    View Slide

  9. Study 1: Estimating word gender
    valence
    • Selected words that occurred frequently in
    corpus
    • Asked people to rate gender valence:

    View Slide

  10. Study 1: Method
    • Words that occurred 15 or more times (n = 1,163)
    • Included both nouns and verbs
    • Disambiguated word senses (e.g. “cross (thing)”)
    • 3 attention check questions
    • 164 participants
    • Gender: 81 = females; 82 = males; 1 other
    • Mean age = 35.16
    • Modal education: Undergraduate degree
    • Instructions:

    View Slide

  11. 0
    50
    100
    150
    200
    1 2 3 4 5
    Mean gender rating (feminine−ness)
    count
    Mean ratings by item
    “sir”
    “men”
    “him”
    “girls”
    “her”
    “mommy”
    “chicken”
    “book”
    “six”

    View Slide

  12. 0.0
    0.3
    0.6
    0.9
    1 2 3 4 5
    Mean gender rating (feminine−ness)
    density
    gender
    male
    female
    Mean rating by participant gender
    Ratings by participant gender
    0.0
    0.3
    0.6
    0.9
    1 2 3 4 5
    Mean gender rating (feminine−ness)
    gender
    male
    female
    Mean rating by participant gender
    females < males
    females > males
    1
    2
    3
    4
    5
    1 2 3 4 5
    male−participant feminine−ness rating
    female−participant feminine−ness rating
    females < males
    females > males
    1
    2
    3
    4
    5
    1 2 3 4 5
    male−participant feminine−ness rating
    female−participant feminine−ness rating
    females < males
    absolutely
    all
    allowed
    almost
    along
    always
    anyone
    apple
    battle
    beach
    became
    bees
    believe
    below
    big
    bigger
    bird
    black
    blew
    block
    book
    boom
    bottom
    box
    boy
    brain
    breakfast
    brother
    bunny
    butter
    cap
    captain
    carry
    case
    chase
    choo
    choose
    climbed close
    clouds
    coat (clothes)
    cock
    color
    coming
    cookie
    corn
    crawled
    cross (action)
    curious
    dig
    dinner
    dinosaur
    dreams
    drive
    dry
    eggs
    empty
    every
    everybody
    eyes
    fall
    falling
    far
    farmer
    favorite
    fell
    fight
    filled
    finished
    fire
    first
    flowers
    foot
    forever
    free
    freight
    friend
    friendly
    frog
    give
    gone
    good
    grace
    gray
    had
    hair
    hang
    happily
    hear
    heard
    help
    hid
    his
    hurt
    ice
    jump
    kissed
    kisses
    lay
    leaves (on trees)
    leg
    letter
    library
    long
    lot
    makes
    mama
    mayor
    me met
    mind
    miss (action)
    mouse
    much
    myself
    news
    north
    not
    nothing
    orange (fruit)
    other
    our
    out
    owl
    part (..of something)
    part (as in leave)
    peter
    pickle
    pond
    puppy
    pushed
    quack (duck sound)
    quick
    quiet
    raised
    reach
    replied
    river
    robin
    rocks
    rope
    rules
    sandwich
    says
    secret
    sheep
    sick
    sit
    sleeping
    smile
    smiled
    snake
    snow
    softly
    somewhere
    song
    space
    spaghetti
    speak
    special
    stinky
    stole
    stood
    stop
    story
    straight
    string
    surprise
    table
    taken
    teeth
    tell
    terribly
    thin
    though
    thought
    throw
    time
    toes
    tonight
    track
    trash
    truck
    until
    upon
    us
    walked
    watched
    wet
    why
    wild
    wings
    wolf
    wonder
    wonderful
    wood
    write
    writing
    you
    females > males
    1
    2
    3
    4
    5
    1 2 3 4 5
    male−participant feminine−ness rating
    female−participant feminine−ness rating

    View Slide

  13. Are gendering ratings distinct from
    other word properties?
    Yes, but correlated. In regression, all
    measures predict
    independent variance,
    except dominance
    Feminine words are:
    1. Learned earlier
    2. Lower frequency
    3. More positively
    valenced
    4. Associate with lower
    arousal
    5. Less concrete
    Log frequency
    Dominance
    Valence
    Arousal
    Concreteness
    AoA
    Word length
    (nchars)

    View Slide

  14. • Word meaning = co-occurring words in a corpus (“Distributional
    semantics”; Firth, 1957)
    • Auto-encoding neural network algorithm (e.g. word2vec, Mikolov et al., 2013a)
    • Represents each word as an N-dimensional vector
    • From vectors, derive a measure of the semantic similarity between two
    words by taking the distance between their vectors (e.g., cosine distance).
    “cat”: [1 x 200 vector ]
    “dog”: [1 x 200 vector ]
    cat dog…
    Wikipedia
    pages word2vec
    ...
    Representing semantics with word
    embeddings

    View Slide

  15. able
    above
    absolutely
    across
    afraid
    after
    afternoon
    again
    against
    ago
    ah
    ahead
    air
    alice
    all
    alligator
    allowed
    almost
    alone
    along
    already
    also
    always
    am
    and
    angry
    animals
    another
    answer
    answered
    any
    anymore
    anyone
    anything
    anyway
    anywhere
    apple
    apples
    arms
    around
    arrived
    ask
    asked
    asleep
    ate
    away
    awful
    baby
    back
    bad
    bag
    ball
    balloon
    band
    bang
    bark
    barn
    basket
    bat
    bath
    bats
    battle
    beach
    beans
    bear
    bears
    beautiful
    became
    because
    become
    bed
    bee
    been
    bees
    before
    began
    beginning
    behind
    being
    believe
    bell
    bells
    belly
    below
    berries
    best
    bet
    better
    between
    big
    bigger
    bike
    bird
    birds
    birthday
    bit
    bite
    black
    blew
    block
    blow
    blue
    boat
    book
    books
    boom
    born
    both
    bottle
    bottom
    bow
    box
    boy
    boys
    brain
    branch
    branches
    bread
    break
    breakfast
    breath
    bridge
    bright
    bring
    broke
    broken
    brother
    brothers
    brought
    brown
    brush
    build
    built
    bump
    bunny
    bus
    bush
    bushes
    busy
    but
    butter
    butterfly
    button
    buttons
    buy
    buzz
    bye
    cake
    call
    called
    calling
    came
    can
    cannot
    cans
    cap
    captain
    car
    care
    carried
    carrots
    carry
    cars
    case
    cat
    catch
    cats
    caught
    cause
    cave
    cellar
    certainly
    chair
    chairs
    change
    chase
    cheese
    chicken
    child
    children
    chin
    chocolate
    choo
    choose
    christmas
    circle
    city
    class
    clean
    clear
    clever
    climb
    climbed
    clock
    close
    closed
    clothes
    clouds
    coat
    cock
    cold
    color
    colors
    come
    comes
    coming
    cookie
    corn
    corner
    could
    count
    countrycourse
    covered
    cow
    cozy
    crawled
    crayon
    crazy
    cream
    cried
    cross
    crow
    cry
    cup
    curious
    cut
    dad
    daddy
    dance
    danced
    dancing
    dark
    day
    days
    dear
    decided
    deep
    delicious
    den
    department
    desk
    did
    different
    dig
    dinner
    dinosaur
    dinosaurs
    dirt
    discovered
    do
    doctor
    does
    dog
    dogs
    doing
    dolls
    done
    door
    doors
    doughnuts
    down
    downstairs
    dr
    dragons
    dream
    dreams
    dress
    dressed
    drink
    drive
    driver
    drives
    dropped
    drum
    dry
    duck
    dug
    dump
    during
    dust
    each
    ear
    early
    ears
    easy
    eat
    eaten
    eating
    egg
    eggs
    eight
    elephant
    elephants
    elevator
    elizabeth
    else
    empty
    end
    engine
    enough
    escape
    even
    evening
    ever
    every
    everybody
    everyone
    everything
    everywhere
    exactly
    except
    excuse
    explained
    extra
    eye
    eyes
    face
    fair
    fall
    fallen
    falling
    family
    far
    farmer
    fast
    faster
    fat
    father
    favorite
    fear
    feel
    feeling
    feels
    feet
    fell
    felt
    fence
    few
    field
    fight filled
    finally
    find
    fine
    finger
    fingers
    finished
    fire
    first
    fish
    fit
    five
    flew
    floor
    flower
    flowers
    fly
    flying
    folks
    follow
    followed
    food
    foot
    forest
    forever
    forget
    forgot
    forth
    found
    four
    fox
    free
    freight
    fresh
    friend
    friendly
    friends
    frightened
    frog
    front
    fruit
    full
    fun
    funny
    fur
    game
    garbage
    garden
    gate
    gave
    general
    george
    get
    gets
    getting
    giant
    girl
    girls
    give
    gives
    glad
    glass
    go
    goat
    god
    goes
    going
    gold
    gone
    goo
    good
    goodbye
    goodnight
    got
    grace
    grandma
    grass
    gray
    great
    green
    grew
    ground
    grow
    grown
    grows
    guess
    had
    hair
    half
    hall
    hand
    hands
    hang
    hanging
    happen
    happened
    happily
    happy
    hard
    hare
    has
    hat
    hate
    have
    having
    hay
    he
    head
    heads
    hear
    heard
    heart
    heavy
    held
    hello
    help
    helped
    hen
    henry
    her
    here
    herself
    hey
    hi
    hid
    hide
    high
    higher
    hill
    hills
    him
    himself
    his
    hold
    hole
    home
    honey
    hoo
    hook
    hooray
    hop
    hope
    horn
    horse
    hot
    hour
    house
    houses
    hug
    huge
    hundred
    hung
    hungry
    hurried
    hurry
    hurt
    hush
    ice
    idea
    if
    important
    indeed
    inside
    instead
    into
    its
    jack
    jam
    job
    join
    jump
    jumped
    jumping
    jungle
    just
    keep
    kept
    kid
    kids
    kind
    king
    kiss
    kissed
    kisses
    kitchen
    kitten
    kittens
    knees
    knew
    knock
    knocked
    know
    knows
    ladder
    lady
    ladybug
    land
    large
    last
    late
    later
    laughed
    laughing
    lay
    lazy
    lead
    least
    leave
    leaves
    led
    left
    leg
    legs
    let
    letter
    library
    life
    lifted
    light
    lights
    like
    liked
    likes
    line
    lion
    list
    listen
    listened
    little
    live
    lived
    lives
    living
    lonely
    long
    longer
    look
    looked
    looking
    looks
    lost
    lot
    lots
    loud
    love
    loved lovely
    loves
    low
    luck
    lucky
    lunch
    ma
    mad
    made
    magic
    mail
    make
    makes
    making
    mama
    man
    many
    mary
    master
    matter
    maybe
    mayor
    me
    mean
    means
    meant
    meet
    meeting
    men
    mess
    met
    middle
    might
    miles
    milk
    mind
    mine
    minute
    mirror
    miss
    mixed
    mom
    moment
    mommy
    money
    monkey
    monkeys monster
    moo
    moon
    more
    morning
    most
    mother
    mountain
    mountains
    mouse
    mouth
    move
    moved
    moving
    mr
    mrs
    much
    mud
    music
    must
    my
    myself
    name
    named
    nap
    near
    neck
    need
    nest
    never
    new
    news
    next
    nice
    night
    nine
    no
    nobody
    noise
    none
    north
    nose
    not
    note
    nothing
    now
    ocean
    oclock
    off
    office
    often
    oh
    okay
    old
    once
    one
    ones
    only
    onto
    open
    opened
    orange
    other
    others
    our
    out
    outside
    over
    owl
    own
    page
    pail
    pants
    papa
    paper
    park
    part
    party
    past
    pat
    paul
    paws
    peanut
    peep
    penguin
    penny
    people
    perfect
    perhaps
    person
    pet
    peter
    piano
    pick
    picked
    pickle
    picnic
    picture
    pictures
    pie
    piece
    pig
    pink
    pip
    place
    places
    plain
    plan
    plate
    play
    played
    playing
    please
    pocket
    poem
    pole
    pond
    poor
    pop
    pot
    pretty
    prince
    princess
    probably
    proud
    pull
    pulled
    puppies
    puppy
    purple
    pushed
    put
    quack
    queen
    quick
    quickly
    quiet
    quietly
    quite
    rabbit
    raced
    rain
    rained
    raised
    ran
    rather
    reach
    reached
    read
    ready
    real
    really
    red
    remember
    replied
    rescue
    rest
    returned
    ride
    riding
    right
    ring
    river
    road
    robin
    rocks
    rode
    rolled
    room
    rope
    rose
    round
    rules
    run
    running
    rushed
    sad
    safe
    said
    salt
    same
    sand
    sandwich
    sang
    sat
    saturday
    saw
    say
    saying
    says
    scared
    school
    scratch
    scratched
    sea
    seat
    second
    secret
    see
    seeds
    seem
    seemed
    seen
    sell
    send
    sent
    set
    seven
    shake
    shall
    share
    she
    sheep
    shiny
    ship
    shoe
    shoes
    shook
    shop
    short
    should
    shout
    shouted
    shovel
    show
    shut
    sick
    side
    sighed
    sight
    sign
    silly
    silver
    since
    sing
    singing
    sir
    sister
    sit
    sitting
    six
    sky
    sleep
    sleeping
    slept
    slide
    slow
    slowly
    small
    smart
    smell
    smile
    smiled
    smoke
    snake
    sneeze
    snow
    so
    socks
    soft
    softly
    some
    somebody
    someone
    something
    sometimes
    somewhere
    son
    song
    soon
    sorry
    sort
    sound
    sounds
    soup
    south
    space
    spaghetti
    speak
    special
    splash
    spoke
    spring
    square
    stairs
    stand
    standing
    star
    stars
    start
    started
    station
    stay
    stayed
    steam
    step
    stepped
    stick
    still
    stinky
    stole
    stone
    stood
    stop
    stopped
    store
    stories
    story
    straight
    strange
    strawberry
    street
    streets
    string
    stripes
    strong
    stuck
    such
    suddenly
    suit
    summer
    sun
    supper
    sure
    surprise
    surprised
    swamp
    sweet
    swim
    swish
    table
    tail
    take
    taken
    takes
    taking
    talk
    talking
    tall
    taste
    tea
    teacher
    teeth
    telephone
    tell
    ten
    tent
    terrible
    terribly
    than
    thank
    thanks
    their
    them
    then
    there
    these
    they thin
    thing
    things
    think
    thinking
    thinks
    third
    those
    though
    thought
    thousand
    three
    threw
    through
    throw
    thumb
    thump
    tickle
    tight
    till time
    times
    tiny
    tired
    today
    toes
    together
    told
    tomorrow
    tonight
    too
    took
    tooth
    top toward
    town
    toy
    toys
    track
    traffic
    train
    trains
    trash
    tree
    trees
    trick
    tricks
    tried
    trip
    trouble
    truck
    trucks
    try
    turn
    turned
    turtle
    turtles
    twenty
    two
    uh
    umbrella
    uncle
    under
    until
    up
    upon
    upstairs
    us
    use
    used
    very
    voice
    wait
    waited
    waiting
    wake
    walk
    walked
    walking
    wall
    want
    wanted
    wants
    warm
    wash
    watch
    watched
    watching
    water
    waved
    way
    we
    wear
    wearing
    wee
    week
    well
    went
    were
    wet
    whatever
    wheels
    which
    while
    whiskers
    whispered
    whistle
    white
    whole
    whoo
    whose
    why
    wide
    wife
    wild
    wind
    window
    windows
    wings
    winter
    wish
    wished
    without
    woke
    wolf
    woman
    wonder
    wondered
    wonderful
    wood
    woods
    woof
    word
    words
    work
    worked
    working
    world
    worried
    worry
    would
    wow
    wren
    write
    writing
    wrong
    wrote
    yard
    year
    years
    yelled
    yellow
    yes
    yet
    you
    young
    your
    yourself
    zoo
    Semantics of book words

    View Slide

  16. thinking verbs
    communcation verbs
    precipitation
    clock/drive
    adverbs
    greeting
    positive emotions
    male words
    numbers
    face/body parts
    container
    male names
    egg
    onomatopoeia
    generic pronouns
    emotions
    wall/fence
    physical affection/positive emotion
    baby words
    misc
    music
    light verbs
    conjunctions
    come/go verbs
    light/dark
    ocean
    building types
    positive adjs
    first/second pronouns
    working
    preparation
    books
    misc
    children
    time quantifiers
    vegetation
    manual tasks
    cognitive verbs
    action verbs
    appearances/misc
    mind
    money
    conversational (e.g. ok, yes)
    ordinal
    life
    misc/prepositions
    distance metrics
    valence adjs (awful)
    emotion verbs
    motion verbs
    preposition
    non−mammal s
    adverbs
    geographic features
    thing
    misc
    motion verbs
    adverbial quantifiers
    action verbs
    sounds
    eating/drinking
    sky
    colors
    people pronouns
    locamative verbs
    cognitive vebrs
    quantifiers
    sweet
    language verbs
    garden
    food
    star
    body parts
    house parts
    games
    big vehicles
    misc adjs
    clothing
    temperature
    holiday/time
    action verbs
    helper verbs
    size adjs
    mammals
    prepositions
    fruit
    magic
    family
    fire/smoke
    misc
    furniture
    seasons/boat
    time
    space descripters
    watch/like/crazy
    desk
    ability/adverbs
    vehicles
    preposition/spatial
    eating location

    View Slide

  17. thinking verbs
    communcation verbs
    precipitation
    clock/drive
    adverbs
    greeting
    positive emotions
    male words
    numbers
    face/body parts
    container
    male names
    egg
    onomatopoeia
    generic pronouns
    emotions
    wall/fence
    physical affection/positive emotion
    baby words
    misc
    music
    light verbs
    conjunctions
    come/go verbs
    light/dark
    ocean
    building types
    positive adjs
    first/second pronouns
    working
    preparation
    books
    misc
    children
    time quantifiers
    vegetation
    manual tasks
    cognitive verbs
    action verbs
    appearances/misc
    mind
    money
    conversational (e.g. ok, yes)
    ordinal
    life
    misc/prepositions
    distance metrics
    valence adjs (awful)
    emotion verbs
    motion verbs
    preposition
    non−mammal s
    adverbs
    geographic features
    thing
    misc
    motion verbs
    adverbial quantifiers
    action verbs
    sounds
    eating/drinking
    sky
    colors
    people pronouns
    locamative verbs
    cognitive vebrs
    quantifiers
    sweet
    language verbs
    garden
    food
    star
    body parts
    house parts
    games
    big vehicles
    misc adjs
    clothing
    temperature
    holiday/time
    action verbs
    helper verbs
    size adjs
    mammals
    prepositions
    fruit
    magic
    family
    fire/smoke
    misc
    furniture
    seasons/boat
    time
    space descripters
    watch/like/crazy
    desk
    ability/adverbs
    vehicles
    preposition/spatial
    eating location
    1.5 2.0 2.5 3.0 3.5
    mean female
    gender rating

    View Slide

  18. Study 2: Estimating book valence
    0
    20
    40
    60
    1 2 3 4 5
    Overall Gender Rating (feminine−ness)
    count
    Gender ratings by book
    “Dear Zoo”
    “Where’s Spot”
    “Trashy Town”
    “Hug”
    “Olivia”
    “Time for Bed”
    “Numbers, Colors, and Shapes”
    “Pat the Bunny”
    “Show Dog”
    Girls books have words that are overall more abstract, more positively valenced,
    and lower frequency

    View Slide

  19. 0.00 0.05 0.10
    mean female
    gender rating
    Gender book valence and semantic
    space
    Clusters of book centroids

    View Slide

  20. Study 1 and 2: Discussion
    • Words have gender associations that are
    distinct from other word properties.
    • Semantic clusters vary as a function of gender
    association.
    • Book have gender associations.
    • Books topics vary as a function of gender
    associations.

    View Slide

  21. Study 3: Predicting book gender
    • Visual statistics
    • Book covers from
    Google Books API
    • For each book cover,
    characterize color in
    HSV space (Hue-
    Saturation-
    Value/Lightness)
    HSV space

    View Slide

  22. View Slide

  23. 0.000
    0.005
    0.010
    0.015
    0.0 0.5
    Overall book gender (feminine−ness)
    Cover saturation (hsv)
    Book cover saturation
    0.000
    0.005
    0.010
    0.015
    0.0 0.5
    Overall book gender (feminine−ness)
    Cover saturation (hsv)
    Book cover saturation
    r = -.17*

    View Slide

  24. 0.000
    0.025
    0.050
    0.075
    0.100
    Hue
    prop
    Overall color distribution in book covers

    View Slide

  25. −0.2
    0.0
    0.2
    0.4
    Hue
    Pearsons r between
    gender and prop. color
    Book cover hue and gender
    −0.4
    0.0
    0.4
    0.8
    Hue
    Pearsons r between
    gender and prop. color
    Associated
    with girls
    Associated
    with boys
    Girls books have more blue, purple, pink; boy books more red.

    View Slide

  26. Narrative structure
    VALENCE
    negative
    positive
    Issue #1: How to measure valence?
    Issue #2: Books have different lengths
    Study 3: Predicting book gender

    View Slide

  27. Measuring narrative structure
    from text
    • Valence norms (Warriner, Kuperman, & Brysbaert, 2013)
    • “pleasantness of the stimulus”
    • 9-pt scale ranging from happy to unhappy
    • 13,915 English lemmas
    • 10,684 unique tokens in our corpus, on 37% represented
    in valence norms

    View Slide

  28. • Five positive and 5 negative valenced words (Mohammad &
    Turney, 2013)
    • For each word, take cosine distance to the centroid of the
    positive words and the distance to the negative words
    • Measure of valence: positive – negative
    Deriving valence estimates from word
    embeddings

    View Slide

  29. Human vs. embedding measure of
    valence
    −0.2
    −0.1
    0.0
    0.1
    0.5 1.0 1.5 2.0
    Human judgement of valence (log)
    Embedding measure of valence
    r = .43*

    View Slide

  30. Narrative structure
    VALENCE
    negative
    positive
    Issue #1: How to measure valence?
    Issue #2: Books have different lengths
    With embeddings.

    View Slide

  31. Blue… …oops
    I... …plaza
    45 words
    3552
    words
    Books have very different lengths
    0
    50
    100
    0 5000 10000 15000 20000
    Number of words
    count
    Book length
    Normalize by taking mean valence of words by
    percentile:

    View Slide

  32. −0.040
    −0.038
    −0.036
    −0.034
    −0.032
    0 50 100 150 200 250
    Word percentile
    Embedding measure
    of valence (negativity)
    girl_tile
    1
    2
    girl
    books
    boy
    books
    Narrative structure by gender
    Boy books have larger narrative arc; more conflict

    View Slide

  33. Study 4: Semantics from books
    Evidence that books are gendered, based on adult
    judgements of gender
    But can gendered semantics derived directly from
    statistics of books?
    Train word2vec model on corpus of children’s
    books
    Examine semantics of girls versus boys.

    View Slide

  34. Study 4: Method
    • Train on corpus of children’s text
    • Nearest neighbor analysis
    • For gender pairs (”he” vs. she”) identified N
    closest neighbors in semantic space.
    • Compared to pairs might expect to have overlap
    in semantic space: antonyms. (“hot” vs. “cold”)
    • Get effect size for each word pair type by
    comparing to two random words.
    • Asked how how distinct male-gender associated
    cluster of words was from female-associated
    cluster of words
    • Distinct clusters -> male and female clusters have
    different associations.

    View Slide

  35. able
    above
    absolutely
    across
    afraid
    after
    afternoon
    again
    against
    ago
    ah
    ahead
    air
    alice
    all
    alligator
    allowed
    almost
    alone
    along
    already
    also
    always
    am
    and
    angry
    animals
    another
    answer
    answered
    any
    anymore
    anyone
    anything
    anyway
    anywhere
    apple
    apples
    arms
    around
    arrived
    ask
    asked
    asleep
    ate
    away
    awful
    baby
    back
    bad
    bag
    ball
    balloon
    band
    bang
    bark
    barn
    basket
    bat
    bath
    bats
    battle
    beach
    beans
    bear
    bears
    beautiful
    became
    because
    become
    bed
    bee
    been
    bees
    before
    began
    beginning
    behind
    being
    believe
    bell
    bells
    belly
    below
    berries
    best
    bet
    better
    between
    big
    bigger
    bike
    bird
    birds
    birthday
    bit
    bite
    black
    blew
    block
    blow
    blue
    boat
    book
    books
    boom
    born
    both
    bottle
    bottom
    bow
    box
    boy
    boys
    brain
    branch
    branches
    bread
    break
    breakfast
    breath
    bridge
    bright
    bring
    broke
    broken
    brother
    brothers
    brought
    brown
    brush
    build
    built
    bump
    bunny
    bus
    bush
    bushes
    busy
    but
    butter
    butterfly
    button
    buttons
    buy
    buzz
    bye
    cake
    call
    called
    calling
    came
    can
    cannot
    cans
    cap
    captain
    car
    care
    carried
    carrots
    carry
    cars
    case
    cat
    catch
    cats
    caught
    cause
    cave
    cellar
    certainly
    chair
    chairs
    change
    chase
    cheese
    chicken
    child
    children
    chin
    chocolate
    choo
    choose
    christmas
    circle
    city
    class
    clean
    clear
    clever
    climb
    climbed
    clock
    close
    closed
    clothes
    clouds
    coat
    cock
    cold
    color
    colors
    come
    comes
    coming
    cookie
    corn
    corner
    could
    count
    countrycourse
    covered
    cow
    cozy
    crawled
    crayon
    crazy
    cream
    cried
    cross
    crow
    cry
    cup
    curious
    cut
    dad
    daddy
    dance
    danced
    dancing
    dark
    day
    days
    dear
    decided
    deep
    delicious
    den
    department
    desk
    did
    different
    dig
    dinner
    dinosaur
    dinosaurs
    dirt
    discovered
    do
    doctor
    does
    dog
    dogs
    doing
    dolls
    done
    door
    doors
    doughnuts
    down
    downstairs
    dr
    dragons
    dream
    dreams
    dress
    dressed
    drink
    drive
    driver
    drives
    dropped
    drum
    dry
    duck
    dug
    dump
    during
    dust
    each
    ear
    early
    ears
    easy
    eat
    eaten
    eating
    egg
    eggs
    eight
    elephant
    elephants
    elevator
    elizabeth
    else
    empty
    end
    engine
    enough
    escape
    even
    evening
    ever
    every
    everybody
    everyone
    everything
    everywhere
    exactly
    except
    excuse
    explained
    extra
    eye
    eyes
    face
    fair
    fall
    fallen
    falling
    family
    far
    farmer
    fast
    faster
    fat
    father
    favorite
    fear
    feel
    feeling
    feels
    feet
    fell
    felt
    fence
    few
    field
    fight filled
    finally
    find
    fine
    finger
    fingers
    finished
    fire
    first
    fish
    fit
    five
    flew
    floor
    flower
    flowers
    fly
    flying
    folks
    follow
    followed
    food
    foot
    forest
    forever
    forget
    forgot
    forth
    found
    four
    fox
    free
    freight
    fresh
    friend
    friendly
    friends
    frightened
    frog
    front
    fruit
    full
    fun
    funny
    fur
    game
    garbage
    garden
    gate
    gave
    general
    george
    get
    gets
    getting
    giant
    girl
    girls
    give
    gives
    glad
    glass
    go
    goat
    god
    goes
    going
    gold
    gone
    goo
    good
    goodbye
    goodnight
    got
    grace
    grandma
    grass
    gray
    great
    green
    grew
    ground
    grow
    grown
    grows
    guess
    had
    hair
    half
    hall
    hand
    hands
    hang
    hanging
    happen
    happened
    happily
    happy
    hard
    hare
    has
    hat
    hate
    have
    having
    hay
    he
    head
    heads
    hear
    heard
    heart
    heavy
    held
    hello
    help
    helped
    hen
    henry
    her
    here
    herself
    hey
    hi
    hid
    hide
    high
    higher
    hill
    hills
    him
    himself
    his
    hold
    hole
    home
    honey
    hoo
    hook
    hooray
    hop
    hope
    horn
    horse
    hot
    hour
    house
    houses
    hug
    huge
    hundred
    hung
    hungry
    hurried
    hurry
    hurt
    hush
    ice
    idea
    if
    important
    indeed
    inside
    instead
    into
    its
    jack
    jam
    job
    join
    jump
    jumped
    jumping
    jungle
    just
    keep
    kept
    kid
    kids
    kind
    king
    kiss
    kissed
    kisses
    kitchen
    kitten
    kittens
    knees
    knew
    knock
    knocked
    know
    knows
    ladder
    lady
    ladybug
    land
    large
    last
    late
    later
    laughed
    laughing
    lay
    lazy
    lead
    least
    leave
    leaves
    led
    left
    leg
    legs
    let
    letter
    library
    life
    lifted
    light
    lights
    like
    liked
    likes
    line
    lion
    list
    listen
    listened
    little
    live
    lived
    lives
    living
    lonely
    long
    longer
    look
    looked
    looking
    looks
    lost
    lot
    lots
    loud
    love
    loved lovely
    loves
    low
    luck
    lucky
    lunch
    ma
    mad
    made
    magic
    mail
    make
    makes
    making
    mama
    man
    many
    mary
    master
    matter
    maybe
    mayor
    me
    mean
    means
    meant
    meet
    meeting
    men
    mess
    met
    middle
    might
    miles
    milk
    mind
    mine
    minute
    mirror
    miss
    mixed
    mom
    moment
    mommy
    money
    monkey
    monkeys monster
    moo
    moon
    more
    morning
    most
    mother
    mountain
    mountains
    mouse
    mouth
    move
    moved
    moving
    mr
    mrs
    much
    mud
    music
    must
    my
    myself
    name
    named
    nap
    near
    neck
    need
    nest
    never
    new
    news
    next
    nice
    night
    nine
    no
    nobody
    noise
    none
    north
    nose
    not
    note
    nothing
    now
    ocean
    oclock
    off
    office
    often
    oh
    okay
    old
    once
    one
    ones
    only
    onto
    open
    opened
    orange
    other
    others
    our
    out
    outside
    over
    owl
    own
    page
    pail
    pants
    papa
    paper
    park
    part
    party
    past
    pat
    paul
    paws
    peanut
    peep
    penguin
    penny
    people
    perfect
    perhaps
    person
    pet
    peter
    piano
    pick
    picked
    pickle
    picnic
    picture
    pictures
    pie
    piece
    pig
    pink
    pip
    place
    places
    plain
    plan
    plate
    play
    played
    playing
    please
    pocket
    poem
    pole
    pond
    poor
    pop
    pot
    pretty
    prince
    princess
    probably
    proud
    pull
    pulled
    puppies
    puppy
    purple
    pushed
    put
    quack
    queen
    quick
    quickly
    quiet
    quietly
    quite
    rabbit
    raced
    rain
    rained
    raised
    ran
    rather
    reach
    reached
    read
    ready
    real
    really
    red
    remember
    replied
    rescue
    rest
    returned
    ride
    riding
    right
    ring
    river
    road
    robin
    rocks
    rode
    rolled
    room
    rope
    rose
    round
    rules
    run
    running
    rushed
    sad
    safe
    said
    salt
    same
    sand
    sandwich
    sang
    sat
    saturday
    saw
    say
    saying
    says
    scared
    school
    scratch
    scratched
    sea
    seat
    second
    secret
    see
    seeds
    seem
    seemed
    seen
    sell
    send
    sent
    set
    seven
    shake
    shall
    share
    she
    sheep
    shiny
    ship
    shoe
    shoes
    shook
    shop
    short
    should
    shout
    shouted
    shovel
    show
    shut
    sick
    side
    sighed
    sight
    sign
    silly
    silver
    since
    sing
    singing
    sir
    sister
    sit
    sitting
    six
    sky
    sleep
    sleeping
    slept
    slide
    slow
    slowly
    small
    smart
    smell
    smile
    smiled
    smoke
    snake
    sneeze
    snow
    so
    socks
    soft
    softly
    some
    somebody
    someone
    something
    sometimes
    somewhere
    son
    song
    soon
    sorry
    sort
    sound
    sounds
    soup
    south
    space
    spaghetti
    speak
    special
    splash
    spoke
    spring
    square
    stairs
    stand
    standing
    star
    stars
    start
    started
    station
    stay
    stayed
    steam
    step
    stepped
    stick
    still
    stinky
    stole
    stone
    stood
    stop
    stopped
    store
    stories
    story
    straight
    strange
    strawberry
    street
    streets
    string
    stripes
    strong
    stuck
    such
    suddenly
    suit
    summer
    sun
    supper
    sure
    surprise
    surprised
    swamp
    sweet
    swim
    swish
    table
    tail
    take
    taken
    takes
    taking
    talk
    talking
    tall
    taste
    tea
    teacher
    teeth
    telephone
    tell
    ten
    tent
    terrible
    terribly
    than
    thank
    thanks
    their
    them
    then
    there
    these
    they thin
    thing
    things
    think
    thinking
    thinks
    third
    those
    though
    thought
    thousand
    three
    threw
    through
    throw
    thumb
    thump
    tickle
    tight
    till time
    times
    tiny
    tired
    today
    toes
    together
    told
    tomorrow
    tonight
    too
    took
    tooth
    top toward
    town
    toy
    toys
    track
    traffic
    train
    trains
    trash
    tree
    trees
    trick
    tricks
    tried
    trip
    trouble
    truck
    trucks
    try
    turn
    turned
    turtle
    turtles
    twenty
    two
    uh
    umbrella
    uncle
    under
    until
    up
    upon
    upstairs
    us
    use
    used
    very
    voice
    wait
    waited
    waiting
    wake
    walk
    walked
    walking
    wall
    want
    wanted
    wants
    warm
    wash
    watch
    watched
    watching
    water
    waved
    way
    we
    wear
    wearing
    wee
    week
    well
    went
    were
    wet
    whatever
    wheels
    which
    while
    whiskers
    whispered
    whistle
    white
    whole
    whoo
    whose
    why
    wide
    wife
    wild
    wind
    window
    windows
    wings
    winter
    wish
    wished
    without
    woke
    wolf
    woman
    wonder
    wondered
    wonderful
    wood
    woods
    woof
    word
    words
    work
    worked
    working
    world
    worried
    worry
    would
    wow
    wren
    write
    writing
    wrong
    wrote
    yard
    year
    years
    yelled
    yellow
    yes
    yet
    you
    young
    your
    yourself
    zoo
    Semantics of book words
    SHE
    HOT
    HE
    COLD
    coherence =
    between
    ((within1
    + within2
    )/2)
    DISTINCTIVENESS:

    View Slide

  36. −0.5
    0.0
    0.5
    1.0
    10 20 30 40
    windowsize
    effect size (amount of overlap)
    word_type
    antonyms
    gender_pronoun
    corpus
    adult_full
    kid
    Overlap > random
    Overlap < random
    Overlap = random
    Study 4: Results
    Words associated with girls and boys non-
    overlapping in children’s book semantics!

    View Slide

  37. Future Directions
    • Estimating gender of who books are being
    read to
    • Exploring semantic space in girl vs. boy books
    more directly
    • Comparing gender bias to spoken input
    (CHILDES)
    • Semantic space as a function of development
    • Intervention study?

    View Slide

  38. Conclusion
    • Children’s books vary as a function of their
    gender association
    • Books associated with girls vs. boys have
    different semantic associations
    • Different visual statistics
    • Different narrative statistics
    • Linguistic statistics from books are biased
    • Suggest that, if children are tracking statistics
    in children’s books, children’s books could play
    a role into socializing gender stereotypes.

    View Slide

  39. Thanks!

    View Slide

  40. View Slide

  41. Method
    - Word meaning = co-occurring words in a corpus
    (“Distributional semantics”)
    - Auto-encoding neural network algorithm, Doc2vec (Le &
    Mikolov, 2014)
    Essay 1: [1 x 200 vector ]
    Essay 2: [1 x 200 vector ]
    “cat”: [1 x 200 vector
    ]
    “dog”: [1 x 200 vector
    ]
    Essay 1
    (French)
    Essay 2
    (French)
    Essay 3
    (Korean)
    cat dog…
    Doc2vec Essay 3: [1 x 200 vector
    ]
    ...

    View Slide

  42. Word-Embedding Association Test (WEAT)
    Replicate IAT findings with word embeddings in
    an analog measure
    Method:
    Pre-trained GloVe embeddings distributed by its authors
    “Common Crawl” corpus (840 billion tokens)
    300-d vector, 10 word window
    Replicate with word2vec

    View Slide

  43. Measuring psychological biases
    with word embeddings
    Given target words (X, Y)
    and attribute words (A,
    B), measure difference
    between X and Y in
    terms of their relative
    similarity to A and B
    Effect size

    View Slide

  44. Results
    Replicates all behavioral IAT findings tested on the WEAT
    IAT WEAT
    X vs. Y A vs. B

    View Slide

  45. Do embeddings also encode veridical data?
    Word-Embedding Factual Association Test
    (WEFAT)
    (U.S. Bureau of Labor Statistics)
    Occupation ~ female-ness Androgynous name ~ female-ness
    (1990 U.S. census)

    View Slide

  46. Language
    • Caliskan paper
    • True in lots of languages (Lewis & Lupyan, in prep.)

    View Slide

  47. • Motivation:
    • Kids quickly acquire gender stereotypes
    • Also, parents encouraged to read lots of books

    View Slide

  48. The plan
    [1] Estimating gender valence of a book via gender
    valence of words
    • Words - Distinct from other stuff but correlated
    • Books – there’s a distribution (histogram)
    [2] Predictors of book-level gender valence
    • Books are becoming more girl associated over time
    • Target kid age of book: trend for books to get less girl-
    associated across development
    • Girl books have different colors in book covers
    • Girl books have different sentiment trajectories
    [3] Semantics in boy versus girl books

    View Slide

  49. able
    above
    absolutely
    across
    afraid
    after
    afternoon
    again
    against
    ago
    ah
    ahead
    air
    alice
    all
    alligator
    allowed
    almost
    alone
    along
    already
    also
    always
    am
    and
    angry
    animals
    another
    answer
    answered
    any
    anymore
    anyone
    anything
    anyway
    anywhere
    apple
    apples
    arms
    around
    arrived
    ask
    asked
    asleep
    ate
    away
    awful
    baby
    back
    bad
    bag
    ball
    balloon
    band
    bang
    bark
    barn
    basket
    bat
    bath
    bats
    battle
    beach
    beans
    bear
    bears
    beautiful
    became
    because
    become
    bed
    bee
    been
    bees
    before
    began
    beginning
    behind
    being
    believe
    bell
    bells
    belly
    below
    berries
    best
    bet
    better
    between
    big
    bigger
    bike
    bird
    birds
    birthday
    bit
    bite
    black
    blew
    block
    blow
    blue
    boat
    book
    books
    boom
    born
    both
    bottle
    bottom
    bow
    box
    boy
    boys
    brain
    branch
    branches
    bread
    break
    breakfast
    breath
    bridge
    bright
    bring
    broke
    broken
    brother
    brothers
    brought
    brown
    brush
    build
    built
    bump
    bunny
    bus
    bush
    bushes
    busy
    but
    butter
    butterfly
    button
    buttons
    buy
    buzz
    bye
    cake
    call
    called
    calling
    came
    can
    cannot
    cans
    cap
    captain
    car
    care
    carried
    carrots
    carry
    cars
    case
    cat
    catch
    cats
    caught
    cause
    cave
    cellar
    certainly
    chair
    chairs
    change
    chase
    cheese
    chicken
    child
    children
    chin
    chocolate
    choo
    choose
    christmas
    circle
    city
    class
    clean
    clear
    clever
    climb
    climbed
    clock
    close
    closed
    clothes
    clouds
    coat
    cock
    cold
    color
    colors
    come
    comes
    coming
    cookie
    corn
    corner
    could
    count
    countrycourse
    covered
    cow
    cozy
    crawled
    crayon
    crazy
    cream
    cried
    cross
    crow
    cry
    cup
    curious
    cut
    dad
    daddy
    dance
    danced
    dancing
    dark
    day
    days
    dear
    decided
    deep
    delicious
    den
    department
    desk
    did
    different
    dig
    dinner
    dinosaur
    dinosaurs
    dirt
    discovered
    do
    doctor
    does
    dog
    dogs
    doing
    dolls
    done
    door
    doors
    doughnuts
    down
    downstairs
    dr
    dragons
    dream
    dreams
    dress
    dressed
    drink
    drive
    driver
    drives
    dropped
    drum
    dry
    duck
    dug
    dump
    during
    dust
    each
    ear
    early
    ears
    easy
    eat
    eaten
    eating
    egg
    eggs
    eight
    elephant
    elephants
    elevator
    elizabeth
    else
    empty
    end
    engine
    enough
    escape
    even
    evening
    ever
    every
    everybody
    everyone
    everything
    everywhere
    exactly
    except
    excuse
    explained
    extra
    eye
    eyes
    face
    fair
    fall
    fallen
    falling
    family
    far
    farmer
    fast
    faster
    fat
    father
    favorite
    fear
    feel
    feeling
    feels
    feet
    fell
    felt
    fence
    few
    field
    fight filled
    finally
    find
    fine
    finger
    fingers
    finished
    fire
    first
    fish
    fit
    five
    flew
    floor
    flower
    flowers
    fly
    flying
    folks
    follow
    followed
    food
    foot
    forest
    forever
    forget
    forgot
    forth
    found
    four
    fox
    free
    freight
    fresh
    friend
    friendly
    friends
    frightened
    frog
    front
    fruit
    full
    fun
    funny
    fur
    game
    garbage
    garden
    gate
    gave
    general
    george
    get
    gets
    getting
    giant
    girl
    girls
    give
    gives
    glad
    glass
    go
    goat
    god
    goes
    going
    gold
    gone
    goo
    good
    goodbye
    goodnight
    got
    grace
    grandma
    grass
    gray
    great
    green
    grew
    ground
    grow
    grown
    grows
    guess
    had
    hair
    half
    hall
    hand
    hands
    hang
    hanging
    happen
    happened
    happily
    happy
    hard
    hare
    has
    hat
    hate
    have
    having
    hay
    he
    head
    heads
    hear
    heard
    heart
    heavy
    held
    hello
    help
    helped
    hen
    henry
    her
    here
    herself
    hey
    hi
    hid
    hide
    high
    higher
    hill
    hills
    him
    himself
    his
    hold
    hole
    home
    honey
    hoo
    hook
    hooray
    hop
    hope
    horn
    horse
    hot
    hour
    house
    houses
    hug
    huge
    hundred
    hung
    hungry
    hurried
    hurry
    hurt
    hush
    ice
    idea
    if
    important
    indeed
    inside
    instead
    into
    its
    jack
    jam
    job
    join
    jump
    jumped
    jumping
    jungle
    just
    keep
    kept
    kid
    kids
    kind
    king
    kiss
    kissed
    kisses
    kitchen
    kitten
    kittens
    knees
    knew
    knock
    knocked
    know
    knows
    ladder
    lady
    ladybug
    land
    large
    last
    late
    later
    laughed
    laughing
    lay
    lazy
    lead
    least
    leave
    leaves
    led
    left
    leg
    legs
    let
    letter
    library
    life
    lifted
    light
    lights
    like
    liked
    likes
    line
    lion
    list
    listen
    listened
    little
    live
    lived
    lives
    living
    lonely
    long
    longer
    look
    looked
    looking
    looks
    lost
    lot
    lots
    loud
    love
    loved lovely
    loves
    low
    luck
    lucky
    lunch
    ma
    mad
    made
    magic
    mail
    make
    makes
    making
    mama
    man
    many
    mary
    master
    matter
    maybe
    mayor
    me
    mean
    means
    meant
    meet
    meeting
    men
    mess
    met
    middle
    might
    miles
    milk
    mind
    mine
    minute
    mirror
    miss
    mixed
    mom
    moment
    mommy
    money
    monkey
    monkeys monster
    moo
    moon
    more
    morning
    most
    mother
    mountain
    mountains
    mouse
    mouth
    move
    moved
    moving
    mr
    mrs
    much
    mud
    music
    must
    my
    myself
    name
    named
    nap
    near
    neck
    need
    nest
    never
    new
    news
    next
    nice
    night
    nine
    no
    nobody
    noise
    none
    north
    nose
    not
    note
    nothing
    now
    ocean
    oclock
    off
    office
    often
    oh
    okay
    old
    once
    one
    ones
    only
    onto
    open
    opened
    orange
    other
    others
    our
    out
    outside
    over
    owl
    own
    page
    pail
    pants
    papa
    paper
    park
    part
    party
    past
    pat
    paul
    paws
    peanut
    peep
    penguin
    penny
    people
    perfect
    perhaps
    person
    pet
    peter
    piano
    pick
    picked
    pickle
    picnic
    picture
    pictures
    pie
    piece
    pig
    pink
    pip
    place
    places
    plain
    plan
    plate
    play
    played
    playing
    please
    pocket
    poem
    pole
    pond
    poor
    pop
    pot
    pretty
    prince
    princess
    probably
    proud
    pull
    pulled
    puppies
    puppy
    purple
    pushed
    put
    quack
    queen
    quick
    quickly
    quiet
    quietly
    quite
    rabbit
    raced
    rain
    rained
    raised
    ran
    rather
    reach
    reached
    read
    ready
    real
    really
    red
    remember
    replied
    rescue
    rest
    returned
    ride
    riding
    right
    ring
    river
    road
    robin
    rocks
    rode
    rolled
    room
    rope
    rose
    round
    rules
    run
    running
    rushed
    sad
    safe
    said
    salt
    same
    sand
    sandwich
    sang
    sat
    saturday
    saw
    say
    saying
    says
    scared
    school
    scratch
    scratched
    sea
    seat
    second
    secret
    see
    seeds
    seem
    seemed
    seen
    sell
    send
    sent
    set
    seven
    shake
    shall
    share
    she
    sheep
    shiny
    ship
    shoe
    shoes
    shook
    shop
    short
    should
    shout
    shouted
    shovel
    show
    shut
    sick
    side
    sighed
    sight
    sign
    silly
    silver
    since
    sing
    singing
    sir
    sister
    sit
    sitting
    six
    sky
    sleep
    sleeping
    slept
    slide
    slow
    slowly
    small
    smart
    smell
    smile
    smiled
    smoke
    snake
    sneeze
    snow
    so
    socks
    soft
    softly
    some
    somebody
    someone
    something
    sometimes
    somewhere
    son
    song
    soon
    sorry
    sort
    sound
    sounds
    soup
    south
    space
    spaghetti
    speak
    special
    splash
    spoke
    spring
    square
    stairs
    stand
    standing
    star
    stars
    start
    started
    station
    stay
    stayed
    steam
    step
    stepped
    stick
    still
    stinky
    stole
    stone
    stood
    stop
    stopped
    store
    stories
    story
    straight
    strange
    strawberry
    street
    streets
    string
    stripes
    strong
    stuck
    such
    suddenly
    suit
    summer
    sun
    supper
    sure
    surprise
    surprised
    swamp
    sweet
    swim
    swish
    table
    tail
    take
    taken
    takes
    taking
    talk
    talking
    tall
    taste
    tea
    teacher
    teeth
    telephone
    tell
    ten
    tent
    terrible
    terribly
    than
    thank
    thanks
    their
    them
    then
    there
    these
    they thin
    thing
    things
    think
    thinking
    thinks
    third
    those
    though
    thought
    thousand
    three
    threw
    through
    throw
    thumb
    thump
    tickle
    tight
    till time
    times
    tiny
    tired
    today
    toes
    together
    told
    tomorrow
    tonight
    too
    took
    tooth
    top toward
    town
    toy
    toys
    track
    traffic
    train
    trains
    trash
    tree
    trees
    trick
    tricks
    tried
    trip
    trouble
    truck
    trucks
    try
    turn
    turned
    turtle
    turtles
    twenty
    two
    uh
    umbrella
    uncle
    under
    until
    up
    upon
    upstairs
    us
    use
    used
    very
    voice
    wait
    waited
    waiting
    wake
    walk
    walked
    walking
    wall
    want
    wanted
    wants
    warm
    wash
    watch
    watched
    watching
    water
    waved
    way
    we
    wear
    wearing
    wee
    week
    well
    went
    were
    wet
    whatever
    wheels
    which
    while
    whiskers
    whispered
    whistle
    white
    whole
    whoo
    whose
    why
    wide
    wife
    wild
    wind
    window
    windows
    wings
    winter
    wish
    wished
    without
    woke
    wolf
    woman
    wonder
    wondered
    wonderful
    wood
    woods
    woof
    word
    words
    work
    worked
    working
    world
    worried
    worry
    would
    wow
    wren
    write
    writing
    wrong
    wrote
    yard
    year
    years
    yelled
    yellow
    yes
    yet
    you
    young
    your
    yourself
    zoo

    View Slide

  50. thinking verbs
    communcation verbs
    precipitation
    clock/drive
    adverbs
    greeting
    positive emotions
    male words
    numbers
    face/body parts
    container
    male names
    egg
    onomatopoeia
    generic pronouns
    emotions
    wall/fence
    physical affection/positive emotion
    baby words
    misc
    music
    light verbs
    conjunctions
    come/go verbs
    light/dark
    ocean
    building types
    positive adjs
    first/second pronouns
    working
    preparation
    books
    misc
    children
    time quantifiers
    vegetation
    manual tasks
    cognitive verbs
    action verbs
    appearances/misc
    mind
    money
    conversational (e.g. ok, yes)
    ordinal
    life
    misc/prepositions
    distance metrics
    valence adjs (awful)
    emotion verbs
    motion verbs
    preposition
    non−mammal s
    adverbs
    geographic features
    thing
    misc
    motion verbs
    adverbial quantifiers
    action verbs
    sounds
    eating/drinking
    sky
    colors
    people pronouns
    locamative verbs
    cognitive vebrs
    quantifiers
    sweet
    language verbs
    garden
    food
    star
    body parts
    house parts
    games
    big vehicles
    misc adjs
    clothing
    temperature
    holiday/time
    action verbs
    helper verbs
    size adjs
    mammals
    prepositions
    fruit
    magic
    family
    fire/smoke
    misc
    furniture
    seasons/boat
    time
    space descripters
    watch/like/crazy
    desk
    ability/adverbs
    vehicles
    preposition/spatial
    eating location

    View Slide

  51. thinking verbs
    communcation verbs
    precipitation
    clock/drive
    adverbs
    greeting
    positive emotions
    male words
    numbers
    face/body parts
    container
    male names
    egg
    onomatopoeia
    generic pronouns
    emotions
    wall/fence
    physical affection/positive emotion
    baby words
    misc
    music
    light verbs
    conjunctions
    come/go verbs
    light/dark
    ocean
    building types
    positive adjs
    first/second pronouns
    working
    preparation
    books
    misc
    children
    time quantifiers
    vegetation
    manual tasks
    cognitive verbs
    action verbs
    appearances/misc
    mind
    money
    conversational (e.g. ok, yes)
    ordinal
    life
    misc/prepositions
    distance metrics
    valence adjs (awful)
    emotion verbs
    motion verbs
    preposition
    non−mammal s
    adverbs
    geographic features
    thing
    misc
    motion verbs
    adverbial quantifiers
    action verbs
    sounds
    eating/drinking
    sky
    colors
    people pronouns
    locamative verbs
    cognitive vebrs
    quantifiers
    sweet
    language verbs
    garden
    food
    star
    body parts
    house parts
    games
    big vehicles
    misc adjs
    clothing
    temperature
    holiday/time
    action verbs
    helper verbs
    size adjs
    mammals
    prepositions
    fruit
    magic
    family
    fire/smoke
    misc
    furniture
    seasons/boat
    time
    space descripters
    watch/like/crazy
    desk
    ability/adverbs
    vehicles
    preposition/spatial
    eating location
    1.5 2.0 2.5 3.0 3.5
    mean female
    gender rating

    View Slide

  52. Implicit Association Test (IAT)
    Categories
    X = {marigold, poppy, azalea, crocus, iris}
    Y = {ant, caterpillar, flea, bedbug, spider}
    Attributes
    A = {cheer, freedom, health, love, peace}
    B = {abuse, crash, filth, murder, sickness}
    Participants slower for incongruent mapping (right), suggesting
    bias to associate flowers with pleasantness
    https://implicit.harvard.edu/implicit/takeatest.html
    marigold
    cheer
    ant
    abuse
    compare reaction time
    marigold
    cheer
    ant
    abuse

    View Slide

  53. Predicting gender valence
    • Book covers
    • Books are becoming more girl associated over time
    • Target kid age of book: trend for books to get less
    girl-associated across development
    • Sentiment trajectory

    View Slide

  54. fear
    negative
    sadness
    anger disgust
    positive
    trust
    surprise
    joy
    anticipation
    Deriving word-level measures of
    valence from word embeddings

    View Slide

  55. Deriving word-level measures of
    valence from word embeddings
    • Large-scale distributional semantics models derived from auto-
    encoding neural networks trained on large text corpora.
    • Assumption: meaning of a word can be described by the words
    it tends to co-occur with—words occurring in similar contexts,
    tend to have similar meanings (Firth, 1957).
    • E.g , “dog” and “hound” more similar than “dog” and “banana”
    because “dog” co-occurs with words more in common with
    “hound” than “banana.”
    • Recent developments in machine learning allow the idea of
    distributional semantics
    • to be implemented in a way that takes into account many
    features of language structure while remaining computationally
    tractable. The best known of these word embedding models is
    word2vec (Mikolov, Chen, Corrado, & Dean, 2013). The model

    View Slide

  56. BABY TOUCH AND FEEL: ANIMALS, BARNYARD DANCE
    BLUE HAT GREEN HAT, BUT NOT THE HIPPOPOTAMUS
    DOGS, EVERYONE POOPS
    HAPPY BIRTHDAY BABY!, HIPPOS GO BERSERK
    HORNS TO TOES AND IN BETWEEN, HOW DO DINOSAURS SAY GOOD NIGHT?
    HOW DO DINOSAURS SAY HAPPY BIRTHDAY?, I AM A BIG SISTER!
    I AM A BUNNY, MAISY GOES CAMPING
    MOO BAA LA LA LA!, NIGHT−NIGHT LITTLE POOKIE
    OH MY OH MY OH MY DINOSAURS!, PAJAMA TIME!
    THE CARROT SEED, THE GOING TO BED BOOK
    CAPS FOR SALE, GRANDFATHER'S JOURNEY
    HAROLD AND THE PURPLE CRAYON, KATY AND THE BIG SNOW
    LOVE YOU FOREVER, THE COLOR KITTENS
    THE FANTASTIC FLYING BOOKS OF MR. MORRIS LESSMORE, THE KEEPING QUILT
    THE STORY OF FERDINAND, TRAIN
    WINTER DAYS IN THE BIG WOODS
    ALEXANDER AND THE TERRIBLE HORRIBLE NO GOOD VERY BAD DAY, ARE YOU MY MOTHER?
    ARTHUR WRITES A STORY, FOX IN SOCKS
    GEORGE AND MARTHA, GOLDILICIOUS
    HOW TO BABYSIT A GRANDMA, HOW TO TRAIN A TRAIN
    I LOVE YOU STINKY FACE, MVP: MOST VALUABLE PUPPY
    THE GIVING TREE, THE GROUCHY LADYBUG
    THE GRUFFALO, THE OTHER SIDE
    THE TRUE STORY OF THE 3 LITTLE PIGS!, THOMAS AND FRIENDS: GO TRAIN GO!
    BOB BOOKS SET : BEGINNING READERS, DR. SEUSS'S ABC
    PETE THE CAT: THE WHEELS ON THE BUS, THE WHEELS ON THE BUS
    ALLIGATOR PIE, ARE YOU A COW?
    DEAR ZOO, DON...T LET THE PIGEON DRIVE THE BUS
    GOOD DOG CARL, GOOD NIGHT GORILLA
    HUG, I LOVE MY MOMMY BECAUSE...
    I LOVE YOU THROUGH AND THROUGH, LLAMA LLAMA NIGHTY−NIGHT
    NO DAVID!, POTTY
    THAT IS NOT A GOOD IDEA!, TODAY I...LL BE A PRINCESS
    TUESDAY, WHERE IS BABY'S BELLY BUTTON
    WHERE'S SPOT?
    CHUGGA−CHUGGA CHOO−CHOO, DOGGIES
    EACH PEACH PEAR PLUM, ELEPHANT AND PIGGY: TODAY I WILL FLY!
    GOODNIGHT MOON, GRANDMA WISHES
    JAMBERRY, JUST ME AND MY DAD
    NUMBERS COLORS SHAPES, RAIN MAKES APPLESAUCE
    SNUGGLE PUPPY, SOMETIMES I LIKE TO CURL UP IN A BALL
    TEN LITTLE LADYBUGS, THE PIGEON FINDS A HOT DOG!
    THE WONDERFUL THINGS YOU WILL BE, THIS IS NOT MY HAT
    TICKLE TIME, WELCOME LITTLE ONE
    ARNIE THE DOUGHNUT, BREAD AND JAM FOR FRANCES
    CARS AND TRUCKS AND THINGS THAT GO, CHRYSANTHEMUM
    ELOISE, FROG AND TOAD
    GREEN EGGS AND HAM, HORTON HEARS A WHO!
    KINDERGARTEN HERE I COME!, OH THE PLACES YOU...LL GO
    ONE FISH TWO FISH RED FISH BLUE FISH, STELLALUNA
    THE BERENSTAIN BEARS FORGET THEIR MANNERS, THE BIG HONEY HUNT
    THE CAT IN THE HAT, THE DAY THE CRAYONS QUIT
    THE GARDENER, THE LITTLE ENGINE THAT COULD
    THE LORAX, THE STINKY CHEESE MAN AND OTHER FAIRLY STUPID TALES
    THOMAS THE TANK ENGINE: THE VERY FIRST STORIES, WHEN DINOSAURS CAME WITH EVERYTHING
    A BAD CASE OF STRIPES, AMELIA BEDELIA
    BRAVE IRENE, CURIOUS GEORGE TAKES A JOB
    LIBRARY LION, LILLY...S PURPLE PLASTIC PURSE
    MIKE MULLIGAN AND HIS STEAM SHOVEL, MILLIONS OF CATS
    MISS RUMPHIUS, SAYONARA MRS. KACKLEMAN
    SHE PERSISTED: 3 AMERICAN WOMEN WHO CHANGED THE WORLD, SPAGHETTI IN A HOT DOG BUN: HAVING THE COURAGE TO BE WHO YOU ARE
    SYLVESTER AND THE MAGIC PEBBLE, THE BERENSTAIN BEARS AND THE GREEN−EYED MONSTER
    THE POLAR EXPRESS, THE RED BALLOON
    THE STORY OF BABAR, THE THREE QUESTIONS
    YERTLE THE TURTLE
    ARTHUR...S NOSE, B IS FOR BEAR
    BABY BELUGA, CHICKA CHICKA −2−3
    CHICKA CHICKA BOOM BOOM, HOP ON POP
    KNUFFLE BUNNY, MAISY GOES TO THE LIBRARY
    PAT THE BUNNY, SLOW LORIS
    TEN LITTLE FINGERS AND TEN LITTLE TOES, TIME FOR BED
    LITTLE BLUE TRUCK, MR. BROWN CAN MOO! CAN YOU?
    TRASHY TOWN, WE'RE GOING ON A BEAR HUNT
    BABY BEAR BABY BEAR WHAT DO YOU SEE?, BIG SHARK LITTLE SHARK
    BROWN BEAR BROWN BEAR WHAT DO YOU SEE?, FIRST
    WORDS
    FROM HEAD TO TOE, LLAMA LLAMA HOME WITH MAMA
    LLAMA LLAMA RED PAJAMA, POLAR BEAR POLAR BEAR WHAT DO YOU HEAR?
    THE FOOT BOOK
    ANGELINA ICE SKATES, CORDUROY
    EXTRA YARN, FROGGY GOES TO BED
    GOODNIGHT GOODNIGHT CONSTRUCTION SITE, HOW ROCKET LEARNED TO READ
    MIGHTY MONSTER MACHINES (BLAZE AND THE MONSTER MACHINES), MISS NELSON IS MISSING
    OWN MOON, PUPPY BIRTHDAY TO YOU! (PAW PATROL)
    THE GRUFFALO'S CHILD, THE PAPER BAG PRINCESS
    BELLY BUTTON BOOK, CLIFFORD THE BIG RED DOG
    GOD BLESS YOU AND GOOD NIGHT, GRUMPY BIRD
    HELLO ROCK, I...M A BIG SISTER
    IF YOU EVER WANT TO BRING A PIANO TO THE BEACH DON'T!, IF YOU GIVE A MOOSE A MUFFIN
    IF YOU GIVE A MOUSE A COOKIE, OLIVIA
    OLIVIA. . . AND THE MISSING TOY, ON THE NIGHT YOU WERE BORN
    P IS FOR POTTY!, PETE THE CAT: ROCKING IN MY SCHOOL SHOES
    PRESS HERE, THERE'S A WOCKET IN MY POCKET!
    THERE...S AN ALLIGATOR UNDER MY BED, TRIANGLE (SHAPE TRILOGY)
    WHEREVER YOU ARE MY LOVE WILL FIND YOU
    ANIMALIA, BARK GEORGE
    DINOSAUR RESCUE, HAND HAND FINGERS THUMB
    I WISH YOU MORE, MAMA DON'T ALLOW
    THE VERY HUNGRY CATERPILLAR, WHERE IS THE GREEN SHEEP?
    JENNY AND THE CAT CLUB, THE REAL MOTHER GOOSE
    WHERE THE SIDEWALK ENDS, WINNIE THE POOH
    A SICK DAY FOR AMOS MCGEE, BUNNY PARTY
    CLIFFORD AT THE CIRCUS, GIRAFFES CAN'T DANCE
    GO DOG. GO!, HARRY THE DIRTY DOG
    HOW TO CATCH A STAR, LOST AND FOUND
    MADELINE, SHOW DOG
    THE HAT, THE NIGHT BEFORE MOTHER'S DAY
    THE POUT−POUT FISH, WHISTLE FOR WILLIE
    CHARLIE AND THE NEW BABY, GUESS HOW MUCH I LOVE YOU
    I WANT MY HAT BACK, LITTLE BEAR: WHAT WILL LITTLE BEAR WEAR?
    MORTIMER, OH THE THINKS YOU CAN THINK!
    SQUARE, THE DUCKLING GETS A COOKIE!?
    THE IMPORTANT BOOK, THE MONSTER AT THE END OF THIS BOOK
    THE RUNAWAY BUNNY, YOU GET WHAT YOU GET
    BIG RED BARN, LITTLE BLUE TRUCK LEADS THE WAY
    LITTLE OWL...S NIGHT, THE NAPPING HOUSE
    BEAR SNORES ON, BEAR WANTS MORE
    CLICK CLACK MOO COWS THAT TYPE, DIARY OF A WOMBAT
    DRAGONS LOVE TACOS, DUCK ON A BIKE
    IF ANIMALS KISSED GOOD NIGHT, IN THE NIGHT KITCHEN
    THE SNOWY DAY, WHERE DO DIGGERS SLEEP AT NIGHT?
    WHERE THE WILD THINGS ARE
    BLUEBERRIES FOR SAL, CLOUDY WITH A CHANCE OF MEATBALLS
    CURIOUS GEORGE, LADYBUG GIRL AT THE BEACH
    LON PO PO: A RED RIDING HOOD STORY FROM CHINA, MAKE WAY FOR DUCKLINGS
    THE GARDEN OF ABDUL GASAZI, THE LITTLE HOUSE
    THE POKY LITTLE PUPPY, THE TALE OF PETER RABBIT
    TIKKI TIKKI TEMBO
    0.00 0.05 0.10
    mean female
    gender rating
    0.00 0.05 0.10
    mean female
    gender rating
    Gender book valence and semantic
    space

    View Slide

  57. View Slide

  58. 0.00 0.05 0.10
    mean female
    gender rating
    Gender book valence and semantic
    space
    Clusters of book centroids

    View Slide