Implementing Finite State Intonational Grammars to Understand Gradient Prosodic Manipulations in Infant-directed Speech

Implementing Finite State Intonational Grammars to Understand Gradient Prosodic Manipulations in Infant-directed Speech

Joint work with Sameer ud Dowla Khan and Megha Sundara.
Hanyang International Symposium on Phonetics and Cognitive Sciences of Language 2018. http://site.hanyang.ac.kr/web/hisphoncog/

20a8ff44959a902d76386e2a75592154?s=128

krisyu

May 18, 2018
Tweet

Transcript

  1. IMPLEMENTING FINITE STATE GRAMMARS FOR UNDERSTANDING PROSODIC MANIPULATIONS IN INFANT-DIRECTED

    SPEECH KRISTINE M. YU UMASS AMHERST SAMEER UD DOWLA KHAN REED COLLEGE MEGHA SUNDARA UCLA HISPHONCOG, HANYANG, KOREA MAY 2018
  2. COLLABORATORS 2 Sameer ud Dowla Khan Megha Sundara

  3. RESEARCH QUESTION 3 Many subphonemic and gradient aspects of speech,

    whether segmental or suprasegmental, which were once considered to be beyond the speaker’s control, are now understood as part of the linguistic system stipulated by the grammar of a given language. What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  4. CHALLENGES 4 What are the linguistic functions of gradient modulations

    in the fundamental frequency contour?
  5. CHALLENGES 4 ‣ the entanglement of extra-linguistic and linguistic factors

    in conditioning f0 variation What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  6. CHALLENGES 4 ‣ the entanglement of extra-linguistic and linguistic factors

    in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  7. CHALLENGES 4 ‣ the entanglement of extra-linguistic and linguistic factors

    in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  8. CHALLENGES 4 ‣ the entanglement of extra-linguistic and linguistic factors

    in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts ‣ the contextual dependence of individual tonal elements on one another What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  9. CHALLENGES 4 ‣ the entanglement of extra-linguistic and linguistic factors

    in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts ‣ the contextual dependence of individual tonal elements on one another What are the linguistic functions of gradient modulations in the fundamental frequency contour? Proposed overall strategy: To implement finite state intonational grammars
  10. 5 https://timedotcom.files.wordpress.com/2014/10/89845500.jpg Infant directed speech (IDS) Case study of a

    particular speech style/variety
  11. 6 CHALLENGE 1: ENTANGLEMENT OF EXTRA-LINGUISTIC AND LINGUISTIC FACTORS

  12. CHALLENGE: EXTRA-LINGUISTIC/LINGUISTIC 7 What are the linguistic functions of gradient

    modulations in the fundamental frequency contour?
  13. CHALLENGE: EXTRA-LINGUISTIC/LINGUISTIC 7 ‣ the entanglement of extra-linguistic and linguistic

    factors in conditioning f0 variation What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  14. CHALLENGE: EXTRA-LINGUISTIC/LINGUISTIC 7 ‣ the entanglement of extra-linguistic and linguistic

    factors in conditioning f0 variation What are the linguistic functions of gradient modulations in the fundamental frequency contour? Proposed strategy: Analyze f0 contours using intonational phonology
  15. PROPERTIES OF (CANONICAL) INFANT-DIRECTED SPEECH 8 ▸ Higher and wider

    fundamental frequency (f0) range ▸ Slower speech rate; more/longer pauses Fernald and Simon (1984), Fernald and Mazzie (1991), Albin and Echols (1996), Kitamura et al. (2002), i.a.
  16. IDS AS A SOCIO-AFFECTIVE SIGNAL 9 ▸ Rising pitch contours

    for eliciting attention ▸ Sinusoidal and bell-shaped pitch contours used for maintaining attention and positive rapport (Stern et al. 1982)
  17. BENGALI INFANT-DIRECTED SPEECH 10 Adult-directed speech 10

  18. BENGALI INFANT-DIRECTED SPEECH 10 Adult-directed speech 10

  19. BENGALI INFANT-DIRECTED SPEECH 10 Adult-directed speech 10

  20. BENGALI INFANT-DIRECTED SPEECH 10 Infant-directed speech Adult-directed speech 10

  21. BENGALI INFANT-DIRECTED SPEECH 10 Infant-directed speech Adult-directed speech 10

  22. BENGALI INFANT-DIRECTED SPEECH 10 Infant-directed speech Adult-directed speech 10

  23. BENGALI INFANT-DIRECTED SPEECH 10 Infant-directed speech Adult-directed speech 10

  24. BENGALI INFANT-DIRECTED SPEECH 10 Infant-directed speech Adult-directed speech 10 Not

    just wider f0 range, but different kinds of tones
  25. STRATEGY: INTONATIONAL PHONOLOGY 11 What are the linguistic functions of

    gradient modulations in the fundamental frequency contour?
  26. STRATEGY: INTONATIONAL PHONOLOGY 11 Challenge: What are the linguistic functions

    of gradient modulations in the fundamental frequency contour?
  27. STRATEGY: INTONATIONAL PHONOLOGY 11 Challenge: ‣ the entanglement of extra-linguistic

    and linguistic factors in conditioning f0 variation What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  28. STRATEGY: INTONATIONAL PHONOLOGY 11 Challenge: ‣ the entanglement of extra-linguistic

    and linguistic factors in conditioning f0 variation Strategy: What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  29. STRATEGY: INTONATIONAL PHONOLOGY 11 Challenge: ‣ the entanglement of extra-linguistic

    and linguistic factors in conditioning f0 variation Strategy: ‣ Analyze f0 contours as well-formed sequences of tonal elements (sequences derived from a finite state tonotactic intonational grammar) What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  30. STRATEGY: INTONATIONAL PHONOLOGY 11 Challenge: ‣ the entanglement of extra-linguistic

    and linguistic factors in conditioning f0 variation Strategy: ‣ Analyze f0 contours as well-formed sequences of tonal elements (sequences derived from a finite state tonotactic intonational grammar) ‣ Igarashi et al. (2013): Tokyo Japanese IDS only apparently not a wider f0 range; wider f0 range if looking just at boundary tones What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  31. 12 TONOTACTIC GRAMMARS: TOKYO JAPANESE (Pierrehumbert and Beckman, 1988; Maekawa

    et al., 2002; Venditti, 2005; Igarashi et al. 2013)
  32. 12 TONOTACTIC GRAMMARS: TOKYO JAPANESE (Pierrehumbert and Beckman, 1988; Maekawa

    et al., 2002; Venditti, 2005; Igarashi et al. 2013) “Boundary pitch movement” tones: locus of f0 range expansion
  33. 13 TONOTACTIC GRAMMARS: ENGLISH Pierrehumbert (1980), Pierrehumbert & Beckman (1986),

    Pierrehumbert (2000), Beckman et al. (2005), i.a. “Mainstream American English (MAE)”
  34. 13 TONOTACTIC GRAMMARS: ENGLISH Pierrehumbert (1980), Pierrehumbert & Beckman (1986),

    Pierrehumbert (2000), Beckman et al. (2005), i.a. “Mainstream American English (MAE)” If a sequence can be generated via a path through the machine, then it is well-formed.
  35. STRATEGY: INTONATIONAL PHONOLOGY 14 Challenge: ‣ the entanglement of extra-linguistic

    and linguistic factors in conditioning f0 variation Strategy: ‣ Analyze f0 contours as well-formed sequences of tonal elements (sequences derived from a finite state tonotactic intonational grammar) ‣ Igarashi et al. (2013): Tokyo Japanese IDS only apparently not a wider f0 range; wider f0 range if looking just at boundary tones What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  36. STRATEGY: INTONATIONAL PHONOLOGY 14 Challenge: ‣ the entanglement of extra-linguistic

    and linguistic factors in conditioning f0 variation Strategy: ‣ Analyze f0 contours as well-formed sequences of tonal elements (sequences derived from a finite state tonotactic intonational grammar) ‣ Igarashi et al. (2013): Tokyo Japanese IDS only apparently not a wider f0 range; wider f0 range if looking just at boundary tones What are the linguistic functions of gradient modulations in the fundamental frequency contour? Why finite state grammar?
  37. 15 LEXICON AND FINITE/FINITE STATE GRAMMARS ‣ A grammar derives

    the set of well-formed tonal sequences over a lexicon of tonal elements, i.e., a tonal inventory Common for intonational grammars to be expressed/summarized as a lexicon
  38. 16 CHICKASAW INTONATIONAL LEXICON Gordon (2005)

  39. 17 BENGALI INTONATIONAL LEXICON Khan (2014)

  40. 18 LEXICON AND FINITE/FINITE STATE GRAMMARS ‣ A grammar derives

    the set of well-formed tonal sequences over a lexicon of tonal elements ‣ A finite grammar is just a list of these well-formed sequences ‣ No generalizations Also common for intonational grammars to be expressed/summarized as a finite grammar (list)
  41. 19 Pierrehumbert & Hirschberg (1990) WELL-FORMED ENGLISH TONAL SEQUENCES: LIST

  42. 20 WELL-FORMED FRIULIAN TONAL SEQUENCES: LIST Roseano et al. (2015)

  43. 21 LEXICON, FINITE, AND FINITE STATE GRAMMARS ‣ A grammar

    derives the set of well-formed tonal sequences over a lexicon of tonal elements ‣ A finite grammar is just a list of these well-formed sequences ‣ No generalizations ‣ A finite state grammar compresses the list by encoding generalizations from shared prefixes in well-formed sequences
  44. BENGALI ACCENTUAL PHRASES 22 100 150 200 250 300 Fundamental

    frequency (Hz) Time (s) 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 3.2 L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L*L% L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% rumu nepaler ranir malider namgulo mone rakhte pare ni Rumu Nepal’s queen’s the gardeners’ the names remember couldn’t `Rumu couldn’t remember the names of the gardeners of the queen of Nepal.’ Khan (2008)
  45. BENGALI ACCENTUAL PHRASES 22 100 150 200 250 300 Fundamental

    frequency (Hz) Time (s) 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 3.2 L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L*L% L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% rumu nepaler ranir malider namgulo mone rakhte pare ni Rumu Nepal’s queen’s the gardeners’ the names remember couldn’t `Rumu couldn’t remember the names of the gardeners of the queen of Nepal.’ Khan (2008)
  46. 23 FINITE GRAMMAR (LIST) FOR BENGALI ‣ L* Ha L*

    L% ‣ L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% Repeated L* Ha chunk is a coincidence, no generalization
  47. 24 FINITE STATE GRAMMAR FOR BENGALI* Khan (2008, 2014), Yu,

    Khan & Sundara (under revision) IP AP Ha PA L* Ha AP La PA H* La L* H* ip L- H- L% H% LH% HL% HLH% *non-focus contexts, typical tunes, no “over- ride” of boundary tones
  48. 24 FINITE STATE GRAMMAR FOR BENGALI* Khan (2008, 2014), Yu,

    Khan & Sundara (under revision) IP AP Ha PA L* Ha AP La PA H* La L* H* ip L- H- L% H% LH% HL% HLH% *non-focus contexts, typical tunes, no “over- ride” of boundary tones
  49. 25 CHALLENGES 2/3: EVOLVING HYPOTHESES, GENERALIZABILITY TO LANGUAGE VARIETIES

  50. REVISIONS OF MAE TOBI 26 (Jun 2011)

  51. GUSSENHOVEN (2004, 2016): REVISION OF MAE TOBI 27

  52. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 What are the

    linguistic functions of gradient modulations in the fundamental frequency contour?
  53. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: What are

    the linguistic functions of gradient modulations in the fundamental frequency contour?
  54. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving

    hypotheses about intonational inventory, tonotactics What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  55. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving

    hypotheses about intonational inventory, tonotactics ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  56. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving

    hypotheses about intonational inventory, tonotactics ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  57. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving

    hypotheses about intonational inventory, tonotactics ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Write down current proposed grammar as finite state grammar and compile as finite state machine What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  58. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving

    hypotheses about intonational inventory, tonotactics ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Write down current proposed grammar as finite state grammar and compile as finite state machine ‣ Check intonational transcriptions in new corpus by parsing them with finite state machine What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  59. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving

    hypotheses about intonational inventory, tonotactics ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Write down current proposed grammar as finite state grammar and compile as finite state machine ‣ Check intonational transcriptions in new corpus by parsing them with finite state machine ‣ Diagnose problems → revisions to intonational grammar What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  60. DEFINING BENGALI GRAMMAR IN xfst (excerpts) 29 xfst: Beesley &

    Karttunen (2003)
  61. DEFINING BENGALI GRAMMAR IN xfst (excerpts) 30

  62. DEFINING BENGALI GRAMMAR IN xfst (excerpts) 31

  63. 32 IMPLEMENTED FINITE STATE GRAMMARS English Bengali

  64. 32 IMPLEMENTED FINITE STATE GRAMMARS English Bengali 1. For each

    utterance in corpus, parse intonational transcription using finite state machine: accepted or not? 2. Over corpus, count up how many times each arc traversed to estimate arc weights (probability that an arc is traversed)
  65. PARSING WITH BENGALI FINITE STATE MACHINE 33 ‣ Failed to

    accept 1.5% of exemplars ‣ (21/1367 total; 9/549 in non-IDS, 12/818 in IDS) ‣ Characteristics of the rejected exemplars were the same across speech styles ‣ New `stacked’ boundary tone fHaL% ‣ Unexpected sequences of pitch accents ‣ Distribution of weak accents (*), e.g., ‣ Characteristics of unaccepted tonal sequences provide direction for revisions to grammar (evolving hypotheses)
  66. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 What are the

    linguistic functions of gradient modulations in the fundamental frequency contour?
  67. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: What are

    the linguistic functions of gradient modulations in the fundamental frequency contour?
  68. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: ‣ the

    generalizability of proposed grammars to a wider range of speech styles and contexts What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  69. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: ‣ the

    generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  70. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: ‣ the

    generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Check if all IDS transcriptions are accepted by finite state machine What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  71. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: ‣ the

    generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Check if all IDS transcriptions are accepted by finite state machine ‣ Estimate arc weights on finite state machine using: (1) non-IDS corpus, (2) IDS corpus. What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  72. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: ‣ the

    generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Check if all IDS transcriptions are accepted by finite state machine ‣ Estimate arc weights on finite state machine using: (1) non-IDS corpus, (2) IDS corpus. ‣ See how probable IDS transcriptions are using non-IDS arc weights and vice versa What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  73. HOW PROBABLE ARE IDS SEQUENCES? 35 non−ids ids 0 1

    2 3 4 5 Probability ratio Distribution for each speech style IDS tonal sequences markedly more probable under IDS-tuned FSG than non-IDS-tuned FSG
  74. HOW PROBABLE ARE IDS SEQUENCES? 35 non−ids ids 0 1

    2 3 4 5 Probability ratio Distribution for each speech style Probability ratio = ratio of the probability assigned by the IDS-tuned probabilistic FSG to the probability assigned by non-IDS-tuned probabilistic FSG IDS tonal sequences markedly more probable under IDS-tuned FSG than non-IDS-tuned FSG
  75. HOW PROBABLE ARE IDS SEQUENCES? 35 non−ids ids 0 1

    2 3 4 5 Probability ratio Distribution for each speech style IDS tonal sequences markedly more probable under IDS-tuned FSG than non-IDS-tuned FSG
  76. 36 CHALLENGE 4: CONTEXTUAL DEPENDENCE OF TONES

  77. BAG OF TONES MODEL 37 L% LH% HLH% H% HL%

    0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.00 0.05 0.10 0.15 0.000 0.025 0.050 0.075 0.100 0.125 0.00 0.02 0.04 0.06 non−IDS IDS non−IDS IDS non−IDS IDS non−IDS IDS non−IDS IDS Style Mean relative frequency Intonational phrase L% LH% HLH% H% HL% Yu, Khan & Sundara (2014)
  78. BAG OF TONES MODEL 37 L% LH% HLH% H% HL%

    0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.00 0.05 0.10 0.15 0.000 0.025 0.050 0.075 0.100 0.125 0.00 0.02 0.04 0.06 non−IDS IDS non−IDS IDS non−IDS IDS non−IDS IDS non−IDS IDS Style Mean relative frequency Intonational phrase L% LH% HLH% H% HL% Yu, Khan & Sundara (2014) Ex: Count up how many times there was an HLH% in non-IDS vs. IDS
  79. 38 FIRST-ORDER MODEL Khan (2008, 2014), Yu, Khan & Sundara

    (under revision) IP AP Ha PA L* Ha AP La PA H* La L* H* ip L- H- L% H% LH% HL% HLH%
  80. 38 FIRST-ORDER MODEL Khan (2008, 2014), Yu, Khan & Sundara

    (under revision) IP AP Ha PA L* Ha AP La PA H* La L* H* ip L- H- L% H% LH% HL% HLH% Computed probabilities of traversing an arc conditioned on probabilities of reaching that state emitting arc
  81. STRATEGY: INTONATIONAL PHONOLOGY 39 What are the linguistic functions of

    gradient modulations in the fundamental frequency contour?
  82. STRATEGY: INTONATIONAL PHONOLOGY 39 Challenges: ‣ the contextual dependence of

    individual tonal elements on one another Strategy: ‣ Compute probability of an individual tonal element conditioned on probability of paths that arrive at that state What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  83. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 What are the linguistic

    functions of gradient modulations in the fundamental frequency contour?
  84. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 ‣ the entanglement of

    extra-linguistic and linguistic factors in conditioning f0 variation What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  85. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 ‣ the entanglement of

    extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  86. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 ‣ the entanglement of

    extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  87. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 ‣ the entanglement of

    extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts ‣ the contextual dependence of individual tonal elements on one another What are the linguistic functions of gradient modulations in the fundamental frequency contour?
  88. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 ‣ the entanglement of

    extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts ‣ the contextual dependence of individual tonal elements on one another What are the linguistic functions of gradient modulations in the fundamental frequency contour? Proposed overall strategy: To implement finite state intonational grammars
  89. (2ND ORDER MODEL, IF MORE DATA) 41 (Dainora 2006)

  90. FSG FOR ENGLISH INTONATION 42 Gussenhoven (2004, 2016)

  91. COMPARING FSGS FOR FRENCH AND ENGLISH 43