Implementing Finite State Intonational Grammars to Understand Gradient Prosodic Manipulations in Infant-directed Speech

Joint work with Sameer ud Dowla Khan and Megha Sundara.
Hanyang International Symposium on Phonetics and Cognitive Sciences of Language 2018. http://site.hanyang.ac.kr/web/hisphoncog/

Transcript

1. IMPLEMENTING FINITE STATE GRAMMARS FOR UNDERSTANDING PROSODIC MANIPULATIONS IN INFANT-DIRECTED SPEECH KRISTINE M. YU UMASS AMHERST SAMEER UD DOWLA KHAN REED COLLEGE MEGHA SUNDARA UCLA HISPHONCOG, HANYANG, KOREA MAY 2018
2. COLLABORATORS 2 Sameer ud Dowla Khan Megha Sundara
3. RESEARCH QUESTION 3 Many subphonemic and gradient aspects of speech, whether segmental or suprasegmental, which were once considered to be beyond the speaker’s control, are now understood as part of the linguistic system stipulated by the grammar of a given language. What are the linguistic functions of gradient modulations in the fundamental frequency contour?
4. CHALLENGES 4 What are the linguistic functions of gradient modulations in the fundamental frequency contour?
5. CHALLENGES 4 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation What are the linguistic functions of gradient modulations in the fundamental frequency contour?
6. CHALLENGES 4 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars What are the linguistic functions of gradient modulations in the fundamental frequency contour?
7. CHALLENGES 4 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts What are the linguistic functions of gradient modulations in the fundamental frequency contour?
8. CHALLENGES 4 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts ‣ the contextual dependence of individual tonal elements on one another What are the linguistic functions of gradient modulations in the fundamental frequency contour?
9. CHALLENGES 4 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts ‣ the contextual dependence of individual tonal elements on one another What are the linguistic functions of gradient modulations in the fundamental frequency contour? Proposed overall strategy: To implement finite state intonational grammars
10. 5 https://timedotcom.files.wordpress.com/2014/10/89845500.jpg Infant directed speech (IDS) Case study of a particular speech style/variety
11. 6 CHALLENGE 1: ENTANGLEMENT OF EXTRA-LINGUISTIC AND LINGUISTIC FACTORS
12. CHALLENGE: EXTRA-LINGUISTIC/LINGUISTIC 7 What are the linguistic functions of gradient modulations in the fundamental frequency contour?
13. CHALLENGE: EXTRA-LINGUISTIC/LINGUISTIC 7 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation What are the linguistic functions of gradient modulations in the fundamental frequency contour?
14. CHALLENGE: EXTRA-LINGUISTIC/LINGUISTIC 7 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation What are the linguistic functions of gradient modulations in the fundamental frequency contour? Proposed strategy: Analyze f0 contours using intonational phonology
15. PROPERTIES OF (CANONICAL) INFANT-DIRECTED SPEECH 8 ▸ Higher and wider fundamental frequency (f0) range ▸ Slower speech rate; more/longer pauses Fernald and Simon (1984), Fernald and Mazzie (1991), Albin and Echols (1996), Kitamura et al. (2002), i.a.
16. IDS AS A SOCIO-AFFECTIVE SIGNAL 9 ▸ Rising pitch contours for eliciting attention ▸ Sinusoidal and bell-shaped pitch contours used for maintaining attention and positive rapport (Stern et al. 1982)
17. BENGALI INFANT-DIRECTED SPEECH 10 Adult-directed speech 10
18. BENGALI INFANT-DIRECTED SPEECH 10 Adult-directed speech 10
19. BENGALI INFANT-DIRECTED SPEECH 10 Adult-directed speech 10
20. BENGALI INFANT-DIRECTED SPEECH 10 Infant-directed speech Adult-directed speech 10
21. BENGALI INFANT-DIRECTED SPEECH 10 Infant-directed speech Adult-directed speech 10
22. BENGALI INFANT-DIRECTED SPEECH 10 Infant-directed speech Adult-directed speech 10
23. BENGALI INFANT-DIRECTED SPEECH 10 Infant-directed speech Adult-directed speech 10
24. BENGALI INFANT-DIRECTED SPEECH 10 Infant-directed speech Adult-directed speech 10 Not just wider f0 range, but different kinds of tones
25. STRATEGY: INTONATIONAL PHONOLOGY 11 What are the linguistic functions of gradient modulations in the fundamental frequency contour?
26. STRATEGY: INTONATIONAL PHONOLOGY 11 Challenge: What are the linguistic functions of gradient modulations in the fundamental frequency contour?
27. STRATEGY: INTONATIONAL PHONOLOGY 11 Challenge: ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation What are the linguistic functions of gradient modulations in the fundamental frequency contour?
28. STRATEGY: INTONATIONAL PHONOLOGY 11 Challenge: ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation Strategy: What are the linguistic functions of gradient modulations in the fundamental frequency contour?
29. STRATEGY: INTONATIONAL PHONOLOGY 11 Challenge: ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation Strategy: ‣ Analyze f0 contours as well-formed sequences of tonal elements (sequences derived from a finite state tonotactic intonational grammar) What are the linguistic functions of gradient modulations in the fundamental frequency contour?
30. STRATEGY: INTONATIONAL PHONOLOGY 11 Challenge: ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation Strategy: ‣ Analyze f0 contours as well-formed sequences of tonal elements (sequences derived from a finite state tonotactic intonational grammar) ‣ Igarashi et al. (2013): Tokyo Japanese IDS only apparently not a wider f0 range; wider f0 range if looking just at boundary tones What are the linguistic functions of gradient modulations in the fundamental frequency contour?
31. 12 TONOTACTIC GRAMMARS: TOKYO JAPANESE (Pierrehumbert and Beckman, 1988; Maekawa et al., 2002; Venditti, 2005; Igarashi et al. 2013)
32. 12 TONOTACTIC GRAMMARS: TOKYO JAPANESE (Pierrehumbert and Beckman, 1988; Maekawa et al., 2002; Venditti, 2005; Igarashi et al. 2013) “Boundary pitch movement” tones: locus of f0 range expansion
33. 13 TONOTACTIC GRAMMARS: ENGLISH Pierrehumbert (1980), Pierrehumbert & Beckman (1986), Pierrehumbert (2000), Beckman et al. (2005), i.a. “Mainstream American English (MAE)”
34. 13 TONOTACTIC GRAMMARS: ENGLISH Pierrehumbert (1980), Pierrehumbert & Beckman (1986), Pierrehumbert (2000), Beckman et al. (2005), i.a. “Mainstream American English (MAE)” If a sequence can be generated via a path through the machine, then it is well-formed.
35. STRATEGY: INTONATIONAL PHONOLOGY 14 Challenge: ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation Strategy: ‣ Analyze f0 contours as well-formed sequences of tonal elements (sequences derived from a finite state tonotactic intonational grammar) ‣ Igarashi et al. (2013): Tokyo Japanese IDS only apparently not a wider f0 range; wider f0 range if looking just at boundary tones What are the linguistic functions of gradient modulations in the fundamental frequency contour?
36. STRATEGY: INTONATIONAL PHONOLOGY 14 Challenge: ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation Strategy: ‣ Analyze f0 contours as well-formed sequences of tonal elements (sequences derived from a finite state tonotactic intonational grammar) ‣ Igarashi et al. (2013): Tokyo Japanese IDS only apparently not a wider f0 range; wider f0 range if looking just at boundary tones What are the linguistic functions of gradient modulations in the fundamental frequency contour? Why finite state grammar?
37. 15 LEXICON AND FINITE/FINITE STATE GRAMMARS ‣ A grammar derives the set of well-formed tonal sequences over a lexicon of tonal elements, i.e., a tonal inventory Common for intonational grammars to be expressed/summarized as a lexicon
38. 16 CHICKASAW INTONATIONAL LEXICON Gordon (2005)
39. 17 BENGALI INTONATIONAL LEXICON Khan (2014)
40. 18 LEXICON AND FINITE/FINITE STATE GRAMMARS ‣ A grammar derives the set of well-formed tonal sequences over a lexicon of tonal elements ‣ A finite grammar is just a list of these well-formed sequences ‣ No generalizations Also common for intonational grammars to be expressed/summarized as a finite grammar (list)
41. 19 Pierrehumbert & Hirschberg (1990) WELL-FORMED ENGLISH TONAL SEQUENCES: LIST
42. 20 WELL-FORMED FRIULIAN TONAL SEQUENCES: LIST Roseano et al. (2015)
43. 21 LEXICON, FINITE, AND FINITE STATE GRAMMARS ‣ A grammar derives the set of well-formed tonal sequences over a lexicon of tonal elements ‣ A finite grammar is just a list of these well-formed sequences ‣ No generalizations ‣ A finite state grammar compresses the list by encoding generalizations from shared prefixes in well-formed sequences
44. BENGALI ACCENTUAL PHRASES 22 100 150 200 250 300 Fundamental frequency (Hz) Time (s) 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 3.2 L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L*L% L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% rumu nepaler ranir malider namgulo mone rakhte pare ni Rumu Nepal’s queen’s the gardeners’ the names remember couldn’t `Rumu couldn’t remember the names of the gardeners of the queen of Nepal.’ Khan (2008)
45. BENGALI ACCENTUAL PHRASES 22 100 150 200 250 300 Fundamental frequency (Hz) Time (s) 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 3.2 L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L*L% L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% rumu nepaler ranir malider namgulo mone rakhte pare ni Rumu Nepal’s queen’s the gardeners’ the names remember couldn’t `Rumu couldn’t remember the names of the gardeners of the queen of Nepal.’ Khan (2008)
46. 23 FINITE GRAMMAR (LIST) FOR BENGALI ‣ L* Ha L* L% ‣ L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% ‣ L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* Ha L* L% Repeated L* Ha chunk is a coincidence, no generalization
47. 24 FINITE STATE GRAMMAR FOR BENGALI* Khan (2008, 2014), Yu, Khan & Sundara (under revision) IP AP Ha PA L* Ha AP La PA H* La L* H* ip L- H- L% H% LH% HL% HLH% *non-focus contexts, typical tunes, no “over- ride” of boundary tones
48. 24 FINITE STATE GRAMMAR FOR BENGALI* Khan (2008, 2014), Yu, Khan & Sundara (under revision) IP AP Ha PA L* Ha AP La PA H* La L* H* ip L- H- L% H% LH% HL% HLH% *non-focus contexts, typical tunes, no “over- ride” of boundary tones
49. 25 CHALLENGES 2/3: EVOLVING HYPOTHESES, GENERALIZABILITY TO LANGUAGE VARIETIES
50. REVISIONS OF MAE TOBI 26 (Jun 2011)
51. GUSSENHOVEN (2004, 2016): REVISION OF MAE TOBI 27
52. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 What are the linguistic functions of gradient modulations in the fundamental frequency contour?
53. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: What are the linguistic functions of gradient modulations in the fundamental frequency contour?
54. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving hypotheses about intonational inventory, tonotactics What are the linguistic functions of gradient modulations in the fundamental frequency contour?
55. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving hypotheses about intonational inventory, tonotactics ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts What are the linguistic functions of gradient modulations in the fundamental frequency contour?
56. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving hypotheses about intonational inventory, tonotactics ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: What are the linguistic functions of gradient modulations in the fundamental frequency contour?
57. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving hypotheses about intonational inventory, tonotactics ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Write down current proposed grammar as finite state grammar and compile as finite state machine What are the linguistic functions of gradient modulations in the fundamental frequency contour?
58. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving hypotheses about intonational inventory, tonotactics ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Write down current proposed grammar as finite state grammar and compile as finite state machine ‣ Check intonational transcriptions in new corpus by parsing them with finite state machine What are the linguistic functions of gradient modulations in the fundamental frequency contour?
59. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 28 Challenges: ‣ evolving hypotheses about intonational inventory, tonotactics ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Write down current proposed grammar as finite state grammar and compile as finite state machine ‣ Check intonational transcriptions in new corpus by parsing them with finite state machine ‣ Diagnose problems → revisions to intonational grammar What are the linguistic functions of gradient modulations in the fundamental frequency contour?
60. DEFINING BENGALI GRAMMAR IN xfst (excerpts) 29 xfst: Beesley & Karttunen (2003)
61. DEFINING BENGALI GRAMMAR IN xfst (excerpts) 30
62. DEFINING BENGALI GRAMMAR IN xfst (excerpts) 31
63. 32 IMPLEMENTED FINITE STATE GRAMMARS English Bengali
64. 32 IMPLEMENTED FINITE STATE GRAMMARS English Bengali 1. For each utterance in corpus, parse intonational transcription using finite state machine: accepted or not? 2. Over corpus, count up how many times each arc traversed to estimate arc weights (probability that an arc is traversed)
65. PARSING WITH BENGALI FINITE STATE MACHINE 33 ‣ Failed to accept 1.5% of exemplars ‣ (21/1367 total; 9/549 in non-IDS, 12/818 in IDS) ‣ Characteristics of the rejected exemplars were the same across speech styles ‣ New `stacked’ boundary tone fHaL% ‣ Unexpected sequences of pitch accents ‣ Distribution of weak accents (*), e.g., ‣ Characteristics of unaccepted tonal sequences provide direction for revisions to grammar (evolving hypotheses)
66. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 What are the linguistic functions of gradient modulations in the fundamental frequency contour?
67. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: What are the linguistic functions of gradient modulations in the fundamental frequency contour?
68. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts What are the linguistic functions of gradient modulations in the fundamental frequency contour?
69. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: What are the linguistic functions of gradient modulations in the fundamental frequency contour?
70. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Check if all IDS transcriptions are accepted by finite state machine What are the linguistic functions of gradient modulations in the fundamental frequency contour?
71. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Check if all IDS transcriptions are accepted by finite state machine ‣ Estimate arc weights on finite state machine using: (1) non-IDS corpus, (2) IDS corpus. What are the linguistic functions of gradient modulations in the fundamental frequency contour?
72. STRATEGY: PARSE WITH FINITE STATE GRAMMAR 34 Challenges: ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts Strategy: ‣ Check if all IDS transcriptions are accepted by finite state machine ‣ Estimate arc weights on finite state machine using: (1) non-IDS corpus, (2) IDS corpus. ‣ See how probable IDS transcriptions are using non-IDS arc weights and vice versa What are the linguistic functions of gradient modulations in the fundamental frequency contour?
73. HOW PROBABLE ARE IDS SEQUENCES? 35 non−ids ids 0 1 2 3 4 5 Probability ratio Distribution for each speech style IDS tonal sequences markedly more probable under IDS-tuned FSG than non-IDS-tuned FSG
74. HOW PROBABLE ARE IDS SEQUENCES? 35 non−ids ids 0 1 2 3 4 5 Probability ratio Distribution for each speech style Probability ratio = ratio of the probability assigned by the IDS-tuned probabilistic FSG to the probability assigned by non-IDS-tuned probabilistic FSG IDS tonal sequences markedly more probable under IDS-tuned FSG than non-IDS-tuned FSG
75. HOW PROBABLE ARE IDS SEQUENCES? 35 non−ids ids 0 1 2 3 4 5 Probability ratio Distribution for each speech style IDS tonal sequences markedly more probable under IDS-tuned FSG than non-IDS-tuned FSG
76. 36 CHALLENGE 4: CONTEXTUAL DEPENDENCE OF TONES
77. BAG OF TONES MODEL 37 L% LH% HLH% H% HL% 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.00 0.05 0.10 0.15 0.000 0.025 0.050 0.075 0.100 0.125 0.00 0.02 0.04 0.06 non−IDS IDS non−IDS IDS non−IDS IDS non−IDS IDS non−IDS IDS Style Mean relative frequency Intonational phrase L% LH% HLH% H% HL% Yu, Khan & Sundara (2014)
78. BAG OF TONES MODEL 37 L% LH% HLH% H% HL% 0.0 0.1 0.2 0.3 0.4 0.0 0.1 0.2 0.3 0.00 0.05 0.10 0.15 0.000 0.025 0.050 0.075 0.100 0.125 0.00 0.02 0.04 0.06 non−IDS IDS non−IDS IDS non−IDS IDS non−IDS IDS non−IDS IDS Style Mean relative frequency Intonational phrase L% LH% HLH% H% HL% Yu, Khan & Sundara (2014) Ex: Count up how many times there was an HLH% in non-IDS vs. IDS
79. 38 FIRST-ORDER MODEL Khan (2008, 2014), Yu, Khan & Sundara (under revision) IP AP Ha PA L* Ha AP La PA H* La L* H* ip L- H- L% H% LH% HL% HLH%
80. 38 FIRST-ORDER MODEL Khan (2008, 2014), Yu, Khan & Sundara (under revision) IP AP Ha PA L* Ha AP La PA H* La L* H* ip L- H- L% H% LH% HL% HLH% Computed probabilities of traversing an arc conditioned on probabilities of reaching that state emitting arc
81. STRATEGY: INTONATIONAL PHONOLOGY 39 What are the linguistic functions of gradient modulations in the fundamental frequency contour?
82. STRATEGY: INTONATIONAL PHONOLOGY 39 Challenges: ‣ the contextual dependence of individual tonal elements on one another Strategy: ‣ Compute probability of an individual tonal element conditioned on probability of paths that arrive at that state What are the linguistic functions of gradient modulations in the fundamental frequency contour?
83. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 What are the linguistic functions of gradient modulations in the fundamental frequency contour?
84. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation What are the linguistic functions of gradient modulations in the fundamental frequency contour?
85. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars What are the linguistic functions of gradient modulations in the fundamental frequency contour?
86. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts What are the linguistic functions of gradient modulations in the fundamental frequency contour?
87. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts ‣ the contextual dependence of individual tonal elements on one another What are the linguistic functions of gradient modulations in the fundamental frequency contour?
88. STRATEGY: IMPLEMENT FINITE STATE GRAMMARS 40 ‣ the entanglement of extra-linguistic and linguistic factors in conditioning f0 variation ‣ evolving hypotheses about proposed intonational grammars ‣ the generalizability of proposed grammars to a wider range of speech styles and contexts ‣ the contextual dependence of individual tonal elements on one another What are the linguistic functions of gradient modulations in the fundamental frequency contour? Proposed overall strategy: To implement finite state intonational grammars
89. (2ND ORDER MODEL, IF MORE DATA) 41 (Dainora 2006)
90. FSG FOR ENGLISH INTONATION 42 Gussenhoven (2004, 2016)
91. COMPARING FSGS FOR FRENCH AND ENGLISH 43