ZeuScansion: a Tool for Scansion of English Poetry

Slide 1

Slide 1 text

ZeuScansion ZeuScansion: a tool for scansion of English poetry Manex Agirrezabal1, Mans Hulden2, Bertol Arrieta1, Aitzol Astigarraga1 (1) Euskal Herriko Unibertsitatea / University of the Basque Country (UPV/EHU) (2)University of Helsinki July 15th, 2013 11th Conference on Finite-State Methods and Natural Language Processing St. Andrews, Scotland https://zeuscansion.googlecode.com 1/38

Slide 2

Slide 2 text

ZeuScansion First of all, you should know that... 2/38

Slide 3

Slide 3 text

ZeuScansion First of all, you should know that... I’m not a native English speaker. 2/38

Slide 4

Slide 4 text

ZeuScansion First of all, you should know that... I’m not a native English speaker. I’m not an expert in English poetry. 2/38

Slide 5

Slide 5 text

ZeuScansion Outline 1 Scansion–what is it? Metrical patterns 2 Scansion–why do we need it? 3 Related work 4 Scansion–why it’s diﬃcult? 5 ZeuScansion: the output 6 ZeuScansion: technical details Groves’ rules Closest word ﬁnder Global analysis 7 Evaluation 8 Discussion & future directions 3/38

Slide 6

Slide 6 text

ZeuScansion Scansion–what is it? 1 Scansion–what is it? Metrical patterns 2 Scansion–why do we need it? 3 Related work 4 Scansion–why it’s diﬃcult? 5 ZeuScansion: the output 6 ZeuScansion: technical details Groves’ rules Closest word ﬁnder Global analysis 7 Evaluation 8 Discussion & future directions 4/38

Slide 7

Slide 7 text

ZeuScansion Scansion–what is it? 5/38

Slide 8

Slide 8 text

Slide 9

Slide 9 text

ZeuScansion Scansion–what is it? Rhythm Loudness, pitch, duration, syntax, frequency of polysyllabic words... contribute to the rhythm of the language. ...If the scansion of a line meant all the phonetic fact, no two line would scan the same way... C.S.Lewis en.wikipedia.org Meter The elements that inﬂuence the meter are very limited. In the case of English poetry, meter is described as a sequence of feet (a grouping of syllables usually containing one stressed syllable). en.wikipedia.org 5/38

Slide 10

Slide 10 text

ZeuScansion Scansion–what is it? Scansion Scansion is the act of determining and graphically representing the metrical character of a line of verse. en.wikipedia.org 6/38

Slide 11

Slide 11 text

ZeuScansion Scansion–what is it? Metrical patterns ZeuScansion knows the most common metrical patterns in English poetry. 7/38

Slide 12

Slide 12 text

ZeuScansion Scansion–what is it? Metrical patterns ZeuScansion knows the most common metrical patterns in English poetry. Disyllabic feet - - pyrrhus - ’ iamb ’ - trochee ’ ’ spondee Trisyllabic feet - - - tribrach ’ - - dactyl - ’ - amphibrach - - ’ anapest - ’ ’ bacchius ’ ’ - antibacchius ’ - ’ cretic ’ ’ ’ molossus 7/38

Slide 13

Slide 13 text

ZeuScansion Scansion–what is it? Metrical patterns IAMB [to be] [or not] [to be] ... - ’ - ’ _ ’ 8/38

Slide 14

Slide 14 text

Slide 15

Slide 15 text

ZeuScansion Scansion–what is it? Metrical patterns IAMB [to be] [or not] [to be] ... - ’ - ’ _ ’ TROCHEE [Tyger], [Tyger], [burning] bright ’ - ’ - ’ - [In the] [forests] [of the] night ’ - ’ - ’ - DACTYL [ Woman much] [missed, how you] [call to me], [call to me] ... ’ _ _ ’ _ _ ’ _ _ ’ _ _ 8/38

Slide 16

Slide 16 text

ZeuScansion Scansion–why do we need it? 1 Scansion–what is it? Metrical patterns 2 Scansion–why do we need it? 3 Related work 4 Scansion–why it’s diﬃcult? 5 ZeuScansion: the output 6 ZeuScansion: technical details Groves’ rules Closest word ﬁnder Global analysis 7 Evaluation 8 Discussion & future directions 9/38

Slide 17

Slide 17 text

ZeuScansion Scansion–why do we need it? My PhD work 10/38

Slide 18

Slide 18 text

ZeuScansion Scansion–why do we need it? My PhD work IN PROGRESS!! 10/38

Slide 19

Slide 19 text

ZeuScansion Scansion–why do we need it? My PhD work IN PROGRESS!! Poetry generation system 10/38

Slide 20

Slide 20 text

ZeuScansion Scansion–why do we need it? My PhD work IN PROGRESS!! Poetry generation system Poetry analysis system 10/38

Slide 21

Slide 21 text

ZeuScansion Scansion–why do we need it? My PhD work IN PROGRESS!! Poetry generation system Poetry analysis system Rhymes, 10/38

Slide 22

Slide 22 text

ZeuScansion Scansion–why do we need it? My PhD work IN PROGRESS!! Poetry generation system Poetry analysis system Rhymes, vocabulary, 10/38

Slide 23

Slide 23 text

ZeuScansion Scansion–why do we need it? My PhD work IN PROGRESS!! Poetry generation system Poetry analysis system Rhymes, vocabulary, metaphors... 10/38

Slide 24

Slide 24 text

ZeuScansion Scansion–why do we need it? My PhD work IN PROGRESS!! Poetry generation system Poetry analysis system Rhymes, vocabulary, metaphors... and also meter! 10/38

Slide 25

Slide 25 text

ZeuScansion Related work 1 Scansion–what is it? Metrical patterns 2 Scansion–why do we need it? 3 Related work 4 Scansion–why it’s diﬃcult? 5 ZeuScansion: the output 6 ZeuScansion: technical details Groves’ rules Closest word ﬁnder Global analysis 7 Evaluation 8 Discussion & future directions 11/38

Slide 26

Slide 26 text

ZeuScansion Related work Scandroid (Hartman, 2005): Iambic and anapestic feet only. 12/38

Slide 27

Slide 27 text

ZeuScansion Related work Scandroid (Hartman, 2005): Iambic and anapestic feet only. AnalysePoems (Plamondon, 2006): Identiﬁes patterns, does not impose them. Rhymes also checked. 12/38

Slide 28

Slide 28 text

ZeuScansion Related work Scandroid (Hartman, 2005): Iambic and anapestic feet only. AnalysePoems (Plamondon, 2006): Identiﬁes patterns, does not impose them. Rhymes also checked. Calliope (McAleese, 2007): Scandroid improvement using linguistic theories. 12/38

Slide 29

Slide 29 text

Slide 30

Slide 30 text

ZeuScansion Scansion–why it’s difficult? 1 Scansion–what is it? Metrical patterns 2 Scansion–why do we need it? 3 Related work 4 Scansion–why it’s difficult? 5 ZeuScansion: the output 6 ZeuScansion: technical details Groves’ rules Closest word finder Global analysis 7 Evaluation 8 Discussion & future directions 13/38

Slide 31

Slide 31 text

ZeuScansion Scansion–why it’s diﬃcult? Input: Woman much missed how you call to me call to me Output: [’ - - ][ ’ - - ][’ - -][’ - -] 14/38

Slide 32

Slide 32 text

Slide 33

Slide 33 text

ZeuScansion Scansion–why it’s diﬃcult? Input: Woman much missed how you call to me call to me Output: [’ - - ][ ’ - - ][’ - -][’ - -] Problem no1: we can’t just use a dictionary Woman much missed how you call to me call to me ’ - ’ ‘ ’ ’ ’ - ’ ’ - ’ Problem no2: Divide stresses into feet Woman much missed how you call to me call to me [’ - - ][ ’ - - ][’ - -][’ - -] 14/38

Slide 34

Slide 34 text

Slide 35

Slide 35 text

ZeuScansion Scansion–why it’s diﬃcult? Problem no1: Groves’ rules (POS-tag based) implemented using FST Problem no2: Global analysis Problem no3: FST-based closest word ﬁnder 15/38

Slide 36

Slide 36 text

ZeuScansion ZeuScansion: the output 1 Scansion–what is it? Metrical patterns 2 Scansion–why do we need it? 3 Related work 4 Scansion–why it’s diﬃcult? 5 ZeuScansion: the output 6 ZeuScansion: technical details Groves’ rules Closest word ﬁnder Global analysis 7 Evaluation 8 Discussion & future directions 16/38

Slide 37

Slide 37 text

ZeuScansion ZeuScansion: the output I dwell in possibility (Emily Dickinson) I dwell in Possibility A fairer House than Prose More numerous of Windows Superior for Doors Of Chambers as the Cedars Impregnable of Eye And for an Everlasting Roof The Gambrels of the Sky Of Visitors the fairest For Occupation This The spreading wide my narrow Hands To gather Paradise 1 - / - \-/-- 2 - \- / - / 3 / /-- - \- 4 -/-- - \ 6 - \- - - \- 7 -/-- - / 8 - - - \-/- / 9 - \- - - / 11 - \-- - \- 12 - \-/- - 13 - \- / - /\ \ 14 - /- /-\ 17/38

Slide 38

Slide 38 text

ZeuScansion ZeuScansion: technical details 1 Scansion–what is it? Metrical patterns 2 Scansion–why do we need it? 3 Related work 4 Scansion–why it’s diﬃcult? 5 ZeuScansion: the output 6 ZeuScansion: technical details Groves’ rules Closest word ﬁnder Global analysis 7 Evaluation 8 Discussion & future directions 18/38

Slide 39

Slide 39 text

ZeuScansion ZeuScansion: technical details English poetry text T okenizer POS-tagger 1st step 2nd step RHYTHMI-METRICAL SCANSION GROVES' RULES Metrical information Are the words in the dictionary? Y Closest word ﬁnder N Cleanup 19/38

Slide 40

Slide 40 text

ZeuScansion ZeuScansion: technical details Tokenizer and POS-tagger I dwell in possibility 20/38

Slide 41

Slide 41 text

ZeuScansion ZeuScansion: technical details Tokenizer and POS-tagger I dwell in possibility I dwell in possibility ... 20/38

Slide 42

Slide 42 text

ZeuScansion ZeuScansion: technical details Tokenizer and POS-tagger I dwell in possibility I dwell in possibility ... I+PRP dwell+VBP in+IN possibility+NN ... 20/38

Slide 43

Slide 43 text

ZeuScansion ZeuScansion: technical details Pronunciation lexicon 21/38

Slide 44

Slide 44 text

ZeuScansion ZeuScansion: technical details Pronunciation lexicon NETtalk (Sejnowski and Rosenberg, 1987) CMU Pronouncing Dictionary (Weide, 1998) 21/38

Slide 45

Slide 45 text

ZeuScansion ZeuScansion: technical details Pronunciation lexicon NETtalk (Sejnowski and Rosenberg, 1987) CMU Pronouncing Dictionary (Weide, 1998) I+’+PRP dwell+’+VBP in+’+IN possibility+‘ ’ +NN ... 21/38

Slide 46

Slide 46 text

ZeuScansion ZeuScansion: technical details Groves’ rules -Rules of thumb to provide a reasonable stress asignment for scansion -Needs access to POS information 1such as names, verbs, adjectives and adverbs 22/38

Slide 47

Slide 47 text

Slide 48

Slide 48 text

ZeuScansion ZeuScansion: technical details Groves’ rules -Rules of thumb to provide a reasonable stress asignment for scansion -Needs access to POS information 1 Stress the primarily stressed syllable in content words1 2 Stress the secondarily stressed syllable of polysyllabic content words 1such as names, verbs, adjectives and adverbs 22/38

Slide 49

Slide 49 text

ZeuScansion ZeuScansion: technical details Groves’ rules -Rules of thumb to provide a reasonable stress asignment for scansion -Needs access to POS information 1 Stress the primarily stressed syllable in content words1 2 Stress the secondarily stressed syllable of polysyllabic content words and the most strongly stressed syllable of polysyllabic words. 1such as names, verbs, adjectives and adverbs 22/38

Slide 50

Slide 50 text

ZeuScansion ZeuScansion: technical details Groves’ rules TOKENIZE POS-tagger 1st step 2nd Step CleanUp -------------------------------------------------------------------------------------- I I+PRP I+-+PRP I+-+PRP - dwell dwell+VBP dwell+’+VBP dwell+’+VBP ’ in in+IN in+-+IN in+-+IN - possibility possibility+NN possibility+--’--+NN possibility+‘-’--+NN ‘-’-- English poetry text T okenizer POS-tagger 1st step 2nd step RHYTHMI-METRICAL SCANSION GROVES' RULES Metrical information Are the words in the dictionary? Y Closest word ﬁnder N Cleanup 23/38

Slide 51

Slide 51 text

ZeuScansion ZeuScansion: technical details Closest word ﬁnder FST-based system that ﬁnds the closest spelled word in the dictionary. We need this because English is not a phonemic language 24/38

Slide 52

Slide 52 text

ZeuScansion ZeuScansion: technical details Closest word ﬁnder FST-based system that ﬁnds the closest spelled word in the dictionary. We need this because English is not a phonemic language We chumped and chawed the buttered toast Phantasmagoria and other poems, Lewis Carroll 24/38

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

ZeuScansion ZeuScansion: technical details Closest word ﬁnder Devised word change rules: 25/38

Slide 56

Slide 56 text

ZeuScansion ZeuScansion: technical details Closest word ﬁnder Devised word change rules: 1 At the end of the word, higher cost (Word splitter) 25/38

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Slide 59

Slide 59 text

ZeuScansion ZeuScansion: technical details Closest word ﬁnder Devised word change rules: 1 At the end of the word, higher cost (Word splitter) 2 We only allow a maximum of 2 character changes. 3 Character change order: 1 1 vowel 2 1 consonant 3 2 vowels 4 1 vowel and 1 consonant 5 2 consonants 25/38

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

ZeuScansion ZeuScansion: technical details Closest word ﬁnder c h u m p e d | c h a w e d | | | | | | | | | | | | | | - h u m p e d | c h e w e d The similarly pronounced words presented by the system are humped and chewed. we we+PRP we+-+PRP we+-+PRP - chumped chumped+VBD humped+‘+VBD humped+‘+VBD ‘ and and+CC and+-+CC and+-+CC - chawed chawed+VBD chewed+‘+VBD chewed+‘+VBD ‘ the the+DT the+-+DT the+-+DT - buttered buttered+JJ buttered+‘-+JJ buttered+‘-+JJ ‘- toast toast+NN toast+’+NN toast+’+NN ’ 26/38

Slide 63

Slide 63 text

ZeuScansion ZeuScansion: technical details Global analysis The syllable-stresses are marked 27/38

Slide 64

Slide 64 text

ZeuScansion ZeuScansion: technical details Global analysis The syllable-stresses are marked The system tries to identify the predominant meter of the poem, by ﬁnding plausible feet. 27/38

Slide 65

Slide 65 text

ZeuScansion ZeuScansion: technical details Global analysis By the shores of Gitche Gumee, By the shining Big Sea Water, Stood the wigwam of Nokomis, Daughter of the Moon, Nokomis. Dark behind it rose the forest, Rose the black and gloomy pine trees, Rose the firs with cones upon them; Bright before’ it beat the water, Beat the clear and sunny water, Beat the shining Big Sea Water. The song of Hiawatha, Henry Wadsworth Longfellow - - ‘ - ‘ ? - - ‘ - ’ ’ ’ - ’ - ’ ‘ - ‘ - ’ - - - ’ ‘ - ’ - ‘ - ’ - ’ - ’ - ’ - ‘ - ’ ‘ ’ - ‘ - ‘ - ’ - ’ - ’ - ’ - ’ - ’ - ’ - ‘ - ’ - ’ - ‘ - ’ ’ ’ - 28/38

Slide 66

Slide 66 text

ZeuScansion ZeuScansion: technical details Global analysis Predominant stress structure per line: ’_’_’_’_ 29/38

Slide 67

Slide 67 text

Slide 68

Slide 68 text

ZeuScansion ZeuScansion: technical details Global analysis Predominant stress structure per line: ’_’_’_’_ Diﬀerent ways of splitting it up: 4 trochees | 2 amphibrachs | 2 cretics | 3 iambs ----------------------|----------------------|----------------------|--------------------- [’ -][’ -][’ -][’ -] | ’ [- ’ -] ’ [- ’ -] | [’ - ’] - [’ - ’] - | ’ [- ’][- ’][- ’] - SCORE PER FEET: NAME | FEET | SC.|No MATCHES ----------|------|----|----------- trochee | (’-) | 4 |(4 matches) amphibrach| (-’-)| 3 |(2 matches) cretic | (’-’)| 3 |(2 matches) iamb | (-’) | 3 |(3 matches) 29/38

Slide 69

Slide 69 text

ZeuScansion Evaluation 1 Scansion–what is it? Metrical patterns 2 Scansion–why do we need it? 3 Related work 4 Scansion–why it’s diﬃcult? 5 ZeuScansion: the output 6 ZeuScansion: technical details Groves’ rules Closest word ﬁnder Global analysis 7 Evaluation 8 Discussion & future directions 30/38

Slide 70

Slide 70 text

ZeuScansion Evaluation For the evaluation we used the “For Better for Verse” poetry corpus. http://prosody.lib.virginia.edu/ 31/38

Slide 71

Slide 71 text

ZeuScansion Evaluation For Better for Verse corpus: 55 diﬀerent poems 2Some lines have several alternative scansions 32/38

Slide 72

Slide 72 text

ZeuScansion Evaluation For Better for Verse corpus: 55 diﬀerent poems 759 scanned (by experts) poetry lines2 2Some lines have several alternative scansions 32/38

Slide 73

Slide 73 text

ZeuScansion Evaluation For Better for Verse corpus: 55 diﬀerent poems 759 scanned (by experts) poetry lines2 200 of them were correctly scanned by ZeuScansion (26.35 %) 2Some lines have several alternative scansions 32/38

Slide 74

Slide 74 text

ZeuScansion Evaluation For Better for Verse corpus: 55 diﬀerent poems 759 scanned (by experts) poetry lines2 200 of them were correctly scanned by ZeuScansion (26.35 %) 7,076 syllables 2Some lines have several alternative scansions 32/38

Slide 75

Slide 75 text

ZeuScansion Evaluation For Better for Verse corpus: 55 diﬀerent poems 759 scanned (by experts) poetry lines2 200 of them were correctly scanned by ZeuScansion (26.35 %) 7,076 syllables 6,002 of them were correctly scanned by ZeuScansion (84.82 %) 2Some lines have several alternative scansions 32/38

Slide 76

Slide 76 text

ZeuScansion Discussion & future directions 1 Scansion–what is it? Metrical patterns 2 Scansion–why do we need it? 3 Related work 4 Scansion–why it’s diﬃcult? 5 ZeuScansion: the output 6 ZeuScansion: technical details Groves’ rules Closest word ﬁnder Global analysis 7 Evaluation 8 Discussion & future directions 33/38

Slide 77

Slide 77 text

ZeuScansion Discussion & future directions Basic system for scansion of English poetry Phonetically closest word ﬁnder The evaluation results are promising 34/38

Slide 78

Slide 78 text

ZeuScansion Discussion & future directions Basic system for scansion of English poetry Phonetically closest word ﬁnder The evaluation results are promising FUTURE WORK: Do statistical inference about global metric pattern of the poem Improve closest word ﬁnder performance Replace HMM POS-tagger with a deterministic FST-based tagger (e.g. Brill’s tagger) 34/38

Slide 79

Slide 79 text

ZeuScansion Acknowledgments Eskerrik asko! Thanks! 35/38

Slide 80

Slide 80 text

ZeuScansion Acknowledgments Eskerrik asko! Thanks! Herbert Tucker and the Scholars’ Lab, from the University of Virginia. 35/38

Slide 81

Slide 81 text

ZeuScansion Acknowledgments Eskerrik asko! Thanks! Herbert Tucker and the Scholars’ Lab, from the University of Virginia. All the people involved in the “For Better for Verse” project. 35/38