Unsupervised Partial Parsing

Slides from my dissertation defense in computational linguistics

Transcript

1. Unsupervised Partial Parsing Elias Ponvert Department of Linguistics The University of Texas at Austin Dissertation Defense July 27, 2011 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 1 / 62
2. 1 Goals and contributions 2 Unsupervised partial parsing Main results Discussion 3 Cascaded parsing Main results Discussion 4 Concluding remarks Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 2 / 62
3. Research goals Generally: Develop computational models to learn human language Hello! Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 3 / 62
4. Research goals Specifically: Learn to predict constituent structure from raw text the cat saw the red dog run ⇓ Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 3 / 62
5. Why unsupervised parsing? 1 Less reliance on annotated training Hello! 2 Apply to new languages and domains Særær man annær man mæþæn Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 4 / 62
6. Assumptions made in parser learning S NP VP PP P on NP N Sunday Det the A brown N bear V sleeps , , Getting these labels right AS WELL AS the structure of the tree is hard Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 5 / 62
7. Assumptions made in parser learning P on N Sunday Det the A brown N bear V sleeps , , So the task is to identify the structure alone Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 5 / 62
8. Assumptions made in parser learning on Sunday the brown bear sleeps , Learning operates from gold-standard parts-of-speech (POS) rather than raw text P N Det A N V , on Sunday , the brown bear sleeps P N , Det A N V Klein & Manning 2003 CCM Bod 2006a, 2006b Klein & Manning 2005 DMV Successors to DMV: - Smith 2006, Smith & Cohen 2009, Headden et al 2009, Spitkovsky et al 2010ab, &c J. Gao et al 2003, 2004 Seginer 2007 this work Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 5 / 62
9. Unsupervised parsing: desiderata Raw text Standard NLP / extensible Scalable and fast Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 6 / 62
10. Contributions • Unsupervised parsing satisfying these desiderata is possible • Unsupervised partial parsing: predicting local constituents with high accuracy • Cascaded models: building constituent structure bottom up Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 7 / 62
11. Outline 1 Goals and contributions 2 Unsupervised partial parsing Main results Discussion 3 Cascaded parsing Main results Discussion 4 Concluding remarks Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 8 / 62
12. A new approach: start from the bottom Unsupervised Partial Parsing = segmentation of (non-overlapping) multiword constituents Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 9 / 62
13. Unsupervised segmentation of constituents leaves some room for interpretation Possible segmentations • ( the cat ) in ( the hat ) knows ( a lot ) about that • ( the cat ) ( in the hat ) knows ( a lot ) ( about that ) • ( the cat in the hat ) knows ( a lot about that ) • ( the cat in the hat ) ( knows a lot about that ) • ( the cat in the hat ) ( knows a lot ) ( about that ) Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 10 / 62
14. Defining UPP by evaluation 1. Constituent chunks: non-hierarchical multiword constituents S NP D The N Cat PP P in NP D the N hat VP V knows NP D a N lot PP P about NP N that Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 11 / 62
15. Defining UPP by evaluation 2. Base NPs: non-recursive noun phrases S NP D The N Cat PP P in NP D the N hat VP V knows NP D a N lot PP P about NP N that Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 11 / 62
16. Multilingual data for direct evaluation English WSJ German Negra Chinese CTB Sentences Types Tokens WSJ Penn Treebank 49K 44K 1M Negra Negra German Corpus 21K 49K 300K CTB Penn Chinese Treebank 19K 37K 430K Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 12 / 62
17. Constituent chunks and NPs in the data WSJ Chunks 203K NPs 172K Chunks ∩ NPs 161K Negra Chunks 59K NPs 33K Chunks ∩ NPs 23K CTB Chunks 92K NPs 56K Chunks ∩ NPs 43K Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 13 / 62
18. The benchmark: CCL parser the cat saw the red dog run the 0 ￿￿ cat 0 ￿￿ 1 ￿￿ saw 0 ￿￿ ￿￿ ￿￿ 0 ￿￿ the 0 ￿￿ red 0 ￿￿ 0 ￿￿ dog 0 ￿￿ run 0 ￿￿ Common Cover Links representation Constituency tree Seginer (2007 ACL; 2007 PhD UvA) Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 14 / 62
19. Hypothesis Segmentation can be learned by generalizing on phrasal boundaries Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 15 / 62
20. UPP as a tagging problem B the I cat O in B the I hat the cat in the hat B Beginning of a constituent I Inside a constituent O Not inside a constituent Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 16 / 62
21. Learning from boundaries B the I cat O in B the I hat the cat in the hat STOP # STOP # Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 17 / 62
22. Unsupervised learning tag model for UPP B the I cat O in the I hat STOP # STOP # O B I O B I O B O Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 18 / 62
23. Unsupervised learning tag model for UPP B the I cat O in the I hat STOP # STOP # O B I O B I O B O Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 18 / 62
24. Unsupervised learning tag model for UPP B the I cat O in the I hat STOP # STOP # O B I O B I O B O Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 18 / 62
25. Unsupervised learning tag model for UPP B the I cat O in the I hat STOP # STOP # O B I O B I O B O Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 18 / 62
26. Unsupervised learning tag model for UPP B the I cat O in the I hat STOP # STOP # O B I O B I O B O Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 18 / 62
27. Unsupervised learning tag model for UPP B the I cat O in the I hat STOP # STOP # O B I O B I O B O Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 18 / 62
28. Decoding the tag model for UPP B the I cat in the I hat STOP # STOP # O B Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 19 / 62
29. Decoding the tag model for UPP B the I cat in the I hat STOP # STOP # O B Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 19 / 62
30. Learning from punctuation B on I sunday B the I brown I bear STOP # STOP # on sunday , the brown bear sleeps STOP , O sleeps Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 20 / 62
31. UPP: Models P( ) ≈ P( ) P( | ) B the I cat O in B the I hat Hidden Markov Model B I the B the B I Probabilistic right linear grammar P( ) = P( ) P( | ) the B I B I B I O B I the cat in the hat B I the Learning: expectation maximization (EM) via forward-backward (run to convergence) Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 21 / 62
32. UPP: Models P( ) ≈ P( ) P( | ) B the I cat O in B the I hat Hidden Markov Model B I the B the B I Probabilistic right linear grammar P( ) = P( ) P( | ) the B I B I B I O B I the cat in the hat B I the Decoding: Viterbi Smoothing: additive smoothing on emissions Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 21 / 62
33. UPP: Constraints on sequences B the I cat O in B the I hat the cat in the hat STOP # STOP # STOP B O I Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 22 / 62
34. UPP evaluation: Setup • Evaluation by comparison to treebank data • Standard train / development / test splits • Precision and recall on matched constituents • Benchmark: CCL • Both get tokenization, punctuation, sentence boundaries Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 23 / 62
35. UPP evaluation: Chunking (F-score) 0 10 20 30 40 50 60 70 80 CTB Negra WSJ CCL∗ HMM Chunker PRLG Chunker CCL non-hierarchical constituents First-level parsing output Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 24 / 62
36. UPP evaluation: Base NPs (F-score) 0 10 20 30 40 50 60 70 80 CTB Negra WSJ CCL∗ HMM Chunker PRLG Chunker CCL non-hierarchical constituents First-level parsing output Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 25 / 62
37. PRLG example output (the seeds) already are in (the script) (little chance) that (shane longman) is going to recoup today it would have (severe implications) for (farmers ’ policy) holders (thames ’s u.s. marketing agent) (donald taffner) is preparing to do just that and all (the while) (the bonds) are in (the baby ’s diaper) (mr. rustin) is (senior correspondent) in (the journal ’s london bureau) Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 26 / 62
38. UPP: Review • Sequence models can generalize on indicators for phrasal boundaries • Leads to improved unsupervised segmentation • Learn to predict NPs with high accuracy • (English and German especially) Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 27 / 62
39. Outline 1 Goals and contributions 2 Unsupervised partial parsing Main results Discussion 3 Cascaded parsing Main results Discussion 4 Concluding remarks Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 28 / 62
40. Question How do UPP models capture noun phrase structure? Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 29 / 62
41. What UPP models learn B 100 · P(w|B) the 21.0 a 8.7 to 6.5 ’s 2.8 in 1.9 mr. 1.8 its 1.6 of 1.4 an 1.4 and 1.4 I 100 · P(w|I) % 1.8 million 1.6 be 1.3 company 0.9 year 0.8 market 0.7 billion 0.6 share 0.5 new 0.5 than 0.5 O 100 · P(w|O) of 5.8 and 4.0 in 3.7 that 2.2 to 2.1 for 2.0 is 2.0 it 1.7 said 1.7 on 1.5 HMM Emissions: WSJ Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 30 / 62
42. What UPP models learn B 100 · P(w|B) der the 13.0 die the 12.2 den the 4.4 und and 3.3 im in 3.2 das the 2.9 des the 2.7 dem the 2.4 eine a 2.1 ein a 2.0 I 100 · P(w|I) uhr o’clock 0.8 juni June 0.6 jahren years 0.4 prozent percent 0.4 mark currency 0.3 stadt city 0.3 000 0.3 millionen millions 0.3 jahre year 0.3 frankfurter Frankfurt 0.3 O 100 · P(w|O) in in 3.4 und and 2.7 mit with 1.7 f¨ ur for 1.6 auf on 1.5 zu to 1.4 von of 1.3 sich oneself 1.3 ist is 1.3 nicht not 1.2 HMM Emissions: Negra Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 30 / 62
43. What UPP models learn B 100 · P(w|B) 的 de, of 14.3 一 one 3.1 和 and 1.1 两 two 0.9 这 this 0.8 有 have 0.8 经济 economy 0.7 各 each 0.7 全 all 0.7 不 no 0.6 I 100 · P(w|I) 的 de 3.9 了 (perf. asp.) 2.2 个 ge (measure) 1.5 年 year 1.3 说 say 1.0 中 middle 0.9 上 on, above 0.9 人 person 0.7 大 big 0.7 国 country 0.6 O 100 · P(w|O) 在 at, in 3.4 是 is 2.4 中国 China 1.4 也 also 1.2 不 no 1.2 对 pair 1.1 和 and 1.0 的 de 1.0 将 fut. tns. 1.0 有 have 1.0 HMM Emissions: CTB Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 30 / 62
44. Question What about the PRLG, why does it do so much better than the HMM? Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 31 / 62
45. Question P( ) ≈ P( ) P( | ) B the I cat O in B the I hat Hidden Markov Model B I the B the B I Probabilistic right linear grammar P( ) = P( ) P( | ) the B I B I B I O B I the cat in the hat B I the Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 31 / 62
46. What’s wrong with this picture? B 100 · P(w|B) the 21.0 a 8.7 to 6.5 ’s 2.8 in 1.9 mr. 1.8 its 1.6 of 1.4 an 1.4 and 1.4 I 100 · P(w|I) % 1.8 million 1.6 be 1.3 company 0.9 year 0.8 market 0.7 billion 0.6 share 0.5 new 0.5 than 0.5 O 100 · P(w|O) of 5.8 and 4.0 in 3.7 that 2.2 to 2.1 for 2.0 is 2.0 it 1.7 said 1.7 on 1.5 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 32 / 62
47. What’s wrong with this picture? B 100 · P(w|B) the 21.0 a 8.7 to 6.5 ’s 2.8 in 1.9 mr. 1.8 its 1.6 of 1.4 an 1.4 and 1.4 I 100 · P(w|I) % 1.8 million 1.6 be 1.3 company 0.9 year 0.8 market 0.7 billion 0.6 share 0.5 new 0.5 than 0.5 O 100 · P(w|O) of 5.8 and 4.0 in 3.7 that 2.2 to 2.1 for 2.0 is 2.0 it 1.7 said 1.7 on 1.5 • ’s occurs (immediately) before several terms that appear after B Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 32 / 62
48. PRLG rule probabilities B 100 · P(B → w q) B → the I 28.2 B → a I 11.7 B → mr. I 2.4 B → its I 2.2 B → an I 1.9 B → his I 1.0 B → this I 1.0 B → their I 1.0 B → some I 0.7 B → new I 0.6 I 100 · P(I → w q) I → ’s I 2.6 I → and I 1.3 I → % O 1.1 I → million O 0.6 I → new I 0.5 I → million STOP 0.5 I → company O 0.5 I → year O 0.4 I → & I 0.4 I → million I 0.4 O 100 · P(O → w q) O → of B 3.8 O → to O 3.6 O → in B 2.5 O → and O 1.7 O → to B 1.7 O → of O 1.6 O → in O 1.5 O → and B 1.4 O → for B 1.3 O → it O 1.3 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 33 / 62
49. PRLG rule probabilities B 100 · P(B → w q) B → the I 28.2 B → a I 11.7 B → mr. I 2.4 B → its I 2.2 B → an I 1.9 B → his I 1.0 B → this I 1.0 B → their I 1.0 B → some I 0.7 B → new I 0.6 I 100 · P(I → w q) I → ’s I 2.6 I → and I 1.3 I → % O 1.1 I → million O 0.6 I → new I 0.5 I → million STOP 0.5 I → company O 0.5 I → year O 0.4 I → & I 0.4 I → million I 0.4 O 100 · P(O → w q) O → of B 3.8 O → to O 3.6 O → in B 2.5 O → and O 1.7 O → to B 1.7 O → of O 1.6 O → in O 1.5 O → and B 1.4 O → for B 1.3 O → it O 1.3 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 33 / 62
50. PRLG rule probabilities B 100 · P(B → w q) B → the I 28.2 B → a I 11.7 B → mr. I 2.4 B → its I 2.2 B → an I 1.9 B → his I 1.0 B → this I 1.0 B → their I 1.0 B → some I 0.7 B → new I 0.6 I 100 · P(I → w q) I → ’s I 2.6 I → and I 1.3 I → % O 1.1 I → million O 0.6 I → new I 0.5 I → million STOP 0.5 I → company O 0.5 I → year O 0.4 I → & I 0.4 I → million I 0.4 O 100 · P(O → w q) O → of B 3.8 O → to O 3.6 O → in B 2.5 O → and O 1.7 O → to B 1.7 O → of O 1.6 O → in O 1.5 O → and B 1.4 O → for B 1.3 O → it O 1.3 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 33 / 62
51. PRLG rule probabilities B 100 · P(B → w q) B → the I 28.2 B → a I 11.7 B → mr. I 2.4 B → its I 2.2 B → an I 1.9 B → his I 1.0 B → this I 1.0 B → their I 1.0 B → some I 0.7 B → new I 0.6 I 100 · P(I → w q) I → ’s I 2.6 I → and I 1.3 I → % O 1.1 I → million O 0.6 I → new I 0.5 I → million STOP 0.5 I → company O 0.5 I → year O 0.4 I → & I 0.4 I → million I 0.4 O 100 · P(O → w q) O → of B 3.8 O → to O 3.6 O → in B 2.5 O → and O 1.7 O → to B 1.7 O → of O 1.6 O → in O 1.5 O → and B 1.4 O → for B 1.3 O → it O 1.3 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 33 / 62
52. PRLG rule probabilities B 100 · P(B → w q) B → the I 28.2 B → a I 11.7 B → mr. I 2.4 B → its I 2.2 B → an I 1.9 B → his I 1.0 B → this I 1.0 B → their I 1.0 B → some I 0.7 B → new I 0.6 I 100 · P(I → w q) I → ’s I 2.6 I → and I 1.3 I → % O 1.1 I → million O 0.6 I → new I 0.5 I → million STOP 0.5 I → company O 0.5 I → year O 0.4 I → & I 0.4 I → million I 0.4 O 100 · P(O → w q) O → of B 3.8 O → to O 3.6 O → in B 2.5 O → and O 1.7 O → to B 1.7 O → of O 1.6 O → in O 1.5 O → and B 1.4 O → for B 1.3 O → it O 1.3 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 33 / 62
53. Learning curves: Base NPs 10 20 30 40K 20 40 60 80 sentences 10 20 30 40K 20 60 100 20 40 60 80 F-score EM iter sentences 1 0 20 40 60 80 100 20 40 60 80 EM iter PRLG chunking model: WSJ Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 34 / 62
54. Learning curves: Base NPs 5 10 15K 10 20 30 40 50 sentences 5 10 15K 20 80 140 20 40 F-score EM iter sentences 1 0 50 100 150 10 20 30 40 50 EM iter PRLG chunking model: Negra Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 34 / 62
55. Learning curves: Base NPs 5 10 15K 0 10 20 30 sentences 5 10 15K 20 60 100 10 20 30 F-score EM iter sentences 1 0 20 40 60 80 100 0 10 20 30 EM iter PRLG chunking model: CTB Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 34 / 62
56. Question How much can these models learn? Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 35 / 62
57. Against a supervised benchmark ∼4500 10K 20K 30K 40K 20 40 60 80 WSJ Sentences Base NPs F-score Supervised PRLG Unsupervised PRLG Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 36 / 62
58. Against a supervised benchmark ∼2200 5K 10K 15K 10 20 30 40 50 Negra Sentences Base NPs F-score Supervised PRLG Unsupervised PRLG Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 36 / 62
59. Against a supervised benchmark 5 10 15K 10 20 30 40 50 CTB Sentences Base NPs F-score Supervised PRLG Unsupervised PRLG Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 36 / 62
60. Negra/CTB training much smaller than WSJ 10K 20K 30K 40K 20 40 60 80 Negra PRLG WSJ PRLG CTB PRLG Sentences Base NPs F-score Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 37 / 62
61. Treebank precision S NP D The N Cat PP P in NP D the N hat VP V knows NP D a N lot PP P about NP N that (the cat in the hat) knows (a lot) (about that) • Constituent chunks: Prec = 2/3, Rec = 2/3, F = 2/3 • Base NPs: Prec = 1/3, Rec = 1/2 • Treebank precision: 3/3 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 38 / 62
62. On chunking the CTB 3 20 40 60 80 10 30 50 EM Iterations Treebank precision Base NPs F-score Constituent chunk F-score Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 39 / 62
63. Question. Do these models scale? Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 40 / 62
64. Chunking with training from Gigaword NYT +160K +320K +480K +640K 50 60 70 80 90 +NYT Sentences Treebank precision Base NPs F Const. chunks F Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 41 / 62
65. Chunking with training from Gigaword NYT WSJ +160K +320K +480K +640K 50 60 70 80 90 +NYT Sentences Treebank precision Base NPs F Const. chunks F Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 41 / 62
66. Outline 1 Goals and contributions 2 Unsupervised partial parsing Main results Discussion 3 Cascaded parsing Main results Discussion 4 Concluding remarks Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 42 / 62
67. Question Are we limited to segmentation? Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 43 / 62
68. Hypothesis Identification of higher level constituents can also be learned by generalizing on phrasal boundaries Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 44 / 62
69. Cascaded UPP: 1 Segment raw text there is no asbestos in our products now there is no asbestos in our products now Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 45 / 62
70. Cascaded UPP: 2 Choose stand-ins for phrases our products is no asbestos there is no asbestos in our products now there in now is our Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 45 / 62
71. Cascaded UPP: 3 Segment text + phrasal stand-ins there in now is our there in now is our Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 45 / 62
72. Cascaded UPP: 4 Choose stand-ins and repeat steps 3–4 our products in is no asbestos there there in now is our is in now Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 45 / 62
73. Cascaded UPP: 5 Unwind to output tree our products in is no asbestos there is in now there is no asbestos in our products now Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 45 / 62
74. Cascaded UPP: Review • Separate models learned at each cascade level • Models share hyper-parameters (smoothing etc) • Choice of pseudowords as phrasal stand-ins • Pseudoword-identification: corpus frequency • Cascade run to convergence Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 46 / 62
75. Right-branching baseline the quick brown fox jumped over the lazy dog the quick brown fox jumped over the lazy dog Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 47 / 62
76. Right-branching baseline a Lorillard spokeswoman said , this is an old story a Lorillard spokeswoman said this is an old story Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 47 / 62
77. Cascaded UPP: Evaluation 0 10 20 30 40 50 CTB Negra WSJ Constituents F-score Baseline CCL Cascaded HMM Cascaded PRLG Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 48 / 62
78. Another benchmark: CCM Constituent-context model (Klein & Manning, 2002) • Generative probabilistic model • Gold-standard POS • Short sentences Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 49 / 62
79. Evaluation on ≤10 word setences 0 10 20 30 40 50 60 70 CTB Negra WSJ Constituents F-score Baseline CCM CCL Cascaded HMM Cascaded PRLG Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 50 / 62
80. Example parses two share a house almost devoid of furniture Gold standard two share a house almost devoid offurniture Cascaded PRLG – WSJ correct incorrect Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 51 / 62
81. Example parses what is one to think of all this Gold standard what is one to think of all this Cascaded PRLG – WSJ correct incorrect Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 51 / 62
82. Example parses die the csu CSU tut does das this in in bayern Bavaria doch nevertheless auch also sehr very erfolgreich successfully Nevertheless, the CSU does this in Bavaria very successfully as well Gold standard die csu tut das in bayern doch auch sehr erfolgreich Cascaded PRLG – Negra correct incorrect Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 52 / 62
83. Example parses bei with den the windsors Windsors bleibt stays alles everything in in der the familie family With the Windsors everything stays in the family. Gold standard bei den windsors bleibt alles in der familie Cascaded PRLG – Negra correct incorrect Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 52 / 62
84. Example parses immer ever mehr more anlagenteile machine parts ¨ uberaltern over-age (with) more and more machine parts over-age Cascaded PRLG – Negra correct incorrect Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 52 / 62
85. Outline 1 Goals and contributions 2 Unsupervised partial parsing Main results Discussion 3 Cascaded parsing Main results Discussion 4 Concluding remarks Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 53 / 62
86. Question How do these cascaded chunkers work? Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 54 / 62
87. Recall of NPs and PPs NPs PPs Lev 1 Lev 2 Lev 1 Lev 2 WSJ PRLG 77.5 78.3 9.1 77.6 Negra HMM 54.7 62.3 24.8 48.1 CTB PRLG 30.9 33.6 31.6 47.1 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 55 / 62
88. Prec / Rec trade-offs in the cascade 1 2 3 4 5 6 7 20 40 60 80 Levels Precision Recall F-score 1 WSJ PRLG Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 56 / 62
89. Prec / Rec trade-offs in the cascade 1 2 3 4 5 6 7 30 40 50 Levels Precision Recall F-score 1 Negra PRLG Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 56 / 62
90. Prec / Rec trade-offs in the cascade 1 2 3 4 5 6 7 20 30 40 50 Levels Precision Recall F-score 1 CTB PRLG Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 56 / 62
91. Learning curves 10K 20K 30K 40K 35 40 45 50 WSJ Sentences F-score PRLG CCL HMM Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 57 / 62
92. Learning curves 5K 10K 15K 25 30 35 40 Negra Sentences F-score PRLG HMM CCL Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 57 / 62
93. Learning curves 5K 10K 15K 20 30 40 CTB Sentences F-score PRLG HMM CCL Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 57 / 62
94. Outline 1 Goals and contributions 2 Unsupervised partial parsing Main results Discussion 3 Cascaded parsing Main results Discussion 4 Concluding remarks Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 58 / 62
95. What we’ve learned • Unsupervised identification of base NPs and local constituents is possible • A cascade of chunking models for raw text parsing has state-of-the-art results Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 59 / 62
96. Future directions • Improvements to the sequence models • Better phrasal stand-in (pseudoword) construction • Learning joint models rather than a cascade Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 60 / 62
97. Historical note First known computational natural language parser Transformations and Discourse Analysis Project Zellig Harris & colleagues, UPenn 1950s - 1960s Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 61 / 62
98. Historical note To the best of our knowledge, this is the first application of FSTs to parsing. The program consisted of the following phases: 1. Dictionary look-up. 2. Replacement of some ‘grammatical idioms’ by a single part of speech. 3. Rule based part of speech disambiguation. 4. A right to left FST composed with a left to right FST for computing ‘simple noun phrases’. Joshi & Hopely 1997 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 61 / 62
99. Historical note To the best of our knowledge, this is the first application of FSTs to parsing. The program consisted of the following phases: 4. A left to right FST for computing ‘simple adjuncts’ such as prepositional phrases and adverbial phrases. 5. A left to right FST for computing simple verb clusters. 6. A left to right ‘FST’ for computing clauses. Joshi & Hopely 1997 Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 61 / 62
100. Thanks! Elias Ponvert (UT Austin) Unsupervised Partial Parsing Dissertation Defense 62 / 62