Share
Embed
Transcript
36 slides
-
ࣗݾհ • ໊લ: ฏদ३ (@himkt) • ॴଐ: ஜେֶେֶӃ म࢜̍ •
ଔۀݚڀͰܥྻϥϕϦϯάΛ͔ͭͬͨ => ໘ന͍ʂʂ -
͖ΐ͏ͳ͢͜ͱ • ܥྻϥϕϦϯάͱ • ؆୯ͳઆ໌ • χϡʔϥϧҎલͷܥྻϥϕϦϯά • ͍͔ͭ͘ͷϞσϧͷൺֱ •
χϡʔϥϧܥྻϥϕϦϯά • ݸਓతʹ໘ന͍ͱࢥͬͨϞσϧΛத৺ʹ -
ܥྻϥϕϦϯά • ͳΜΒ͔ͷܥྻσʔλʹରͯ͠ϥϕϧΛ༧ଌ͢Δ • => ؍ଌܥྻX͕༩͑ΒΕͨͱ͖ʹɹɹɹɹ Λ࠷େԽ͢Δ y ؍ଌ͞Ε͍ͯΔܥྻ
༧ଌ͍ͨ͠ϥϕϧྻ xt xt 1 xt+1 xt+2 xt 2 yt 2 yt yt 1 yt+1 yt+2 … … … … P(y1:n | x1:n) -
ࣗવݴޠॲཧʹ͓͚ΔܥྻϥϕϦϯά • ࢺλάਪఆ: ΈΜͳ͔ͭͬͯΔ • ݻ༗දݱநग़: ෳ߹ޠͷநग़ͳͲ • νϟϯΩϯά: จઅ୯ҐͰσʔλΛॲཧ
• ҙຯׂղੳ: ߴͳจͷղੳ ܅ ͷ ໊ ࠷ ߴ # * * * 0 0 0 ݻ༗දݱநग़ͷྫʢநग़͞ΕΔݻ༗දݱ: “܅ͷ໊”ʣ -
ܥྻϥϕϦϯά • => ܥྻͷߏΛଊ͑Δ͜ͱ͕Ͱ͖Δ (֬) ϞσϧΛߟ͑Δ • HMM, MEMM, CRF
؍ଌ͞Ε͍ͯΔܥྻ ༧ଌ͍ͨ͠ϥϕϧྻ xt xt 1 xt+1 xt+2 xt 2 yt 2 yt yt 1 yt+1 yt+2 … … … … -
Hidden Markov Model • ࣌ࠁtͷϥϕϧ࣌ࠁt-1ʹͷΈґଘ͢Δ • ࣌ࠁtͷ୯ޠ࣌ࠁtͷϥϕϧʹͷΈґଘ͢Δ ؍ଌ͞Ε͍ͯΔܥྻ ༧ଌ͍ͨ͠ϥϕϧྻ
xt xt 1 xt+1 xt+2 xt 2 yt 2 yt yt 1 yt+1 yt+2 … … … … -
• => ҎԼͷಉ࣌֬Λߟ͑ΕΑ͍͜ͱʹͳΔ • HMMೖྗܥྻͱϥϕϧྻ͕ੜ͞ΕΔ֬Λֶश͢Δ (ੜϞσϧ) ؍ଌ͞Ε͍ͯΔܥྻ ༧ଌ͍ͨ͠ϥϕϧྻ xt
xt 1 xt+1 xt+2 xt 2 yt 2 yt yt 1 yt+1 yt+2 … … … … Hidden Markov Model P(y1:n, x1:n) = P(y1:n)P(x1:n | y1:n) = n Y i=1 P(yi | yi 1)P(xi | yi) -
͕֬࠷େͱͳΔܥྻΛޮతʹൃݟ͢Δ • ܥྻΛn ϥϕϧͷछྨΛkͱ͢Δͱɼɹ ௨ΓͷY • => ϏλϏΞϧΰϦζϜ • ಈతܭը๏ͷҰछ:
Ͱ֬࠷େͷύεΛൃݟ kn ༧ଌ͍ͨ͠ϥϕϧྻ yt 2 yt yt 1 yt+1 yt+2 … … k௨Γ k௨Γ k௨Γ k௨Γ k௨Γ O(k 2 n) -
ϏλϏΞϧΰϦζϜͷงғؾ C C C 1 2 3 ࣌ࠁU͔ΒUʹ͍ͭͯͬͱભҠίετ͕͍ࢬʁ ֤࣌ࠁʹ͍ͭͯɹɹճͷܭࢉશମͰɹɹɹճͷܭࢉ
ϥϕϧk (k=3) ܥྻn (n=3) C C C k2 k2n cost: 1.2 cost: 4.9 cost: 2.2 -
• HMMXͱYͷಉ࣌֬Λֶश͢ΔੜϞσϧ • ೖྗ୯ޠྻϞσϦϯά͢Δඞཁ͋Δ͔ʁ • =>ࣝผϞσϧ͕ཉ͍͠ • HMMૉੑͷࣗ༝Λ্͛ʹ͍͘ • ୯ޠͷදͷใҐஔͷใΛૉੑʹ͍ͨ͠
• => MEMMɾCRFɾߏԽύʔηϓτϩϯ…etc Hidden Markov Model: cons -
Maximum Entropy Markov Model • ࣝผϞσϧͰ͋Δ • ૉੑؔ f ͕ಋೖ͞Εͨ
؍ଌ͞Ε͍ͯΔܥྻ ༧ଌ͍ͨ͠ϥϕϧྻ yt 2 yt yt 1 yt+1 yt+2 … … x1:n P(y1:n | x1:n) = n Y i=1 P(yi | yi 1, x1:n) = n Y i=1 exp wT f(yi, yi 1, x1:n) Z(yi 1, x1:n) -
ૉੑؔ f ʹ͍ͭͯ • ͋Β͔͡ΊఆΊΒΕ͍͔ͨͭ͘ͷϧʔϧʹج͖ͮ i൪ͷϥϕϧΛ༧ଌ͢ΔͨΊͷૉੑ (ϕΫτϧ) Λநग़ • ͋Β͔͡ΊఆΊΒΕͨϧʔϧΛ·ͱΊͯૉੑςϯϓϨʔτͱ͍͏
i൪ͷ୯ޠ͕େจࣈͰ ͡·͍ͬͯΔ͔Ͳ͏͔ (0/1) i൪ͷ୯ޠ͕-ingͰ ͓Θ͍ͬͯΔ͔Ͳ͏͔ (0/1) f(yi, yi 1, x1:n) = 0 B B B B B B B B B @ 1 0 1 . . . 0 1 0 1 C C C C C C C C C A -
• MEMM֤Ґஔʹରͯ͠ہॴతʹਖ਼نԽΛߦ͏ • => Label Bias, Length Biasͷ͕ى͖Δ Maximum
Entropy Markov Model: cons ؍ଌ͞Ε͍ͯΔܥྻ ༧ଌ͍ͨ͠ϥϕϧྻ yt 2 yt yt 1 yt+1 yt+2 … … x1:n P(y1:n | x1:n) = n Y i=1 P(yi | yi 1, x1:n) = n Y i=1 exp wT f(yi, yi 1, x1:n) Z(yi 1, x1:n) -
Label Bias, Length Bias Kudo, Taku, Kaoru Yamamoto, and
Yuji Matsumoto. “Applying Conditional Random Fields to Japanese Morphological Analysis." EMNLP. Vol. 4. 2004. -
• => ܥྻશମΛߟྀͨ֬͠ (Label Biasͷղফ) Conditional Random Field P(y1:n
| x1:n) = 1 Z(x1:n) n Y i=1 (yi | yi 1, x1:n) = 1 Z(x1:n) n Y i=1 exp wT f(yi, yi 1, x1:n) Z(x1:n) = X y0 n Y i=1 exp wT f(y0 i , y0 i 1 , x1:n) େҬతͳਖ਼نԽ ؍ଌ͞Ε͍ͯΔܥྻ ༧ଌ͍ͨ͠ϥϕϧྻ yt 2 yt yt 1 yt+1 yt+2 … … x1:n -
ܥྻϥϕϦϯάͷϞσϧͷมભ ֶश͕؆୯ͳϞσϧ (HMM) ॊೈʹૉੑΛܾΊΔ͜ͱ͕Ͱ͖ΔϞσϧ (MEMM, CRF, ߏԽύʔηϓτϩϯ, ྨثͷஞ࣮࣍ߦ,…) ૉੑͷܾఆɾநग़Λࣗಈతʹߦ͏ χϡʔϥϧωοτϫʔΫ[Collobert+,
2011͔Β?] ͷྲྀߦ ࠓճͷൃදͰRNNΛͬͨϞσϧͨͪΛհ͠·͢ Collobert, Ronan, et al. "Natural language processing (almost) from scratch." Journal of Machine Learning Research 12.Aug (2011): 2493-2537. -
͡ΊͯLSTM + CRFͷϞσϧΛܥྻϥϕϦϯάʹద༻ 3छͷૉੑ (spelling feature + context feature +
word embedding) • spelling feature (12छྨ) ୯ޠ͕େจࣈͰ࢝·͍ͬͯΔ͔Ͳ͏͔ ୯ޠ͕શͯେจࣈ͔Ͳ͏͔ … • context feature: uni-gram, bi-gram, tri-gram • word embedding: SENNA [Collobert+, 2011]͕ॳظ feature connection trick (MEMM (CRF?) likeͳૉੑͷऔΓѻ͍) Huang+, ArXiv2015 (ArXiv only?) Bidirectional LSTM-CRF Models for Sequence Tagging Collobert, Ronan, et al. "Natural language processing (almost) from scratch." Journal of Machine Learning Research 12.Aug (2011): 2493-2537. -
ωοτϫʔΫߏ • word embeddingҎ֎CRFʹೖྗ (feature connection trick) • LSTMओͱ͍͏ΑΓैདྷͷCRFʹLSTMΛ͚Ճ͑ͨϞσϧʁ
ਤจத͔ΒҾ༻ TQFMMJOHGFBUVSF DPOUFYUGFBUVSF XPSEFNCFEEJOH ૉੑநग़ͱ͍͑$//ʂ -
Chiu+, TACL2016 Named Entity Recognition with Bidirectional LSTM-CNNs BiLSTM
+ CNN 2छྨͷ୯ޠಛྔ (word embedding + additional feature) 2छྨͷจࣈಛྔ (char embedding x CNN + additional feature) CNNΛͬͯ୯ޠͷจࣈใϕʔεಛྔΛநग़ -
ωοτϫʔΫߏ • 2छྨͷ୯ޠಛྔ • ຒΊࠐΈ • Additional feature • খจࣈԽ+ޠװ
• େจࣈԽͨ͠୯ޠ • খจࣈԽͨ͠୯ޠ • CNN͕நग़ͨ͠จࣈಛྔ • ຒΊࠐΈ • Additional feature ਤจத͔ΒҾ༻ -
• prefix-suffixͷΑ͏ͳใΛଊ͑Δ͜ͱ͕Ͱ͖Δ • Additional Char Feature: େจࣈɾখจࣈɾه߸ɾͦͷଞ จࣈϕʔεͷಛྔ ਤจத͔ΒҾ༻
ͱͯྑ͍ੑೳ ΑΓਓखͷૉੑʹґଘ͠ͳ͍Ϟσϧ࡞Εͳ͍͔ʁ -
֎෦ͷϥϕϧσʔλΛΘͳ͍BiLSTM-CRF ڭࢣσʔλதͰֶशͨ͠จࣈϕʔεͷ୯ޠಛྔ + ϥϕϧΛ༩͍ͯ͠ͳ͍େنίʔύεͰֶशͨ͠୯ޠຒΊࠐΈ[Ling15] CNNΛͬͯ୯ޠͷจࣈใϕʔεಛྔΛநग़ Lample+, NAACL-HLT2016 Neural
Architectures for Named Entity Recognition Yulia, Wang Ling Lin Chu-Cheng, et al. "Not all contexts are created equal: Better word representations with variable attention.” Proc. EMNLP (2015). -
Ma+, ACL2016 End-to-End Sequence Labeling via Bidirectional LSTM-CNNs-CRF ਤจத͔ΒҾ༻
Pennington Jeffrey, Richard Socher, and Christopher Manning. "Glove: Global vectors for word representation.” Proc. EMNLP. 2014. -45. $// $3'Ϟσϧࣗମ͔ͳΓෳࡶʹͳ͖ͬͯͨ ଞͷλεΫσʔλͱͷΈ߹ΘͤͰ͖Δʁ • ୯ޠಛྔ (GloVe 100d [Pennington2014]) • จࣈಛྔ (จࣈCNNͷΈ) -
ਤจத͔ΒҾ༻ ֶशࡁΈͷݴޠϞσϧ [Chelba+, 2013] + CNN/RNNͰߏͨ͠จࣈϕʔεͷಛྔ + ୯ޠͷࢄදݱ Λૉੑͱͨ͠BiGRU+ CRF
Peters+, ACL2017 Semi-supervised sequence tagging with bidirectional language models Chelba et al. "One billion word benchmark for measuring progress in statistical language modeling." arXiv preprint arXiv:1312.3005 (2013). -
࣍ͷ୯ޠɾલͷ୯ޠΛͯΔΛಉ࣌ʹղ͘ Ճͷϥϕϧ͖σʔλඞཁͳ͠ Rei+, ACL2017 Semi-supervised Multitask Learning for Sequence
Labeling ਤจத͔ΒҾ༻ ˜ E = E + ( ! E + E ) -
จࣈใ͔Β୯ޠΛͯΔݴޠϞσϧΛಉ࣌ʹֶश͢Δ Highway network [Srivastava+, 2015] Λ࠾༻ͯ͠Λ·͍ͨͰใΛޮతʹ͑Δ Ճͷϥϕϧ͖σʔλෆཁ Liu+, AAAI2018
Empower Sequence Labeling with Task-Aware Neural Language Model ਤจத͔ΒҾ༻ Srivastava, Rupesh Kumar, Klaus Greff, and Jürgen Schmidhuber. “Highway networks." arXiv preprint arXiv:1505.00387 (2015). -
• ΈΜͳެ։͞ΕֶͨशࡁΈࢄදݱΛ͍ͬͯΔ • SENNA -> GloVe, Word2Vec (Skip-gram? CBoW?) •
୯ޠͷࢄදݱ50࣍ݩɼ100࣍ݩͱൺֱతΊ͕ଟ͍ (*) ֤ݚڀͰΘΕ͍ͯΔ୯ޠϕΫτϧ Paper Word Embedding 1 collobert2011 SENNA50d 2 Huang2015 SENNA50d 3 Chiu2016 SENNA50d (beat: GloVe, Word2Vec) 4 Lample2016 Skip-n-gram (en: 100d, other: 64d) 5 Ma2016 GloVe100d 6 Paters2017 SENNA50d 7 Rei2017 Word2Vec (Google News 300d, PubMed+PMC 200d) 8 Liu2018 GloVe100d *SENNA50࣍ݩσʔλͷΈެ։͞Ε͍ͯΔ -
·ͱΊ • HMMʹΑΔܥྻϥϕϦϯά • ࣝผϞσϧʹ͍ͨ͠ɼॊೈʹૉੑΛೖΕ͍ͨ • MEMM, CRFͷΑ͏ͳࣝผϞσϧ͕ఏҊ͞Εͨ • ॊೈʹૉੑΛऔΓೖΕΔ͜ͱ͕Մೳʹ
(ૉੑΤϯδχΞϦϯά) • χϡʔϥϧܥྻϥϕϦϯά͕ੜ·Εͨ • ૉੑΛࣗಈతʹநग़͢Δ࣌ • ڭࢣ͋ΓϞσϧɾϚϧνλεΫֶश • ϥϕϧ͚ෆཁͳσʔλΛ׆༻͢Δ • खݩʹ͋ΔڭࢣσʔλͰͬͱ͏·͘Δ -
จͷଞʹεϥΠυ࡞࣌ʹࢀߟʹͨ͠ࢿྉ • ۙͷࣗવݴޠॲཧʹ͓͚ΔܥྻϥϕϦϯάͷಈ͕·ͱ·ͬͨࢿྉ (p.15~) • https://www.slideshare.net/stairlab/2017-73465873 • ྨ->ܥྻϥϕϦϯά ϩδεςΟοΫճؼ->CRFͷྲྀΕͰ·ͱΊΒΕͨࢿྉ •
http://www.ism.ac.jp/editsec/toukei/pdf/64-2-179.pdf • HMM -> MEMM -> CRFͷల։͕આ໌͞Εͨࢿྉ • http://www.cs.stanford.edu/~nmramesh/crf • https://ssli.ee.washington.edu/people/duh/projects/CRFintro.pdf • https://abicky.net/2010/06/21/082851/ -
͓·͚ ެ։͞Ε͍ͯΔ࣮ • Ma+, 2016: https://github.com/XuezheMax/NeuroNLP2 • Lample+, 2016: https://github.com/glample/tagger
• Rei+, 2017: https://github.com/marekrei/sequence-labeler • Liu+, 2018: https://github.com/LiyuanLucasLiu/LM-LSTM-CRF -
͓·͚ ൺֱ͋Ε͜Ε P(xi | c0, , c0,1, . .
. , ci 1, ) P(xi | x1, . . . , xi 1) Liu+ͷݴޠϞσϧ ;ͭ͏ͷݴޠϞσϧ E = T X t=1 log P(yt | dt) E = s(y) + log X ˜ y2 ˜ Y exp s(˜ y) E = s(y) + log X ˜ y2 ˜ Y exp s(˜ y) + ⇣ ( T 1 X t=1 log P(wt+1 | ! mt) + ( T X t=2 log P(wt 1 | mt) ⌘ Liu+, Rei+ͷϩεؔ ͦͷଞͷϞσϧͷϩεؔ Chiu+, ͷϩεؔ ↑ݴޠϞσϧͷ߲