Pythonで作って学ぶ形態素解析

PyCon JP 2015での発表資料です。

1. Python Ͱ࡞ֶͬͯͿܗଶૉղੳ 2015/10/11, PyConJP 2015 Tomoko Uchida
2. Who am I ଧాஐࢠ @moco_beta ݩɿWebαʔϏεاۀͰPythonΤϯδχΞΛ͍ͯ͠·ͨ͠ ݱࡏɿݕࡧΤϯδϯ Solr, Elasticsearch ಋೖࢧԉɺӡ༻αϙʔτΛ͍ͯ͠·͢ ʢגʣϩϯ΢Ποτॴଐ ීஈ͸ओʹJavaͱ͖Ͳ͖Scala԰͞ΜɻPython ͷ͓࢓ࣄ͖ͨΒ͍͍ͳ :-) ڵຯ͕͋Δ͜ͱɿ৘ใݕࡧͱɺͦͷؔ࿈ٕज़ ࣗવݴޠॲཧɾػցֶशɹ୤ϫφϏʔΊͯ͟͠ษڧத 🐥 2
3. ໨࣍ Janome ঺հ ܗଶૉղੳʁ Janome Λࢧ͑ΔΞϧΰϦζϜ Janome ͷ։ൃ;Γ͔͑Γ ͖͔͚ͬɾ࣮૷ͷ࿩ɾνϡʔχϯάɾ2ܥ/3ܥରԠ 3
4. Janome ͱ͸ http://mocobeta.github.io/janome/ “janome (ऄͷ໨) ͸, Pure Python Ͱॻ͔Εͨ, ࣙॻ಺แͷܗଶૉղੳثͰ͢.” (venv) $ pip install janome (venv) $ python >>> from janome.tokenizer import Tokenizer >>> t = Tokenizer() >>> for token in t.tokenize("͢΋΋΋΋΋΋΋΋ͷ͏ͪ"): … print(token) … ͢΋΋ ໊ࢺ,Ұൠ,*,*,*,*,͢΋΋,εϞϞ,εϞϞ ΋ ॿࢺ,܎ॿࢺ,*,*,*,*,΋,Ϟ,Ϟ ΋΋ ໊ࢺ,Ұൠ,*,*,*,*,΋΋,ϞϞ,ϞϞ ΋ ॿࢺ,܎ॿࢺ,*,*,*,*,΋,Ϟ,Ϟ ΋΋ ໊ࢺ,Ұൠ,*,*,*,*,΋΋,ϞϞ,ϞϞ ͷ ॿࢺ,࿈ମԽ,*,*,*,*,ͷ,ϊ,ϊ ͏ͪ ໊ࢺ,ඇཱࣗ,෭ࢺՄೳ,*,*,*,͏ͪ,΢ν,΢ν 4
5. ಛ௃ͱ͔ ࣙॻɺݴޠϞσϧ͸ mecab-ipadic-2.7.0-20070801 Λ࢖༻ ͍͍ͩͨ͸ MeCab ͱಉ͡ղੳ݁ՌʹͳΓ·͢ɻະ஌ޠॲཧͰࠩ ҟ͕Ͱ·͢ Pure Python ͔ͭඪ४ϥΠϒϥϦͷΈ࢖༻ ؀ڥ໰ΘͣͲ͜Ͱ΋ಈ͘…͸ͣ Ϣʔβʔࣙॻαϙʔτ (MeCab ࣙॻϑΥʔϚοτ) ୯ޠ௥Ճ͕ࢼͤ·͢ 5
6. ܗଶૉղੳʁ ೔ຊޠͷ෼͔ͪॻ͖ʢ୯ޠ෼ׂʣʁ ੯͍͠ɺ͚ͲͦΕ͚ͩ͡Όͳ͍ (͓໿ଋ) Wikipedia ʮܗଶૉղੳʯ͔Β “ର৅ݴޠͷจ๏ͷ஌ࣝʢจ๏ͷϧʔϧͷू·Γʣ΍ࣙॻʢ඼ࢺ ౳ͷ৘ใ෇͖ͷ୯ޠϦετʣΛ৘ใݯͱͯ͠༻͍ɺࣗવݴޠͰॻ ͔ΕͨจΛܗଶૉʢMorpheme, ͓͓·͔ʹ͍͑͹ɺݴޠͰҙຯ Λ࣋ͭ࠷খ୯Ґʣͷྻʹ෼ׂ͠ɺͦΕͧΕͷ඼ࢺΛ൑ผ͢Δ࡞ ۀΛࢦ͢ɻ” 6
7. ܗଶૉʁʁ඼ࢺʁʁ ఆٛ͸ʁ ͦ͏͍͏Ϟϊ͕͋Δͱ͍͏͜ͱͰ͓Ͷ͕͍͠·͢ m(__)m ؾʹͳΔํ͸ઐ໳ॻ΁ ࣗવݴޠॲཧֶձʮݴޠॲཧֶࣙయʯ ίϩφࣾʮࣗવݴޠॲཧͷجૅʯ 7
8. Janome Λࢧ͑ΔΞϧΰϦζϜ ʮ͢΋΋΋΋΋΋΋΋ͷ͏ͪʯ ϓϩάϥϜͰͲ͏΍ͬͯ෼ׂ͠·͔͢ʁ ʢΑ͘஌͍ͬͯΔํ͸͝ΊΜͳ͍͞…ʣ 8
9. ͢΋΋ / ΋ / ΋΋ / ΋ / ΋΋ / ͷ / ͏ͪ Janome Λࢧ͑ΔΞϧΰϦζϜ ໊ࢺ ໊ࢺ ໊ࢺ ॿࢺ ॿࢺ ॿࢺ ໊ࢺ ඞཁͳ஌ࣝ • ޠኮɿʮ͢΋΋ʯʮ΋΋ʯͱ͍͏໊ࢺɺʮ΋ʯʮͷʯͱ͍͏ॿࢺ • ʢࣙॻʣ • ೔ຊޠΒ͠͞ɿ໊ࢺͷ͋ͱʹ͸ॿࢺ͕͖΍͍͢ • ʢݴޠϞσϧʣ
10. Janome Λࢧ͑ΔΞϧΰϦζϜ ࣙॻͱݴޠϞσϧ ਓखͰ੔උ ίʔύεʢֶशσʔλʣ͔Βڭࢣ͋Γػցֶश Ұൠʹ͸͜ͷϋΠϒϦου ͳ͓ૉਓʹ͸ແཧɾɾɾͱ·Ͱ͸͍Θͳ͍͚ͲϞσϧ͔Β࡞Δͷ͸େม Janome ͸طଘͷࣙॻ mecab-ipadic Λआ༻͍ͯ͠·͢ 10
11. Janome Ͱ࣮૷ͨ͠ͷ͸ɺ͍ΘΏΔղੳΤϯδϯ෦෼ ࣙॻҾ͖ ղੳ࣮ߦɿ΋ͬͱ΋ʮ೔ຊޠΒ͍͠ʯ෼ׂϙΠϯτΛ͞ ͕͢ Janome Λࢧ͑ΔΞϧΰϦζϜ 11
12. ࣙॻҾ͖ ࣙॻ͸ɺαΠζ͕ίϯύΫτͰߴ଎ʹҾ͚Δ͜ͱ͕ඞཁ ϋογϡϚοϓ (Python ͷ dict) Ͱ΋͍͍͚Ͳɺ ίϞϯϓϨϑΟοΫεϚονΛ࢖͏ͱࣙॻҾ͖ͷճ਺͕ݮΔͷͰޮ཰త ͘͞Β ࣙॻ ͞ɹಈࢺʮ͢Δʯͷ׆༻ܗ ͘͞ɹಈࢺʮ͘͞ʯͷجຊܗ ͘͞ɹܗ༰ࢺʮ͍͘͞ʯͷ׆༻ܗ ͘͞Βɹ໊ࢺ … ଞ 11 ୯ޠ ೖྗ ग़ྗ
13. ࣙॻҾ͖ σʔλߏ଄ɾΞϧΰϦζϜ ύτϦγΞ໦ (JUMAN) μϒϧ഑ྻ (ChaSen, MeCab) FST (Kuromoji/Lucene൛, Janome) http://taku910.github.io/mecab/ http://www.slideshare.net/lucenerevolution/automaton-invasionlucenerevolution2012 13
14. FST ? Finite State Transducers (ܾఆੑ༗ݶΦʔτϚτϯͷҰछ) ਖ਼֬ʹ͸ Minimal Acyclic Subsequential Transducers ೖྗͷϓϨϑΟοΫεɾαϑΟοΫε྆ํΛڞ༗͢Δͷ ͰɺτϥΠ໦ͱൺֱͯ͠ίϯύΫτ (࿦จ) http://citeseerx.ist.psu.edu/viewdoc/summary? doi=10.1.1.24.3698 14
15. FST TL; DR 15
16. ਤղFST খ͞ͳࣙॻΤϯτϦ { ‘apr’: ’30’, ‘aug’: ’31’, ‘dec’: ’31’, 
 ‘feb’: [’28’, ’29’], ‘jan,’: ’31’, ‘jul’: ’31’, ‘jun’: ’30’ } ͱ͍͏, ̓ͭͷ ೖྗ => ग़ྗ ͷϖΞ͔ΒFSTΛ࡞Γ·͢ ೖྗ͸ܗଶૉͷจࣈྻʢද૚ܗʣɺग़ྗ͸Ϛον͢Δܗଶૉͷ IDͱࢥ͍ͬͯͩ͘͞ 16
17. ‘apr’ => 30 Λ౤ೖ 17
18. ‘aug’ => 31 Λ౤ೖ 18
19. ‘dec’ => 31 Λ౤ೖ 19
20. ʢུʣ ଓ͖͸webͰ http://mocobeta-backup.tumblr.com/post/111076688132/lucene-fst-1 20
21. Ͱ͖ͨʋ(´ʔʆ)ϊ 21
22. FST ༨ஊ1ɿJava ੡ݕࡧΤϯδϯ Apache Lucene 4ܥͰ୯ޠࣙॻ΍ ΫΤϦΤϯδϯͱͯ͠࠾༻͞Ε·ͨ͠ ༨ஊ2ɿJanome ͸ Lucene ͷ࣮૷ΛҰ෦ࢀߟʹ͍ͯ͠·͢ 22
23. ղੳ ʮ೔ຊޠΒ͠͞ʯΛܭࢉ͢Δ 23
24. ղੳ ʮ͢΋΋΋΋΋΋΋΋ͷ͏ͪʯʢ;ͨͨͼʣͷ۠੾Γํ͸ͨ͘͞Μ͋Δ 1.͢΋΋ / ΋ / ΋΋ / ΋ / ΋΋ / ͷ / ͏ͪ 2.͢΋΋ / ΋ / ΋΋ / ΋΋ / ΋ / ͷ / ͏ͪ 3.͢΋΋ / ΋΋ / ΋ / ΋΋ / ΋ / ͷ / ͏ͪ ͳͲͳͲ… ࣙॻΛҾ͖ͭͭɺ·ͣ͸͢΂ͯͷ෼ׂͷީิΛ਺্͑͛Δ ໊ࢺ ໊ࢺ ໊ࢺ ॿࢺ ॿࢺ ॿࢺ ໊ࢺ ໊ࢺ ॿࢺ ໊ࢺ ໊ࢺ ॿࢺ ॿࢺ ໊ࢺ ໊ࢺ ໊ࢺ ໊ࢺ ໊ࢺ ॿࢺ ॿࢺ ॿࢺ 24
25. ղੳ ީิͷͳ͔͔ΒɺҰ൪೔ຊޠΒ͍͠ύλʔϯΛબͿ 1. ͢΋΋ / ΋ / ΋΋ / ΋ / ΋΋ / ͷ / ͏ͪ 2. ͢΋΋ / ΋ / ΋΋ / ΋΋ / ΋ / ͷ / ͏ͪ ʮ2. ΑΓ1.ͷ΄͏͕ࣗવʯΛܭࢉͰٻΊΔʹ͸ʁ ໊ࢺ ໊ࢺ ໊ࢺ ॿࢺ ॿࢺ ॿࢺ ໊ࢺ ໊ࢺ ॿࢺ ໊ࢺ ໊ࢺ ॿࢺ ॿࢺ ໊ࢺ 25
26. ղੳ ݴޠϞσϧʢӅΕϚϧίϑϞσϧʣ ʮ͋Δ୯ޠͷग़ݱ͠΍͢͞ʯʢ֬཰ʣ ʮલޙͷ඼ࢺͷͭͳ͕Γ΍͢͞ʯʢ֬཰ʣ ֬཰Λίετ஋ͱಡΈ׵͑Ε͹଍͠ࢉͰʮ෼ׂύλʔϯͷ ೔ຊޠΒ͠͞ʯΛධՁͰ͖Δ 26
27. ղੳ ֬཰Ϟσϧʹ΋ͱ͖ͮɺҰ൪ʮ΋ͬͱ΋Β͍͠ฒͼʯΛٻ ΊΔޮ཰తͳΞϧΰϦζϜ ϏλϏΞϧΰϦζϜ ʢಈతܭը๏ͷҰछʣ ॻ੶/ߨٛࢿྉͳͲղઆଟ਺ ʮࣗવݴޠॲཧͷجૅʯ ɹ͓͢͢Ί
28. Janome ։ൃ;Γ͔͑Γ ͨ·ʹɺͲͷ͘Β͍ͷظؒͰ։ൃͨ͠ͷ͔ฉ͍ͯͩ͘͞Δํ͕͍ΔͷͰɺGithub Ϧϙ δτϦ (https://github.com/mocobeta/janome) ͷίϛοτཤྺΛ௥ͬͯΈͨ 2015/1/20 ࠒ: ४උ (FSTͷ࿦จ, LuceneͷιʔεಡΈ࢝Ίͨ) 2015/2/14 ࠒ: FSTͷϕʔεΛ࡞Γ࢝ΊΔ 2015/3/14 ࠒ: γεςϜࣙॻ (mecab-ipadic) Λ࡞Γ࢝ΊΔ 2015/4/1 ࠒ: ϥςΟεΛ࡞Γ࢝ΊΔ 2015/4/7: Ϣʔβʔࣙॻ͕ͭ͘ 2015/4/8: PyPI ʹొ࿥ɺެ։ ։ൃ޻਺͜Μͳײ͡ FSTͷཧղͱ࣮૷ʹ2ϲ݄ mecab-ipadic ͷ಺แࣙॻԽʹ0.5ϲ݄ ϏλϏͷ࣮૷ʹ਺೔ ϦϦʔε࡞ۀʹ1೔ pythonεΩϧ͸ॳʙதڃ͘Β͍
29. ։ൃͷ͖͔͚ͬ Q: ͱ͜ΖͰͳΜͰ࡞Ζ͏ͱ ࢥͬͨͷ A: ܗଶૉղੳثͷ࣮૷͸ࣗવ ݴޠॲཧͷ 101 Ͱ͢ ^^
 (by @ikawaha) (kuromoji.js ࡞ऀ) (kagome ࡞ऀ) ΋͔ͯ͠͠: ྲྀߦͬͯΔ…?(ҧ)
30. FST & ಺แγεςϜࣙॻͷ࣮૷ ͕͜͜Ͱ͖Ε͹ऴͬͨΑ͏ͳ΋ͷʂ ʢͨͿΜʣ FSTʢΦʔτϚτϯʣ͸ʮลͷू· ΓʯͱΈͯόΠτ഑ྻʹམͱ͜͠Ή (Apache Lucene ํࣜ) string ͸ encode(), decode() Ͱ bytes ʹύοΫ int ͸ struct.pack(), unpack() Ͱ bytes ʹύοΫ http://mocobeta-backup.tumblr.com/post/113693778372/lucene-fst-2 janome ͰͷΤϯίʔυྫ
31. FST & ಺แγεςϜࣙॻͷ࣮૷ mecab-ipadic ͷࣙॻΤϯτϦΛFSTʢόΠτ഑ྻʣʹͯ͠ό ΠφϦσʔλͱͯ͠อଘ ࣙॻΤϯτϦͷৄࡉ৘ใʢ඼ࢺɺ׆༻ܗɺಡΈɺetc.ʣ ͸FSTͱ͸ผϑΝΠϧʹ΋ͭ ࿈઀ίετ(඼ࢺͷ઀ଓίετ)͸ೋ࣍ݩ഑ྻʹͯ͠อଘ ղੳث(Tokenizer)ͷॳظԽ࣌ʹ͜ΕΒΛಡΈࠐΉ 31
32. ࣙॻαΠζ ॳظϦϦʔε࣌͸ 20MB 0.2 ܥͰগ͠খ͘͞ͳͬͯ 13MB TODO: Πϯετʔϧͷͱ͖ʹɺιʔεຒΊࠐΈͷࣙॻΛϓϦίϯύΠϧ ͢Δͱ͜ΖͰϝϞϦ3-4GBඞཁ ͱΓ͍͔͋͑ͣͭ͘ࢼͨ͠΍ΓํͷதͰҰ൪଎͔ͬͨΜͩ… ࣗ෼ͷVPS؀ڥ(2GB)Ͱಈ͔ͳ͍ͷʹؾ͍ͮͨͷͰͳΜͱ͔͠·͢ orz 32
33. ϓϩϑΝΠϧΛͱͬͯ஗͍ͱ͜ΖΛಛఆͯ͠ɺ׳Εͳ͍ͳΓʹࢼߦࡨޡͯ͠ΈΔ ʢ஗͍ͱ͜Ζ̍ʣࣙॻҾ͖ʢFSTͷݕࡧʣ͕࣮ߦ࣌ؒͷ൒෼Λ઎ΊΔ ΩϟογϡΛೖΕͨΓ ʢ͜͜͸FSTͷ࣮૷͕͍·͍͔ͪͩΒ͔ͳʣ ΦϒδΣΫτੜ੒Λ΍Ί͍ͯΖ͍Ζ Tuple ʹͨ͠Γ ʢ஗͍ͱ͜Ζ̎ʣ࿈઀ίετݕࡧ͕࣮ߦ࣌ؒͷ 1/4 Λ઎ΊΔ ࿈઀ίετͷ࣋ͪํΛɺ࠷ॳ dict (ϋογϡςʔϒϧ) ͩͬͨͷΛɺʢMeCabʹͳΒ͍ʣೋ࣍ݩ഑ྻʹ ม͑Δͱߴ଎ʹɻσʔλߏ଄ͱͬͯ΋େࣄ ^^; ʢ஗͍ͱ͜Ζ̏ʣϥςΟεͷϊʔυੜ੒͕࣮ߦ࣌ؒͷ 1/4 Λ઎ΊΔ (͋ͱϝϞϦ΋৯͏) TODO νϡʔχϯά 33
34. 2ܥ/3ܥରԠ ࠷ॳͷϦϦʔε࣌͸3ܥͷΈ ࢢ৔(?)ͷ൓Ԡɿʮ2ܥ͸…ʯ ʢͰ͢ΑͶʔ (janome ͷ৔߹) string, bytes ͷඇޓ׵͚ͩ৻ॏʹରԠ͢Ε ͹ɺ͋ͱ͸؆୯ͳஔ͖׵͑ͰࡁΜͩ 2ܥରԠ൛ΛͱΓ͋͑ͣผύοέʔδͰެ։ 34
35. 2ܥ/3ܥରԠ ผύοέʔδʹ෼͔Ε͍ͯͨ2ܥ/3ܥͷύοέʔδΛͻͱͭ ʹϚʔδ͢ΔਆPRΛ͍͖ͨͩɺແࣄ౷߹ ͋Γ͕ͱ͏͍͟͝·͢ʂ
36. ͓ΘΓ “͜ΕͳΒࣗ෼΋ΦϨΦϨܗଶૉղੳث࡞Εͦ͏” ͱࢥ͍͚ͬͯͨͩ·ͨ͠Ͱ͠ΐ͏͔ (Τϯδϯ෦෼ͳΒͶ!)
37. MeCab (C++) ͷΫϩʔϯɺ·ͨ͸MeCabࣙॻ(Ϟσϧ)Λआ༻͠ ͍ͯΔܗଶૉղੳث͋Ε͜Ε Igo (Java), igo-python (Python), igo-ruby (Ruby),
 igo-javascript (JavaScript) Kuromoji (Java), kuromoji.js (JavaScript) kagome (Go), janome (Python) ଞ... Appendixɿྺ࢙ʹ͍ͭͯগ͚ͩ͠ 37
38. Appendixɿྺ࢙ʹ͍ͭͯগ͚ͩ͠ ʢגʣδϟετγεςϜ͞·͔Β͓आΓ͠·ͨ͠ JastTechTalk #02 ܗଶૉղੳͷ࣮૷ݴޠTalk http://www.slideshare.net/JSUXDesign/jtt4 38
39. Appendixɿྺ࢙ʹ͍ͭͯগ͚ͩ͠ ΑΓৄ͘͠͸ͪ͜ΒΛɿ https://web.archive.org/web/20130123204941/https:// www.ipa.go.jp/event/ipaforum2007/program/pdf/oss- matsumoto.pdf 39
40. JustTechTalk ! ೔ຊޠͷݴޠॲཧͱ͍͑͹ɺͷδϟετγεςϜ͕ࣾ։࠵ ͍ͯ͠Δࣗવݴޠॲཧ΍ػցֶशͳͲͳͲͷษڧձ ݴޠॲཧɾػցֶशʹڵຯ͕͋Δํ͸άϧʔϓʹೖΖ ͏ ʋ(*´∇ʆ)ů DoorKeeperɿhttps://justsystems.doorkeeper.jp twitterɿ@just_techtalk 40
41. ͋Γ͕ͱ͏͍͟͝·ͨ͠