Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Luceneと日本語の検索

Jun Ohtani
December 17, 2014

 Luceneと日本語の検索

2014年12月のJJUG ナイトセミナーで使用したスライドです。

Jun Ohtani

December 17, 2014
Tweet

More Decks by Jun Ohtani

Other Decks in Programming

Transcript

  1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Jun Ohtani @johtani 2014/12/17 - JJUG φΠτɾηϛφʔ Luceneͱ೔ຊޠͷݕࡧ
  2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ΞδΣϯμ • Luceneʁ • సஔΠϯσοΫεɺN-Gramɺܗଶૉղੳ • ೔ຊޠݕࡧʁ • KuromojiͳͲ • Elasticsearch࢖͏ͱʁ
  3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited about • Me, Jun Ohtani / Technical Adovocate lucene-gosenίϛολʔ ElasticSearch Server೔ຊޠ൛ͷ຋༁ elasticsearch-extended-analysisͷ։ൃ http://blog.johtani.info • Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel
 Professional services: Support & development subscriptions Trainings
  4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Luceneͷಛ௃ • ߴ଎ͳݕࡧϥΠϒϥϦ • Javaϕʔε • Apache License 2.0 • ࠷৽൛ɿ4.10.2ʢ΋͏͙͢ɺ4.10.3ϦϦʔεʣ • JDK 7ʢu25͔Βu55ͷؒʹϦϦʔε͞ΕͨJDK͸ό ά͕͋ΔͨΊ࢖༻ෆՄʣ
  5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Luceneͷओͳػೳ • సஔΠϯσοΫεͷ࡞੒ͱݕࡧ • ݕࡧ݁ՌͷείΞϦϯά • ๛෋ͳΫΤϦʢۙ๣ɺϑϨʔζɺൣғͳͲʣ • ϑΟʔϧυࢦఆʹΑΔݕࡧ
  6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ υΩϡϝϯτʹؚ·ΕΔଐੑ
  7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ༻ޠ • ΫΤϦ ݕࡧ৚݅ɺݕࡧࣜ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ
  8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥
  9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ
  10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ
  11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ
  12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited https://github.com/johtani/jjug- example αϯϓϧίʔυ
  20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ
  21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ 
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ
  22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ
  23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ܗଶૉղੳ • ϝϦοτɿ ҙຯͷ͋Δ୯ޠͷ੾Ε໨
 ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ ग़ෆೳɻ ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ
  24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited N-Gram • ϝϦοτɿ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ΠϯσοΫεංେԽ ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
  25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Analyzerͷ֓ཁ Analyzer υΩϡϝϯτ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ... ΠϯσοΫε
  26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Analyzerͷ֓ཁ Analyzer ݕࡧ
 Ωʔϫʔυ ΠϯσοΫε ΠϯσοΫε
 ݕࡧॲཧ ݕࡧ݁Ռ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ...
  27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Analyzer Analyzerͷߏ੒ char_filter tokenizer char_filter token_
 filter token_
 filter Text Tokens
  28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Char Filter • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ • ྫɿhtml_strip ೖྗɿ ग़ྗɿ <title>Elasticsearch is not a service of AWS</title> Elasticsearch is not a service of AWS
  29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Tokenizer • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹ ґଘ • ྫɿstandard ೖྗɿ ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS
  30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Tokenizer • ྫɿkeyword ग़ྗɿ • ྫɿkuromoji_tokenizer ೖྗɿ ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ Elasticsearch is not a service of AWS ण͕࢘ඒຯ͔ͬͨ͠
  31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited TokenFilter • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ • ྫɿlowercase ೖྗɿ ग़ྗɿ • ྫɿstop ೖྗɿ ग़ྗɿ Elasticsearch is not a service of AWS elasticsearch is not a service of aws Elasticsearch is not a service of AWS Elasticsearch service AWS
  32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited TokenFilter • ྫɿkuromoji_baseform ೖྗɿ ग़ྗɿ • ྫɿkuromoji_readingform ೖྗɿ ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ ण࢘ ͕ ඒຯ͍͠ ͨ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ sushi ga oishika ta
  33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited kuromoji • Christian Moen͞ΜʹΑΓLuceneʹدଃ • ࣙॻʹ͸MeCabͷIPADICࣙॻ/ϞσϧΛར༻ • LuceneͷAnalyzerʹؚ·ΕΔ
  34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited lucene-gosen • LGPL v2 • MeCabͷࣙॻσʔλΛར༻Մೳ • NAISTɺIPADicΛબ୒Մೳ • όΠφϦࣙॻͷϏϧυπʔϧ͋Γ • https://github.com/lucene-gosen/lucene-gosen
  35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ͦͷ΄͔ͷओͳػೳ • GEO • ϋΠϥΠτ • ϑΝηοτ • άϧʔϐϯά
  36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Luceneͷόʔδϣϯ • εςʔϒϧɿ4ܥ • ։ൃɿ5ܥ - ࣍ظόʔδϣϯ • Ϛελɿ6ܥ - Java 8ඞਢ
  37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited LuceneͷϞδϡʔϧ • core جຊύοέʔδ • analysis ݴޠ΍υϝΠϯͷ௥ՃAnalysisύοέʔδ • codec σʔλܗࣜʢcodecɺpostings formatʣʹؔ͢Δύοέʔδ • queries ௥ՃͷΫΤϦɺϑΟϧλ
  38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited LuceneͷϞδϡʔϧ • queryparser ΫΤϦύʔαʔ͓ΑͼϑϨʔϜϫʔΫ • classification LuceneͷΠϯσοΫε৘ใΛ༻͍ͨClassificationϞδϡʔϧ • expressions ಈతͳ஋ͷܭࢉΛϓϥΨϒϧʹมߋՄೳʹ͢ΔϞδϡʔϧ • facet ϑΝηοτʹؔ͢ΔϞδϡʔϧ
  39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited LuceneͷϞδϡʔϧ • grouping ݕࡧ݁Ռͷάϧʔϐϯά༻Ϟδϡʔϧ • highlighter ݕࡧ݁ՌͷϋΠϥΠτ༻Ϟδϡʔϧ • join ΠϯσοΫεɺݕࡧ࣌ͷδϣΠϯ༻Ϟδϡʔϧ • spatial • ஍ཧ৘ใݕࡧ༻Ϟδϡʔϧ
  40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited LuceneͷϞδϡʔϧ • suggest ΦʔταδΣετɺεϖϧνΣοΫ༻Ϟδϡʔϧ • test-framework Luceneʹؔ࿈͢ΔΞϓϦέʔγϣϯͷͨΊͷςετϑϨʔϜ ϫʔΫ • demo LuceneͷػೳΛ঺հ͢ΔσϞʢϑΝηοτͱ͔ʣ
  41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϑϦʔϫʔυݕࡧ
  42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ߜΓࠐΈ
  43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϋΠϥΠτ
  44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ιʔτ
  45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϖʔδϯά
  46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ूܭ
  47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited αδΣετ
  48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ
  49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ElasticsearchͰ೔ຊޠ • elasticsearch-analysis-kuromoji
 https://github.com/elasticsearch/elasticsearch- analysis-kuromoji • ΠϯετʔϧʢElasticsearch 1.4.xܥͷ৔߹ʣ bin/plugin -i elasticsearch/elasticsearch-analysis-kuromoji/2.4.1
  50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ࢀߟจݙ - Lucene • ॻ੶ - Lucene in Action 2nd EditionʢLucene 3ʣ
 http://www.manning.com/hatcher3/ • ެࣜαΠτ
 http://lucene.apache.org/core/ • ೔ຊޠͷࢀߟϒϩά
 http://lucene.jugem.jp
 
 http://mocobeta-backup.tumblr.com/tagged/ lucene
  51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ࢀߟจݙ - Elasticsearch • Github https://github.com/elasticsearch • ΨΠυ http://www.elasticsearch.org/guide/ • ॻ੶ ElasticSearchServer೔ຊޠ൛