Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Luceneと日本語の検索

657aeeff3fc467567dacebf8a1ea0b23?s=47 Jun Ohtani
December 17, 2014

 Luceneと日本語の検索

2014年12月のJJUG ナイトセミナーで使用したスライドです。

657aeeff3fc467567dacebf8a1ea0b23?s=128

Jun Ohtani

December 17, 2014
Tweet

More Decks by Jun Ohtani

Other Decks in Programming

Transcript

  1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Jun Ohtani @johtani 2014/12/17 - JJUG φΠτɾηϛφʔ Luceneͱ೔ຊޠͷݕࡧ
  2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ΞδΣϯμ • Luceneʁ • సஔΠϯσοΫεɺN-Gramɺܗଶૉղੳ • ೔ຊޠݕࡧʁ • KuromojiͳͲ • Elasticsearch࢖͏ͱʁ
  3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited about • Me, Jun Ohtani / Technical Adovocate lucene-gosenίϛολʔ ElasticSearch Server೔ຊޠ൛ͷ຋༁ elasticsearch-extended-analysisͷ։ൃ http://blog.johtani.info • Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel
 Professional services: Support & development subscriptions Trainings
  4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Luceneͱ͸ʁ
  5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Luceneͷಛ௃ • ߴ଎ͳݕࡧϥΠϒϥϦ • Javaϕʔε • Apache License 2.0 • ࠷৽൛ɿ4.10.2ʢ΋͏͙͢ɺ4.10.3ϦϦʔεʣ • JDK 7ʢu25͔Βu55ͷؒʹϦϦʔε͞ΕͨJDK͸ό ά͕͋ΔͨΊ࢖༻ෆՄʣ
  6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Luceneͷओͳػೳ • సஔΠϯσοΫεͷ࡞੒ͱݕࡧ • ݕࡧ݁ՌͷείΞϦϯά • ๛෋ͳΫΤϦʢۙ๣ɺϑϨʔζɺൣғͳͲʣ • ϑΟʔϧυࢦఆʹΑΔݕࡧ
  7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited సஔΠϯσοΫεʁ
  8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ υΩϡϝϯτʹؚ·ΕΔଐੑ
  9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ༻ޠ • ΫΤϦ ݕࡧ৚݅ɺݕࡧࣜ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ
  10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥
  11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ
  12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ
  13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ
  14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited https://github.com/johtani/jjug- example αϯϓϧίʔυ
  22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ୯ޠͷநग़
  23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ
  24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ 
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ
  25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ
  26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ܗଶૉղੳ • ϝϦοτɿ ҙຯͷ͋Δ୯ޠͷ੾Ε໨
 ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ ग़ෆೳɻ ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ
  27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited N-Gram • ϝϦοτɿ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ΠϯσοΫεංେԽ ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
  28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Analyzerͷ֓ཁ Analyzer υΩϡϝϯτ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ... ΠϯσοΫε
  29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Analyzerͷ֓ཁ Analyzer ݕࡧ
 Ωʔϫʔυ ΠϯσοΫε ΠϯσοΫε
 ݕࡧॲཧ ݕࡧ݁Ռ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ...
  30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Analyzer Analyzerͷߏ੒ char_filter tokenizer char_filter token_
 filter token_
 filter Text Tokens
  31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Char Filter • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ • ྫɿhtml_strip ೖྗɿ ग़ྗɿ <title>Elasticsearch is not a service of AWS</title> Elasticsearch is not a service of AWS
  32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Tokenizer • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹ ґଘ • ྫɿstandard ೖྗɿ ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS
  33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Tokenizer • ྫɿkeyword ग़ྗɿ • ྫɿkuromoji_tokenizer ೖྗɿ ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ Elasticsearch is not a service of AWS ण͕࢘ඒຯ͔ͬͨ͠
  34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited TokenFilter • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ • ྫɿlowercase ೖྗɿ ग़ྗɿ • ྫɿstop ೖྗɿ ग़ྗɿ Elasticsearch is not a service of AWS elasticsearch is not a service of aws Elasticsearch is not a service of AWS Elasticsearch service AWS
  35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited TokenFilter • ྫɿkuromoji_baseform ೖྗɿ ग़ྗɿ • ྫɿkuromoji_readingform ೖྗɿ ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ ण࢘ ͕ ඒຯ͍͠ ͨ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ sushi ga oishika ta
  36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Analysis - ೔ຊޠͩͱʁ
  37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited kuromoji • Christian Moen͞ΜʹΑΓLuceneʹدଃ • ࣙॻʹ͸MeCabͷIPADICࣙॻ/ϞσϧΛར༻ • LuceneͷAnalyzerʹؚ·ΕΔ
  38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited lucene-gosen • LGPL v2 • MeCabͷࣙॻσʔλΛར༻Մೳ • NAISTɺIPADicΛબ୒Մೳ • όΠφϦࣙॻͷϏϧυπʔϧ͋Γ • https://github.com/lucene-gosen/lucene-gosen
  39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ͦͷ΄͔ͷLuceneͷػೳ
  40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ͦͷ΄͔ͷओͳػೳ • GEO • ϋΠϥΠτ • ϑΝηοτ • άϧʔϐϯά
  41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Luceneͷόʔδϣϯ • εςʔϒϧɿ4ܥ • ։ൃɿ5ܥ - ࣍ظόʔδϣϯ • Ϛελɿ6ܥ - Java 8ඞਢ
  42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited LuceneͷϞδϡʔϧ • core جຊύοέʔδ • analysis ݴޠ΍υϝΠϯͷ௥ՃAnalysisύοέʔδ • codec σʔλܗࣜʢcodecɺpostings formatʣʹؔ͢Δύοέʔδ • queries ௥ՃͷΫΤϦɺϑΟϧλ
  43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited LuceneͷϞδϡʔϧ • queryparser ΫΤϦύʔαʔ͓ΑͼϑϨʔϜϫʔΫ • classification LuceneͷΠϯσοΫε৘ใΛ༻͍ͨClassificationϞδϡʔϧ • expressions ಈతͳ஋ͷܭࢉΛϓϥΨϒϧʹมߋՄೳʹ͢ΔϞδϡʔϧ • facet ϑΝηοτʹؔ͢ΔϞδϡʔϧ
  44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited LuceneͷϞδϡʔϧ • grouping ݕࡧ݁Ռͷάϧʔϐϯά༻Ϟδϡʔϧ • highlighter ݕࡧ݁ՌͷϋΠϥΠτ༻Ϟδϡʔϧ • join ΠϯσοΫεɺݕࡧ࣌ͷδϣΠϯ༻Ϟδϡʔϧ • spatial • ஍ཧ৘ใݕࡧ༻Ϟδϡʔϧ
  45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited LuceneͷϞδϡʔϧ • suggest ΦʔταδΣετɺεϖϧνΣοΫ༻Ϟδϡʔϧ • test-framework Luceneʹؔ࿈͢ΔΞϓϦέʔγϣϯͷͨΊͷςετϑϨʔϜ ϫʔΫ • demo LuceneͷػೳΛ঺հ͢ΔσϞʢϑΝηοτͱ͔ʣ
  46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Elasticsearchͱ͸ʁ
  47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϑϦʔϫʔυݕࡧ
  48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ߜΓࠐΈ
  49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϋΠϥΠτ
  50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ιʔτ
  51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϖʔδϯά
  52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ूܭ
  53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited αδΣετ
  54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ
  55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ElasticsearchͰ೔ຊޠ • elasticsearch-analysis-kuromoji
 https://github.com/elasticsearch/elasticsearch- analysis-kuromoji • ΠϯετʔϧʢElasticsearch 1.4.xܥͷ৔߹ʣ bin/plugin -i elasticsearch/elasticsearch-analysis-kuromoji/2.4.1
  56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ࢀߟจݙ - Lucene • ॻ੶ - Lucene in Action 2nd EditionʢLucene 3ʣ
 http://www.manning.com/hatcher3/ • ެࣜαΠτ
 http://lucene.apache.org/core/ • ೔ຊޠͷࢀߟϒϩά
 http://lucene.jugem.jp
 
 http://mocobeta-backup.tumblr.com/tagged/ lucene
  57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission

    is strictly prohibited ࢀߟจݙ - Elasticsearch • Github https://github.com/elasticsearch • ΨΠυ http://www.elasticsearch.org/guide/ • ॻ੶ ElasticSearchServer೔ຊޠ൛