$30 off During Our Annual Pro Sale. View Details »

Luceneと日本語の検索

Jun Ohtani
December 17, 2014

 Luceneと日本語の検索

2014年12月のJJUG ナイトセミナーで使用したスライドです。

Jun Ohtani

December 17, 2014
Tweet

More Decks by Jun Ohtani

Other Decks in Programming

Transcript

  1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Jun Ohtani
    @johtani
    2014/12/17 - JJUG φΠτɾηϛφʔ
    Luceneͱ೔ຊޠͷݕࡧ

    View Slide

  2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ΞδΣϯμ
    • Luceneʁ
    • సஔΠϯσοΫεɺN-Gramɺܗଶૉղੳ
    • ೔ຊޠݕࡧʁ
    • KuromojiͳͲ
    • Elasticsearch࢖͏ͱʁ

    View Slide

  3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    about
    • Me, Jun Ohtani / Technical Adovocate
    lucene-gosenίϛολʔ
    ElasticSearch Server೔ຊޠ൛ͷ຋༁
    elasticsearch-extended-analysisͷ։ൃ
    http://blog.johtani.info
    • Elasticsearch, founded in 2012
    Products: Elasticsearch, Logstash, Kibana, Marvel

    Professional services: Support & development subscriptions
    Trainings

    View Slide

  4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Luceneͱ͸ʁ

    View Slide

  5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Luceneͷಛ௃
    • ߴ଎ͳݕࡧϥΠϒϥϦ
    • Javaϕʔε
    • Apache License 2.0
    • ࠷৽൛ɿ4.10.2ʢ΋͏͙͢ɺ4.10.3ϦϦʔεʣ
    • JDK 7ʢu25͔Βu55ͷؒʹϦϦʔε͞ΕͨJDK͸ό
    ά͕͋ΔͨΊ࢖༻ෆՄʣ

    View Slide

  6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Luceneͷओͳػೳ
    • సஔΠϯσοΫεͷ࡞੒ͱݕࡧ
    • ݕࡧ݁ՌͷείΞϦϯά
    • ๛෋ͳΫΤϦʢۙ๣ɺϑϨʔζɺൣғͳͲʣ
    • ϑΟʔϧυࢦఆʹΑΔݕࡧ

    View Slide

  7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    సஔΠϯσοΫεʁ

    View Slide

  8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ༻ޠ
    • ΠϯσοΫε
    ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ
    • υΩϡϝϯτʢจॻʣ
    ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ
    • ϑΟʔϧυ
    υΩϡϝϯτʹؚ·ΕΔଐੑ

    View Slide

  9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ༻ޠ
    • ΫΤϦ
    ݕࡧ৚݅ɺݕࡧࣜ
    • λʔϜʢTermʣɺτʔΫϯʢTokenʣ
    ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ
    จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ
    ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ

    View Slide

  10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    υΩϡϝϯτͷొ࿥

    View Slide

  11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ


    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ

    View Slide

  12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ


    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ
    ୯ޠ͔Βidͷ഑ྻ͕
    Ҿ͚ΔΑ͏ʹ

    View Slide

  13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ݕࡧ৚݅ೖྗ
    ΧπΦɹαβΤ

    View Slide

  14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    https://github.com/johtani/jjug-
    example
    αϯϓϧίʔυ

    View Slide

  22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ୯ޠͷநग़

    View Slide

  23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ୯ޠͷ۠੾Γํ
    • ӳޠͷ৔߹
    I speak about Lucene in JJUG night seminar.
    • ೔ຊޠͷ৔߹
    ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ

    View Slide

  24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ୯ޠͷ۠੾Γํ
    • ӳޠͷ৔߹
    I speak about Lucene in JJUG night seminar.
    εϖʔε͕੾Ε໨ͱΘ͔Δ
    • ೔ຊޠͷ৔߹
    ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ

    Ͳ͜Ͱ۠੾Ε͹Α͍ʁ

    View Slide

  25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    N-Gramͱܗଶૉղੳ
    • సஔΠϯσοΫεͷΩʔͷ࡞Γํ
    ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ
    Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒
    • N-Gram
    NจࣈͣͭจষΛ۠੾Δ
    • ܗଶૉղੳ
    ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ

    View Slide

  26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ܗଶૉղੳ
    • ϝϦοτɿ
    ҙຯͷ͋Δ୯ޠͷ੾Ε໨

    ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ
    • σϝϦοτɿ
    ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ
    ग़ෆೳɻ
    ΧπΦ͸αβΤͷఋ
    ΧπΦ ͸ ͷ
    αβΤ ఋ

    View Slide

  27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    N-Gram
    • ϝϦοτɿ
    ະ஌ޠʹରԠՄೳ
    • σϝϦοτɿ
    ΠϯσοΫεංେԽ
    ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ
    ΧπΦ͸αβΤͷఋ
    Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

    View Slide

  28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analyzerͷ֓ཁ
    Analyzer
    υΩϡϝϯτ
    Term Id
    ΧπΦ 1
    αβΤ 1ɺ2
    ϫΧϝ 2
    … ...
    ΠϯσοΫε

    View Slide

  29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analyzerͷ֓ཁ
    Analyzer
    ݕࡧ

    Ωʔϫʔυ
    ΠϯσοΫε
    ΠϯσοΫε

    ݕࡧॲཧ
    ݕࡧ݁Ռ
    Term Id
    ΧπΦ 1
    αβΤ 1ɺ2
    ϫΧϝ 2
    … ...

    View Slide

  30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analyzer
    Analyzerͷߏ੒
    char_filter tokenizer
    char_filter
    token_

    filter
    token_

    filter
    Text Tokens

    View Slide

  31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Char Filter
    • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ
    • ྫɿhtml_strip
    ೖྗɿ
    ग़ྗɿ
    Elasticsearch is not a service of AWS
    Elasticsearch is not a service of AWS

    View Slide

  32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Tokenizer
    • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ
    • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹ
    ґଘ
    • ྫɿstandard
    ೖྗɿ
    ग़ྗɿ Elasticsearch is not a service of AWS
    Elasticsearch is not a service of AWS

    View Slide

  33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Tokenizer
    • ྫɿkeyword
    ग़ྗɿ
    • ྫɿkuromoji_tokenizer
    ೖྗɿ
    ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    Elasticsearch is not a service of AWS
    ण͕࢘ඒຯ͔ͬͨ͠

    View Slide

  34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    TokenFilter
    • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ
    • ྫɿlowercase
    ೖྗɿ
    ग़ྗɿ
    • ྫɿstop
    ೖྗɿ
    ग़ྗɿ
    Elasticsearch is not a service of AWS
    elasticsearch is not a service of aws
    Elasticsearch is not a service of AWS
    Elasticsearch service AWS

    View Slide

  35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    TokenFilter
    • ྫɿkuromoji_baseform
    ೖྗɿ
    ग़ྗɿ
    • ྫɿkuromoji_readingform
    ೖྗɿ
    ग़ྗɿ
    ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    ण࢘ ͕ ඒຯ͍͠ ͨ
    ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    sushi ga oishika ta

    View Slide

  36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysis - ೔ຊޠͩͱʁ

    View Slide

  37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    kuromoji
    • Christian Moen͞ΜʹΑΓLuceneʹدଃ
    • ࣙॻʹ͸MeCabͷIPADICࣙॻ/ϞσϧΛར༻
    • LuceneͷAnalyzerʹؚ·ΕΔ

    View Slide

  38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    lucene-gosen
    • LGPL v2
    • MeCabͷࣙॻσʔλΛར༻Մೳ
    • NAISTɺIPADicΛબ୒Մೳ
    • όΠφϦࣙॻͷϏϧυπʔϧ͋Γ
    • https://github.com/lucene-gosen/lucene-gosen

    View Slide

  39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ͦͷ΄͔ͷLuceneͷػೳ

    View Slide

  40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ͦͷ΄͔ͷओͳػೳ
    • GEO
    • ϋΠϥΠτ
    • ϑΝηοτ
    • άϧʔϐϯά

    View Slide

  41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Luceneͷόʔδϣϯ
    • εςʔϒϧɿ4ܥ
    • ։ൃɿ5ܥ - ࣍ظόʔδϣϯ
    • Ϛελɿ6ܥ - Java 8ඞਢ

    View Slide

  42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    LuceneͷϞδϡʔϧ
    • core
    جຊύοέʔδ
    • analysis
    ݴޠ΍υϝΠϯͷ௥ՃAnalysisύοέʔδ
    • codec
    σʔλܗࣜʢcodecɺpostings formatʣʹؔ͢Δύοέʔδ
    • queries
    ௥ՃͷΫΤϦɺϑΟϧλ

    View Slide

  43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    LuceneͷϞδϡʔϧ
    • queryparser
    ΫΤϦύʔαʔ͓ΑͼϑϨʔϜϫʔΫ
    • classification
    LuceneͷΠϯσοΫε৘ใΛ༻͍ͨClassificationϞδϡʔϧ
    • expressions
    ಈతͳ஋ͷܭࢉΛϓϥΨϒϧʹมߋՄೳʹ͢ΔϞδϡʔϧ
    • facet
    ϑΝηοτʹؔ͢ΔϞδϡʔϧ

    View Slide

  44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    LuceneͷϞδϡʔϧ
    • grouping
    ݕࡧ݁Ռͷάϧʔϐϯά༻Ϟδϡʔϧ
    • highlighter
    ݕࡧ݁ՌͷϋΠϥΠτ༻Ϟδϡʔϧ
    • join
    ΠϯσοΫεɺݕࡧ࣌ͷδϣΠϯ༻Ϟδϡʔϧ
    • spatial
    • ஍ཧ৘ใݕࡧ༻Ϟδϡʔϧ

    View Slide

  45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    LuceneͷϞδϡʔϧ
    • suggest
    ΦʔταδΣετɺεϖϧνΣοΫ༻Ϟδϡʔϧ
    • test-framework
    Luceneʹؔ࿈͢ΔΞϓϦέʔγϣϯͷͨΊͷςετϑϨʔϜ
    ϫʔΫ
    • demo
    LuceneͷػೳΛ঺հ͢ΔσϞʢϑΝηοτͱ͔ʣ

    View Slide

  46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearchͱ͸ʁ

    View Slide

  47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϑϦʔϫʔυݕࡧ

    View Slide

  48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ߜΓࠐΈ

    View Slide

  49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϋΠϥΠτ

    View Slide

  50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ιʔτ

    View Slide

  51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϖʔδϯά

    View Slide

  52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ूܭ

    View Slide

  53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    αδΣετ

    View Slide

  54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch in 10 seconds
    • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST
    & JSON
    • Φʔϓϯιʔε: Apache License 2.0
    • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ
    • JavaͰ࣮૷ɻ֦ு΋༰қ

    View Slide

  55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ElasticsearchͰ೔ຊޠ
    • elasticsearch-analysis-kuromoji

    https://github.com/elasticsearch/elasticsearch-
    analysis-kuromoji
    • ΠϯετʔϧʢElasticsearch 1.4.xܥͷ৔߹ʣ
    bin/plugin -i elasticsearch/elasticsearch-analysis-kuromoji/2.4.1

    View Slide

  56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ࢀߟจݙ - Lucene
    • ॻ੶ - Lucene in Action 2nd EditionʢLucene 3ʣ

    http://www.manning.com/hatcher3/
    • ެࣜαΠτ

    http://lucene.apache.org/core/
    • ೔ຊޠͷࢀߟϒϩά

    http://lucene.jugem.jp


    http://mocobeta-backup.tumblr.com/tagged/
    lucene

    View Slide

  57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ࢀߟจݙ - Elasticsearch
    • Github
    https://github.com/elasticsearch
    • ΨΠυ
    http://www.elasticsearch.org/guide/
    • ॻ੶
    ElasticSearchServer೔ຊޠ൛

    View Slide