Luceneと日本語の検索

2014年12月のJJUG ナイトセミナーで使用したスライドです。

Transcript

1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   Jun Ohtani
   @johtani
   2014/12/17 - JJUG φΠτɾηϛφʔ
   Luceneͱ೔ຊޠͷݕࡧ
   

   View Slide

2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   ΞδΣϯμ
   • Luceneʁ
   • సஔΠϯσοΫεɺN-Gramɺܗଶૉղੳ
   • ೔ຊޠݕࡧʁ
   • KuromojiͳͲ
   • Elasticsearch࢖͏ͱʁ
   

   View Slide

3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   about
   • Me, Jun Ohtani / Technical Adovocate
   lucene-gosenίϛολʔ
   ElasticSearch Server೔ຊޠ൛ͷ຋༁
   elasticsearch-extended-analysisͷ։ൃ
   http://blog.johtani.info
   • Elasticsearch, founded in 2012
   Products: Elasticsearch, Logstash, Kibana, Marvel

   Professional services: Support & development subscriptions
   Trainings
   

   View Slide

4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   Luceneͱ͸ʁ
   

   View Slide

5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   Luceneͷಛ௃
   • ߴ଎ͳݕࡧϥΠϒϥϦ
   • Javaϕʔε
   • Apache License 2.0
   • ࠷৽൛ɿ4.10.2ʢ΋͏͙͢ɺ4.10.3ϦϦʔεʣ
   • JDK 7ʢu25͔Βu55ͷؒʹϦϦʔε͞ΕͨJDK͸ό
   ά͕͋ΔͨΊ࢖༻ෆՄʣ
   

   View Slide

6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   Luceneͷओͳػೳ
   • సஔΠϯσοΫεͷ࡞੒ͱݕࡧ
   • ݕࡧ݁ՌͷείΞϦϯά
   • ๛෋ͳΫΤϦʢۙ๣ɺϑϨʔζɺൣғͳͲʣ
   • ϑΟʔϧυࢦఆʹΑΔݕࡧ
   

   View Slide

7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   సஔΠϯσοΫεʁ
   

   View Slide

8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   ༻ޠ
   • ΠϯσοΫε
   ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ
   • υΩϡϝϯτʢจॻʣ
   ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ
   • ϑΟʔϧυ
   υΩϡϝϯτʹؚ·ΕΔଐੑ
   

   View Slide

9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   ༻ޠ
   • ΫΤϦ
   ݕࡧ৚݅ɺݕࡧࣜ
   • λʔϜʢTermʣɺτʔΫϯʢTokenʣ
   ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ
   จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ
   ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ
   

   View Slide

10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    υΩϡϝϯτͷొ࿥
    

    View Slide

11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ
    ఋ
    ࢞
    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ
    

    View Slide

12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ
    ఋ
    ࢞
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ
    ୯ޠ͔Βidͷ഑ྻ͕
    Ҿ͚ΔΑ͏ʹ
    

    View Slide

13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ݕࡧ৚݅ೖྗ
    ΧπΦɹαβΤ
    

    View Slide

14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    https://github.com/johtani/jjug-
    example
    αϯϓϧίʔυ
    

    View Slide

22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ୯ޠͷநग़
    

    View Slide

23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ୯ޠͷ۠੾Γํ
    • ӳޠͷ৔߹
    I speak about Lucene in JJUG night seminar.
    • ೔ຊޠͷ৔߹
    ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ
    

    View Slide

24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ୯ޠͷ۠੾Γํ
    • ӳޠͷ৔߹
    I speak about Lucene in JJUG night seminar.
    εϖʔε͕੾Ε໨ͱΘ͔Δ
    • ೔ຊޠͷ৔߹
    ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ
    

    Ͳ͜Ͱ۠੾Ε͹Α͍ʁ
    

    View Slide

25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    N-Gramͱܗଶૉղੳ
    • సஔΠϯσοΫεͷΩʔͷ࡞Γํ
    ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ
    Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒
    • N-Gram
    NจࣈͣͭจষΛ۠੾Δ
    • ܗଶૉղੳ
    ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ
    

    View Slide

26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ܗଶૉղੳ
    • ϝϦοτɿ
    ҙຯͷ͋Δ୯ޠͷ੾Ε໨

    ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ
    • σϝϦοτɿ
    ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ
    ग़ෆೳɻ
    ΧπΦ͸αβΤͷఋ
    ΧπΦ ͸ ͷ
    αβΤ ఋ
    

    View Slide

27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    N-Gram
    • ϝϦοτɿ
    ະ஌ޠʹରԠՄೳ
    • σϝϦοτɿ
    ΠϯσοΫεංେԽ
    ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ
    ΧπΦ͸αβΤͷఋ
    Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
    

    View Slide

28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analyzerͷ֓ཁ
    Analyzer
    υΩϡϝϯτ
    Term Id
    ΧπΦ 1
    αβΤ 1ɺ2
    ϫΧϝ 2
    … ...
    ΠϯσοΫε
    

    View Slide

29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analyzerͷ֓ཁ
    Analyzer
    ݕࡧ

    Ωʔϫʔυ
    ΠϯσοΫε
    ΠϯσοΫε

    ݕࡧॲཧ
    ݕࡧ݁Ռ
    Term Id
    ΧπΦ 1
    αβΤ 1ɺ2
    ϫΧϝ 2
    … ...
    

    View Slide

30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analyzer
    Analyzerͷߏ੒
    char_filter tokenizer
    char_filter
    token_

    filter
    token_

    filter
    Text Tokens
    

    View Slide

31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Char Filter
    • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ
    • ྫɿhtml_strip
    ೖྗɿ
    ग़ྗɿ
    Elasticsearch is not a service of AWS
    Elasticsearch is not a service of AWS
    

    View Slide

32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Tokenizer
    • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ
    • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹ
    ґଘ
    • ྫɿstandard
    ೖྗɿ
    ग़ྗɿ Elasticsearch is not a service of AWS
    Elasticsearch is not a service of AWS
    

    View Slide

33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Tokenizer
    • ྫɿkeyword
    ग़ྗɿ
    • ྫɿkuromoji_tokenizer
    ೖྗɿ
    ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    Elasticsearch is not a service of AWS
    ण͕࢘ඒຯ͔ͬͨ͠
    

    View Slide

34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    TokenFilter
    • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ
    • ྫɿlowercase
    ೖྗɿ
    ग़ྗɿ
    • ྫɿstop
    ೖྗɿ
    ग़ྗɿ
    Elasticsearch is not a service of AWS
    elasticsearch is not a service of aws
    Elasticsearch is not a service of AWS
    Elasticsearch service AWS
    

    View Slide

35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    TokenFilter
    • ྫɿkuromoji_baseform
    ೖྗɿ
    ग़ྗɿ
    • ྫɿkuromoji_readingform
    ೖྗɿ
    ग़ྗɿ
    ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    ण࢘ ͕ ඒຯ͍͠ ͨ
    ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    sushi ga oishika ta
    

    View Slide

36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysis - ೔ຊޠͩͱʁ
    

    View Slide

37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    kuromoji
    • Christian Moen͞ΜʹΑΓLuceneʹدଃ
    • ࣙॻʹ͸MeCabͷIPADICࣙॻ/ϞσϧΛར༻
    • LuceneͷAnalyzerʹؚ·ΕΔ
    

    View Slide

38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    lucene-gosen
    • LGPL v2
    • MeCabͷࣙॻσʔλΛར༻Մೳ
    • NAISTɺIPADicΛબ୒Մೳ
    • όΠφϦࣙॻͷϏϧυπʔϧ͋Γ
    • https://github.com/lucene-gosen/lucene-gosen
    

    View Slide

39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ͦͷ΄͔ͷLuceneͷػೳ
    

    View Slide

40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ͦͷ΄͔ͷओͳػೳ
    • GEO
    • ϋΠϥΠτ
    • ϑΝηοτ
    • άϧʔϐϯά
    

    View Slide

41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Luceneͷόʔδϣϯ
    • εςʔϒϧɿ4ܥ
    • ։ൃɿ5ܥ - ࣍ظόʔδϣϯ
    • Ϛελɿ6ܥ - Java 8ඞਢ
    

    View Slide

42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    LuceneͷϞδϡʔϧ
    • core
    جຊύοέʔδ
    • analysis
    ݴޠ΍υϝΠϯͷ௥ՃAnalysisύοέʔδ
    • codec
    σʔλܗࣜʢcodecɺpostings formatʣʹؔ͢Δύοέʔδ
    • queries
    ௥ՃͷΫΤϦɺϑΟϧλ
    

    View Slide

43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    LuceneͷϞδϡʔϧ
    • queryparser
    ΫΤϦύʔαʔ͓ΑͼϑϨʔϜϫʔΫ
    • classification
    LuceneͷΠϯσοΫε৘ใΛ༻͍ͨClassificationϞδϡʔϧ
    • expressions
    ಈతͳ஋ͷܭࢉΛϓϥΨϒϧʹมߋՄೳʹ͢ΔϞδϡʔϧ
    • facet
    ϑΝηοτʹؔ͢ΔϞδϡʔϧ
    

    View Slide

44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    LuceneͷϞδϡʔϧ
    • grouping
    ݕࡧ݁Ռͷάϧʔϐϯά༻Ϟδϡʔϧ
    • highlighter
    ݕࡧ݁ՌͷϋΠϥΠτ༻Ϟδϡʔϧ
    • join
    ΠϯσοΫεɺݕࡧ࣌ͷδϣΠϯ༻Ϟδϡʔϧ
    • spatial
    • ஍ཧ৘ใݕࡧ༻Ϟδϡʔϧ
    

    View Slide

45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    LuceneͷϞδϡʔϧ
    • suggest
    ΦʔταδΣετɺεϖϧνΣοΫ༻Ϟδϡʔϧ
    • test-framework
    Luceneʹؔ࿈͢ΔΞϓϦέʔγϣϯͷͨΊͷςετϑϨʔϜ
    ϫʔΫ
    • demo
    LuceneͷػೳΛ঺հ͢ΔσϞʢϑΝηοτͱ͔ʣ
    

    View Slide

46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearchͱ͸ʁ
    

    View Slide

47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϑϦʔϫʔυݕࡧ
    

    View Slide

48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ߜΓࠐΈ
    

    View Slide

49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϋΠϥΠτ
    

    View Slide

50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ιʔτ
    

    View Slide

51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϖʔδϯά
    

    View Slide

52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ूܭ
    

    View Slide

53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    αδΣετ
    

    View Slide

54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch in 10 seconds
    • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST
    & JSON
    • Φʔϓϯιʔε: Apache License 2.0
    • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ
    • JavaͰ࣮૷ɻ֦ு΋༰қ
    

    View Slide

55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ElasticsearchͰ೔ຊޠ
    • elasticsearch-analysis-kuromoji

    https://github.com/elasticsearch/elasticsearch-
    analysis-kuromoji
    • ΠϯετʔϧʢElasticsearch 1.4.xܥͷ৔߹ʣ
    bin/plugin -i elasticsearch/elasticsearch-analysis-kuromoji/2.4.1
    

    View Slide

56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ࢀߟจݙ - Lucene
    • ॻ੶ - Lucene in Action 2nd EditionʢLucene 3ʣ

    http://www.manning.com/hatcher3/
    • ެࣜαΠτ

    http://lucene.apache.org/core/
    • ೔ຊޠͷࢀߟϒϩά

    http://lucene.jugem.jp

    

    http://mocobeta-backup.tumblr.com/tagged/
    lucene
    

    View Slide

57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ࢀߟจݙ - Elasticsearch
    • Github
    https://github.com/elasticsearch
    • ΨΠυ
    http://www.elasticsearch.org/guide/
    • ॻ੶
    ElasticSearchServer೔ຊޠ൛
    

    View Slide