Luceneと日本語の検索

2014年12月のJJUG ナイトセミナーで使用したスライドです。

Transcript

1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Jun Ohtani @johtani 2014/12/17 - JJUG φΠτɾηϛφʔ Luceneͱ೔ຊޠͷݕࡧ
2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΞδΣϯμ • Luceneʁ • సஔΠϯσοΫεɺN-Gramɺܗଶૉղੳ • ೔ຊޠݕࡧʁ • KuromojiͳͲ • Elasticsearch࢖͏ͱʁ
3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited about • Me, Jun Ohtani / Technical Adovocate lucene-gosenίϛολʔ ElasticSearch Server೔ຊޠ൛ͷ຋༁ elasticsearch-extended-analysisͷ։ൃ http://blog.johtani.info • Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel
 Professional services: Support & development subscriptions Trainings
4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Luceneͱ͸ʁ
5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Luceneͷಛ௃ • ߴ଎ͳݕࡧϥΠϒϥϦ • Javaϕʔε • Apache License 2.0 • ࠷৽൛ɿ4.10.2ʢ΋͏͙͢ɺ4.10.3ϦϦʔεʣ • JDK 7ʢu25͔Βu55ͷؒʹϦϦʔε͞ΕͨJDK͸ό ά͕͋ΔͨΊ࢖༻ෆՄʣ
6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Luceneͷओͳػೳ • సஔΠϯσοΫεͷ࡞੒ͱݕࡧ • ݕࡧ݁ՌͷείΞϦϯά • ๛෋ͳΫΤϦʢۙ๣ɺϑϨʔζɺൣғͳͲʣ • ϑΟʔϧυࢦఆʹΑΔݕࡧ
7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited సஔΠϯσοΫεʁ
8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ υΩϡϝϯτʹؚ·ΕΔଐੑ
9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ༻ޠ • ΫΤϦ ݕࡧ৚݅ɺݕࡧࣜ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ
10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥
11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ
12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ
13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ
14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited https://github.com/johtani/jjug- example αϯϓϧίʔυ
22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷநग़
23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ
24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ 
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ
25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ
26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ܗଶૉղੳ • ϝϦοτɿ ҙຯͷ͋Δ୯ޠͷ੾Ε໨
 ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ ग़ෆೳɻ ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ
27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited N-Gram • ϝϦοτɿ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ΠϯσοΫεංେԽ ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analyzerͷ֓ཁ Analyzer υΩϡϝϯτ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ... ΠϯσοΫε
29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analyzerͷ֓ཁ Analyzer ݕࡧ
 Ωʔϫʔυ ΠϯσοΫε ΠϯσοΫε
 ݕࡧॲཧ ݕࡧ݁Ռ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ...
30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analyzer Analyzerͷߏ੒ char_filter tokenizer char_filter token_
 filter token_
 filter Text Tokens
31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Char Filter • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ • ྫɿhtml_strip ೖྗɿ ग़ྗɿ <title>Elasticsearch is not a service of AWS</title> Elasticsearch is not a service of AWS
32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Tokenizer • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹ ґଘ • ྫɿstandard ೖྗɿ ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS
33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Tokenizer • ྫɿkeyword ग़ྗɿ • ྫɿkuromoji_tokenizer ೖྗɿ ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ Elasticsearch is not a service of AWS ण͕࢘ඒຯ͔ͬͨ͠
34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited TokenFilter • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ • ྫɿlowercase ೖྗɿ ग़ྗɿ • ྫɿstop ೖྗɿ ग़ྗɿ Elasticsearch is not a service of AWS elasticsearch is not a service of aws Elasticsearch is not a service of AWS Elasticsearch service AWS
35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited TokenFilter • ྫɿkuromoji_baseform ೖྗɿ ग़ྗɿ • ྫɿkuromoji_readingform ೖྗɿ ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ ण࢘ ͕ ඒຯ͍͠ ͨ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ sushi ga oishika ta
36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analysis - ೔ຊޠͩͱʁ
37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited kuromoji • Christian Moen͞ΜʹΑΓLuceneʹدଃ • ࣙॻʹ͸MeCabͷIPADICࣙॻ/ϞσϧΛར༻ • LuceneͷAnalyzerʹؚ·ΕΔ
38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited lucene-gosen • LGPL v2 • MeCabͷࣙॻσʔλΛར༻Մೳ • NAISTɺIPADicΛબ୒Մೳ • όΠφϦࣙॻͷϏϧυπʔϧ͋Γ • https://github.com/lucene-gosen/lucene-gosen
39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ͦͷ΄͔ͷLuceneͷػೳ
40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ͦͷ΄͔ͷओͳػೳ • GEO • ϋΠϥΠτ • ϑΝηοτ • άϧʔϐϯά
41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Luceneͷόʔδϣϯ • εςʔϒϧɿ4ܥ • ։ൃɿ5ܥ - ࣍ظόʔδϣϯ • Ϛελɿ6ܥ - Java 8ඞਢ
42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited LuceneͷϞδϡʔϧ • core جຊύοέʔδ • analysis ݴޠ΍υϝΠϯͷ௥ՃAnalysisύοέʔδ • codec σʔλܗࣜʢcodecɺpostings formatʣʹؔ͢Δύοέʔδ • queries ௥ՃͷΫΤϦɺϑΟϧλ
43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited LuceneͷϞδϡʔϧ • queryparser ΫΤϦύʔαʔ͓ΑͼϑϨʔϜϫʔΫ • classification LuceneͷΠϯσοΫε৘ใΛ༻͍ͨClassificationϞδϡʔϧ • expressions ಈతͳ஋ͷܭࢉΛϓϥΨϒϧʹมߋՄೳʹ͢ΔϞδϡʔϧ • facet ϑΝηοτʹؔ͢ΔϞδϡʔϧ
44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited LuceneͷϞδϡʔϧ • grouping ݕࡧ݁Ռͷάϧʔϐϯά༻Ϟδϡʔϧ • highlighter ݕࡧ݁ՌͷϋΠϥΠτ༻Ϟδϡʔϧ • join ΠϯσοΫεɺݕࡧ࣌ͷδϣΠϯ༻Ϟδϡʔϧ • spatial • ஍ཧ৘ใݕࡧ༻Ϟδϡʔϧ
45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited LuceneͷϞδϡʔϧ • suggest ΦʔταδΣετɺεϖϧνΣοΫ༻Ϟδϡʔϧ • test-framework Luceneʹؔ࿈͢ΔΞϓϦέʔγϣϯͷͨΊͷςετϑϨʔϜ ϫʔΫ • demo LuceneͷػೳΛ঺հ͢ΔσϞʢϑΝηοτͱ͔ʣ
46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearchͱ͸ʁ
47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϑϦʔϫʔυݕࡧ
48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ߜΓࠐΈ
49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϋΠϥΠτ
50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ιʔτ
51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϖʔδϯά
52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ूܭ
53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited αδΣετ
54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ
55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ElasticsearchͰ೔ຊޠ • elasticsearch-analysis-kuromoji
 https://github.com/elasticsearch/elasticsearch- analysis-kuromoji • ΠϯετʔϧʢElasticsearch 1.4.xܥͷ৔߹ʣ bin/plugin -i elasticsearch/elasticsearch-analysis-kuromoji/2.4.1
56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ࢀߟจݙ - Lucene • ॻ੶ - Lucene in Action 2nd EditionʢLucene 3ʣ
 http://www.manning.com/hatcher3/ • ެࣜαΠτ
 http://lucene.apache.org/core/ • ೔ຊޠͷࢀߟϒϩά
 http://lucene.jugem.jp
 
 http://mocobeta-backup.tumblr.com/tagged/ lucene
57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ࢀߟจݙ - Elasticsearch • Github https://github.com/elasticsearch • ΨΠυ http://www.elasticsearch.org/guide/ • ॻ੶ ElasticSearchServer೔ຊޠ൛