#javajo ちょっとだけ検索入門

javajoで使用したスライド

Transcript

1. ͪΐͬͱ͚ͩݕࡧೖ໳ ʢElasticsearch΋ͪΐͬͱ͚ͩೖ໳ʣ Jun Ohtani @johtani 2015/07/23
2. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 2 ΞδΣϯμ • ݕࡧͱ͸ʁ • సஔΠϯσοΫεɺN-Gramɺܗଶૉղੳ • ೔ຊޠݕࡧʁ • KuromojiͳͲ • Elasticsearch࢖͏ͱʁ
3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 3 about Me • Me, Jun Ohtani / Technical Adovocate – lucene-gosenίϛολʔ – ElasticSearch Server೔ຊޠ൛ͷ຋༁ – elasticsearch-extended-analysisͷ։ൃ – http://blog.johtani.info • Elasticsearch, founded in 2012 – Products: Elasticsearch, Logstash, Kibana, 
 Marvel, Shield, Watcher, Found
 Professional services: 
 Support & development subscriptions – Trainings
4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 4 ݕࡧͱ͸ʁ ݕࡧͱ͸ʁ
5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 5 ݕࡧʁ
6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 6 Grep?
7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 7 ࡧҾʁ
8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 8 సஔΠϯσοΫεʁ సஔΠϯσοΫεʁ
9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 9 ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ – ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ – υΩϡϝϯτʹؚ·ΕΔଐੑ
10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 10 ༻ޠ • ΫΤϦ • ݕࡧ৚݅ɺݕࡧࣜ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ – ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ – จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ – ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ
11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 11 υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥
12. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 12 υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ
13. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 13 υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕   Ҿ͚ΔΑ͏ʹ
14. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 14 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ
15. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 15 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
16. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 16 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
17. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 17 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
18. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 18 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
19. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 19 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
20. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 20 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
21. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 21 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 22 ୯ޠͷநग़ ୯ޠͷநग़
23. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 23 ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ
24. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 24 N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ – ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩ ʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram – NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ – ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ
25. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 25 ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ • I speak about Lucene in JJUG night seminar. • εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ • ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ
 
 
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ
26. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 26 N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ – ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσ οΫεͷΩʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram – NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ – ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ
27. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 27 ܗଶૉղੳ • ϝϦοτɿ – ҙຯͷ͋Δ୯ޠͷ੾Ε໨
 ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ – ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ ग़ෆೳɻ ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ
28. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 28 ͲΜͳ͜ͱ͕Ͱ͖Δʁ • ޠװม׵ • ʮҿΈ͍ͨʯʮҿΜͩʯʮҿΉʯˠʮҿΉʯʹ౷Ұ • ඼ࢺ৘ใʹΑΔϑΟϧλϦϯά • ʮ໊ࢺʯʮಈࢺʯʮ෭ࢺʯͳͲ • ಡΈͷར༻ • ʮ৽ฉʯˠʮ͠ΜͿΜʯ
29. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 29 N-Gram • ϝϦοτɿ – ະ஌ޠʹରԠՄೳ • σϝϦοτɿ – ΠϯσοΫεංେԽ – ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
30. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 30 ೔ຊޠͷܗଶૉղੳ͸ʁ ೔ຊޠͷܗଶૉղੳ͸ʁ
31. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 31 kuromoji • Christian Moen͞ΜʹΑΓLuceneʹدଃ • ࣙॻʹ͸MeCabͷIPADICࣙॻ/ϞσϧΛར༻ • LuceneͷAnalyzerʹؚ·ΕΔ
32. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 32 Elasticsearchͱ͸ʁ Elasticsearchͱ͸ʁ
33. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 33 ϑϦʔϫʔυݕࡧ
34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 34 ߜΓࠐΈ
35. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 35 ϋΠϥΠτ
36. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 36 ιʔτ
37. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 37 ϖʔδϯά
38. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 38 ूܭ
39. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 39 αδΣετ
40. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 40 Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ
41. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 41 ElasticsearchͰ೔ຊޠ • elasticsearch-analysis-kuromoji
 https://github.com/elasticsearch/elasticsearch-analysis- kuromoji • ΠϯετʔϧʢElasticsearch 1.7.0ͷ৔߹ʣ bin/plugin -i elasticsearch/elasticsearch-analysis-kuromoji/ 2.7.0
42. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 42 Analysis & Mappings Analysis & Mappings
43. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 43 Analysisͱ͸ʁ • సஔΠϯσοΫεͷ୯ޠʢTermʣΛυΩϡϝϯτ͔Βநग़͢Δॲཧ • ϑΟʔϧυຖʹࢦఆ͞ΕͨΞφϥΠβ͕ॲཧ • ΞφϥΠβ͸CharFilterɺTokenizerɺTokenFilter͔Βߏ੒ • ΠϯσοΫε࣌ɺݕࡧ࣌ʹ࣮ߦ
44. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 44 Analysisͷ֓ཁ Analysis υΩϡϝϯτ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ... ΠϯσοΫε
45. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 45 Analysisͷ֓ཁ Analysis ݕࡧ
 Ωʔϫʔυ ΠϯσοΫε ΠϯσοΫε
 ݕࡧॲཧ ݕࡧ݁Ռ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ...
46. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 46 Analyzer Analysisͷߏ੒ Text Tokens char_filt er tokenizer char_filt er token_
 filter token_
 filter
47. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 47 Analyzerͷઃఆ {"index":{ “analysis":{ "analyzer" : { "my_analyzer" : { "type" : "custom", "tokenizer" : “kuromoji_tokenizer”, “char_filter" : [“char_filter1”,…], “filter" : [“token_filter1”,…] } }
48. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 48 Char Filter • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ • ྫɿhtml_strip – ೖྗɿ – ग़ྗɿ <title>Elasticsearch is not a service of AWS</title> Elasticsearch is not a service of AWS
49. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 49 Tokenizer • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹґଘ • ྫɿstandard – ೖྗɿ – ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS
50. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 50 Tokenizer • ྫɿkeyword – ग़ྗɿ • ྫɿkuromoji_tokenizer – ೖྗɿ – ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ Elasticsearch is not a service of AWS ण͕࢘ඒຯ͔ͬͨ͠
51. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 51 TokenFilter • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ • ྫɿlowercase – ೖྗɿ – ग़ྗɿ • ྫɿstop – ೖྗɿ – ग़ྗɿ Elasticsearch is not a service of AWS elasticsearch is not a service of aws Elasticsearch is not a service of AWS Elasticsearch service AWS
52. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 52 TokenFilter • ྫɿkuromoji_baseform – ೖྗɿ – ग़ྗɿ • ྫɿkuromoji_readingform – ೖྗɿ – ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ ण࢘ ͕ ඒຯ͍͠ ͨ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ sushi ga oishika ta
53. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 53 Mapping • ΠϯσοΫεͷߏ଄ʢschemaʣΛఆٛ • Schema less!͚ͩͲ… • ܗࣜΛࢦఆʢanalyze͢Δ/͠ͳ͍ɺ੔਺/গ਺ɺ೔෇ͳͲʣ
54. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 54 ྫɿWikipediaͷMappingͷҰ෦ "mappings": { "page": { "properties": { "link": { "type": "string", "fields": { "raw": {"type": "string", "index": "not_analyzed"} } }, "text": { "type": "string"}, "title": { "type": "string", "fields": { "raw": {"type": "string", "index": "not_analyzed"} }…
55. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 55 WebϑΥʔϥϜ͸͡Ί·ͨ͠ • ໰୊఺ – Google GroupsͰ͸ElasticsearchɺLogstashͳͲ͕ผʑͷML – ֤ࠃޠͷML΋ผʑ – ϝʔϧͩͱຒ΋Εͯ͠·͏͜ͱ΋ଟʑ • ͱ͍͏͜ͱͰ – WebϑΥʔϥϜΛ্ཱͪ͛·ͨ͠ɻ – https://discuss.elastic.co – ೔ຊޠͷΧςΰϦ΋ɻ – ͥͻొ࿥࣭ͯ͠໰ɾճ౴Λʂ
56. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 56 ࢀߟจݙ • Elasticsearch - The Definitive guide – http://www.elastic.co/guide/en/elasticsearch/guide/current/ index.html • ॻ੶ʢ೔ຊޠʣ – ElasticSearchServer೔ຊޠ൛
 αʔό/ΠϯϑϥΤϯδχΞ
 ɹཆ੒ಡຊɹϩάऩू
57. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 57 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/2015/sf/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions • Elasticsearch as a Service, Found.no • https://www.found.no/
58. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://purchases.elastic.co