#javajo ちょっとだけ検索入門

ͪΐͬͱ͚ͩݕࡧೖ໳ ʢElasticsearch΋ͪΐͬͱ͚ͩೖ໳ʣ Jun Ohtani @johtani 2015/07/23

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written
permission is strictly prohibited 2 ΞδΣϯμ • ݕࡧͱ͸ʁ • సஔΠϯσοΫεɺN-Gramɺܗଶૉղੳ • ೔ຊޠݕࡧʁ • KuromojiͳͲ • Elasticsearch࢖͏ͱʁ

permission is strictly prohibited 3 about Me • Me, Jun Ohtani / Technical Adovocate – lucene-gosenίϛολʔ – ElasticSearch Server೔ຊޠ൛ͷ຋༁ – elasticsearch-extended-analysisͷ։ൃ – http://blog.johtani.info • Elasticsearch, founded in 2012 – Products: Elasticsearch, Logstash, Kibana,   Marvel, Shield, Watcher, Found  Professional services:   Support & development subscriptions – Trainings

permission is strictly prohibited 4 ݕࡧͱ͸ʁ ݕࡧͱ͸ʁ

permission is strictly prohibited 5 ݕࡧʁ

permission is strictly prohibited 6 Grep?

permission is strictly prohibited 7 ࡧҾʁ

permission is strictly prohibited 8 సஔΠϯσοΫεʁ సஔΠϯσοΫεʁ

permission is strictly prohibited 9 ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ – ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ – υΩϡϝϯτʹؚ·ΕΔଐੑ

permission is strictly prohibited 10 ༻ޠ • ΫΤϦ • ݕࡧ৚݅ɺݕࡧࣜ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ – ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ – จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ – ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ

permission is strictly prohibited 11 υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥

permission is strictly prohibited 12 υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ

permission is strictly prohibited 13 υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ

permission is strictly prohibited 14 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ

permission is strictly prohibited 15 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

permission is strictly prohibited 22 ୯ޠͷநग़ ୯ޠͷநग़

permission is strictly prohibited 23 ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ

permission is strictly prohibited 24 N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ – ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩ ʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram – NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ – ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ

permission is strictly prohibited 25 ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ • I speak about Lucene in JJUG night seminar. • εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ • ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ      Ͳ͜Ͱ۠੾Ε͹Α͍ʁ

permission is strictly prohibited 26 N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ – ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσ οΫεͷΩʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram – NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ – ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ

permission is strictly prohibited 27 ܗଶૉղੳ • ϝϦοτɿ – ҙຯͷ͋Δ୯ޠͷ੾Ε໨  ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ – ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ ग़ෆೳɻ ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ

permission is strictly prohibited 28 ͲΜͳ͜ͱ͕Ͱ͖Δʁ • ޠװม׵ • ʮҿΈ͍ͨʯʮҿΜͩʯʮҿΉʯˠʮҿΉʯʹ౷Ұ • ඼ࢺ৘ใʹΑΔϑΟϧλϦϯά • ʮ໊ࢺʯʮಈࢺʯʮ෭ࢺʯͳͲ • ಡΈͷར༻ • ʮ৽ฉʯˠʮ͠ΜͿΜʯ

permission is strictly prohibited 29 N-Gram • ϝϦοτɿ – ະ஌ޠʹରԠՄೳ • σϝϦοτɿ – ΠϯσοΫεංେԽ – ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

permission is strictly prohibited 30 ೔ຊޠͷܗଶૉղੳ͸ʁ ೔ຊޠͷܗଶૉղੳ͸ʁ

permission is strictly prohibited 31 kuromoji • Christian Moen͞ΜʹΑΓLuceneʹدଃ • ࣙॻʹ͸MeCabͷIPADICࣙॻ/ϞσϧΛར༻ • LuceneͷAnalyzerʹؚ·ΕΔ

permission is strictly prohibited 32 Elasticsearchͱ͸ʁ Elasticsearchͱ͸ʁ

permission is strictly prohibited 33 ϑϦʔϫʔυݕࡧ

permission is strictly prohibited 34 ߜΓࠐΈ

permission is strictly prohibited 35 ϋΠϥΠτ

permission is strictly prohibited 36 ιʔτ

permission is strictly prohibited 37 ϖʔδϯά

permission is strictly prohibited 38 ूܭ

permission is strictly prohibited 39 αδΣετ

permission is strictly prohibited 40 Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ

permission is strictly prohibited 41 ElasticsearchͰ೔ຊޠ • elasticsearch-analysis-kuromoji  https://github.com/elasticsearch/elasticsearch-analysis- kuromoji • ΠϯετʔϧʢElasticsearch 1.7.0ͷ৔߹ʣ bin/plugin -i elasticsearch/elasticsearch-analysis-kuromoji/ 2.7.0

permission is strictly prohibited 42 Analysis & Mappings Analysis & Mappings

permission is strictly prohibited 43 Analysisͱ͸ʁ • సஔΠϯσοΫεͷ୯ޠʢTermʣΛυΩϡϝϯτ͔Βநग़͢Δॲཧ • ϑΟʔϧυຖʹࢦఆ͞ΕͨΞφϥΠβ͕ॲཧ • ΞφϥΠβ͸CharFilterɺTokenizerɺTokenFilter͔Βߏ੒ • ΠϯσοΫε࣌ɺݕࡧ࣌ʹ࣮ߦ

permission is strictly prohibited 44 Analysisͷ֓ཁ Analysis υΩϡϝϯτ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ... ΠϯσοΫε

permission is strictly prohibited 45 Analysisͷ֓ཁ Analysis ݕࡧ  Ωʔϫʔυ ΠϯσοΫε ΠϯσοΫε  ݕࡧॲཧ ݕࡧ݁Ռ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ...

permission is strictly prohibited 46 Analyzer Analysisͷߏ੒ Text Tokens char_filt er tokenizer char_filt er token_  filter token_  filter

permission is strictly prohibited 47 Analyzerͷઃఆ {"index":{ “analysis":{ "analyzer" : { "my_analyzer" : { "type" : "custom", "tokenizer" : “kuromoji_tokenizer”, “char_filter" : [“char_filter1”,…], “filter" : [“token_filter1”,…] } }

permission is strictly prohibited 48 Char Filter • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ • ྫɿhtml_strip – ೖྗɿ – ग़ྗɿ <title>Elasticsearch is not a service of AWS</title> Elasticsearch is not a service of AWS

permission is strictly prohibited 49 Tokenizer • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹґଘ • ྫɿstandard – ೖྗɿ – ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS

permission is strictly prohibited 50 Tokenizer • ྫɿkeyword – ग़ྗɿ • ྫɿkuromoji_tokenizer – ೖྗɿ – ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ Elasticsearch is not a service of AWS ण͕࢘ඒຯ͔ͬͨ͠

permission is strictly prohibited 51 TokenFilter • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ • ྫɿlowercase – ೖྗɿ – ग़ྗɿ • ྫɿstop – ೖྗɿ – ग़ྗɿ Elasticsearch is not a service of AWS elasticsearch is not a service of aws Elasticsearch is not a service of AWS Elasticsearch service AWS

permission is strictly prohibited 52 TokenFilter • ྫɿkuromoji_baseform – ೖྗɿ – ग़ྗɿ • ྫɿkuromoji_readingform – ೖྗɿ – ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ ण࢘ ͕ ඒຯ͍͠ ͨ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ sushi ga oishika ta

permission is strictly prohibited 53 Mapping • ΠϯσοΫεͷߏ଄ʢschemaʣΛఆٛ • Schema less!͚ͩͲ… • ܗࣜΛࢦఆʢanalyze͢Δ/͠ͳ͍ɺ੔਺/গ਺ɺ೔෇ͳͲʣ

permission is strictly prohibited 54 ྫɿWikipediaͷMappingͷҰ෦ "mappings": { "page": { "properties": { "link": { "type": "string", "fields": { "raw": {"type": "string", "index": "not_analyzed"} } }, "text": { "type": "string"}, "title": { "type": "string", "fields": { "raw": {"type": "string", "index": "not_analyzed"} }…

permission is strictly prohibited 55 WebϑΥʔϥϜ͸͡Ί·ͨ͠ • ໰୊఺ – Google GroupsͰ͸ElasticsearchɺLogstashͳͲ͕ผʑͷML – ֤ࠃޠͷML΋ผʑ – ϝʔϧͩͱຒ΋Εͯ͠·͏͜ͱ΋ଟʑ • ͱ͍͏͜ͱͰ – WebϑΥʔϥϜΛ্ཱͪ͛·ͨ͠ɻ – https://discuss.elastic.co – ೔ຊޠͷΧςΰϦ΋ɻ – ͥͻొ࿥࣭ͯ͠໰ɾճ౴Λʂ

permission is strictly prohibited 56 ࢀߟจݙ • Elasticsearch - The Definitive guide – http://www.elastic.co/guide/en/elasticsearch/guide/current/ index.html • ॻ੶ʢ೔ຊޠʣ – ElasticSearchServer೔ຊޠ൛  αʔό/ΠϯϑϥΤϯδχΞ  ɹཆ੒ಡຊɹϩάऩू

permission is strictly prohibited 57 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/2015/sf/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions • Elasticsearch as a Service, Found.no • https://www.found.no/

Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re
helping! https://www.elastic.co/subscriptions http://purchases.elastic.co

#javajo ちょっとだけ検索入門

#javajo ちょっとだけ検索入門

More Decks by Jun Ohtani

Other Decks in Technology

Featured

Transcript