#javajo ちょっとだけ検索入門

by Jun Ohtani

Slide 1

Slide 1 text

ͪΐͬͱ͚ͩݕࡧೖ໳ ʢElasticsearch΋ͪΐͬͱ͚ͩೖ໳ʣ Jun Ohtani @johtani 2015/07/23

Slide 2

Slide 2 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 2 ΞδΣϯμ • ݕࡧͱ͸ʁ • సஔΠϯσοΫεɺN-Gramɺܗଶૉղੳ • ೔ຊޠݕࡧʁ • KuromojiͳͲ • Elasticsearch࢖͏ͱʁ

Slide 3

Slide 3 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 3 about Me • Me, Jun Ohtani / Technical Adovocate – lucene-gosenίϛολʔ – ElasticSearch Server೔ຊޠ൛ͷ຋༁ – elasticsearch-extended-analysisͷ։ൃ – http://blog.johtani.info • Elasticsearch, founded in 2012 – Products: Elasticsearch, Logstash, Kibana,   Marvel, Shield, Watcher, Found  Professional services:   Support & development subscriptions – Trainings

Slide 4

Slide 4 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 4 ݕࡧͱ͸ʁ ݕࡧͱ͸ʁ

Slide 5

Slide 5 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 5 ݕࡧʁ

Slide 6

Slide 6 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 6 Grep?

Slide 7

Slide 7 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 7 ࡧҾʁ

Slide 8

Slide 8 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 8 సஔΠϯσοΫεʁ సஔΠϯσοΫεʁ

Slide 9

Slide 9 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 9 ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ – ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ – υΩϡϝϯτʹؚ·ΕΔଐੑ

Slide 10

Slide 10 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 10 ༻ޠ • ΫΤϦ • ݕࡧ৚݅ɺݕࡧࣜ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ – ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ – จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ – ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ

Slide 11

Slide 11 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 11 υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥

Slide 12

Slide 12 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 12 υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ

Slide 13

Slide 13 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 13 υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ

Slide 14

Slide 14 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 14 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ

Slide 15

Slide 15 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 15 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 16

Slide 16 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 16 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 17

Slide 17 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 17 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 18

Slide 18 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 18 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 19

Slide 19 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 19 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 20

Slide 20 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 20 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 21

Slide 21 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 21 ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 22

Slide 22 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 22 ୯ޠͷநग़ ୯ޠͷநग़

Slide 23

Slide 23 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 23 ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ

Slide 24

Slide 24 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 24 N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ – ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩ ʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram – NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ – ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ

Slide 25

Slide 25 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 25 ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ • I speak about Lucene in JJUG night seminar. • εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ • ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ      Ͳ͜Ͱ۠੾Ε͹Α͍ʁ

Slide 26

Slide 26 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 26 N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ – ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσ οΫεͷΩʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram – NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ – ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ

Slide 27

Slide 27 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 27 ܗଶૉղੳ • ϝϦοτɿ – ҙຯͷ͋Δ୯ޠͷ੾Ε໨  ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ – ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ ग़ෆೳɻ ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ

Slide 28

Slide 28 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 28 ͲΜͳ͜ͱ͕Ͱ͖Δʁ • ޠװม׵ • ʮҿΈ͍ͨʯʮҿΜͩʯʮҿΉʯˠʮҿΉʯʹ౷Ұ • ඼ࢺ৘ใʹΑΔϑΟϧλϦϯά • ʮ໊ࢺʯʮಈࢺʯʮ෭ࢺʯͳͲ • ಡΈͷར༻ • ʮ৽ฉʯˠʮ͠ΜͿΜʯ

Slide 29

Slide 29 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 29 N-Gram • ϝϦοτɿ – ະ஌ޠʹରԠՄೳ • σϝϦοτɿ – ΠϯσοΫεංେԽ – ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

Slide 30

Slide 30 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 30 ೔ຊޠͷܗଶૉղੳ͸ʁ ೔ຊޠͷܗଶૉղੳ͸ʁ

Slide 31

Slide 31 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 31 kuromoji • Christian Moen͞ΜʹΑΓLuceneʹدଃ • ࣙॻʹ͸MeCabͷIPADICࣙॻ/ϞσϧΛར༻ • LuceneͷAnalyzerʹؚ·ΕΔ

Slide 32

Slide 32 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 32 Elasticsearchͱ͸ʁ Elasticsearchͱ͸ʁ

Slide 33

Slide 33 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 33 ϑϦʔϫʔυݕࡧ

Slide 34

Slide 34 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 34 ߜΓࠐΈ

Slide 35

Slide 35 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 35 ϋΠϥΠτ

Slide 36

Slide 36 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 36 ιʔτ

Slide 37

Slide 37 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 37 ϖʔδϯά

Slide 38

Slide 38 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 38 ूܭ

Slide 39

Slide 39 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 39 αδΣετ

Slide 40

Slide 40 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 40 Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ

Slide 41

Slide 41 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 41 ElasticsearchͰ೔ຊޠ • elasticsearch-analysis-kuromoji  https://github.com/elasticsearch/elasticsearch-analysis- kuromoji • ΠϯετʔϧʢElasticsearch 1.7.0ͷ৔߹ʣ bin/plugin -i elasticsearch/elasticsearch-analysis-kuromoji/ 2.7.0

Slide 42

Slide 42 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 42 Analysis & Mappings Analysis & Mappings

Slide 43

Slide 43 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 43 Analysisͱ͸ʁ • సஔΠϯσοΫεͷ୯ޠʢTermʣΛυΩϡϝϯτ͔Βநग़͢Δॲཧ • ϑΟʔϧυຖʹࢦఆ͞ΕͨΞφϥΠβ͕ॲཧ • ΞφϥΠβ͸CharFilterɺTokenizerɺTokenFilter͔Βߏ੒ • ΠϯσοΫε࣌ɺݕࡧ࣌ʹ࣮ߦ

Slide 44

Slide 44 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 44 Analysisͷ֓ཁ Analysis υΩϡϝϯτ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ... ΠϯσοΫε

Slide 45

Slide 45 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 45 Analysisͷ֓ཁ Analysis ݕࡧ  Ωʔϫʔυ ΠϯσοΫε ΠϯσοΫε  ݕࡧॲཧ ݕࡧ݁Ռ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ...

Slide 46

Slide 46 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 46 Analyzer Analysisͷߏ੒ Text Tokens char_filt er tokenizer char_filt er token_  filter token_  filter

Slide 47

Slide 47 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 47 Analyzerͷઃఆ {"index":{ “analysis":{ "analyzer" : { "my_analyzer" : { "type" : "custom", "tokenizer" : “kuromoji_tokenizer”, “char_filter" : [“char_filter1”,…], “filter" : [“token_filter1”,…] } }

Slide 48

Slide 48 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 48 Char Filter • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ • ྫɿhtml_strip – ೖྗɿ – ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS

Slide 49

Slide 49 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 49 Tokenizer • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹґଘ • ྫɿstandard – ೖྗɿ – ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS

Slide 50

Slide 50 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 50 Tokenizer • ྫɿkeyword – ग़ྗɿ • ྫɿkuromoji_tokenizer – ೖྗɿ – ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ Elasticsearch is not a service of AWS ण͕࢘ඒຯ͔ͬͨ͠

Slide 51

Slide 51 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 51 TokenFilter • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ • ྫɿlowercase – ೖྗɿ – ग़ྗɿ • ྫɿstop – ೖྗɿ – ग़ྗɿ Elasticsearch is not a service of AWS elasticsearch is not a service of aws Elasticsearch is not a service of AWS Elasticsearch service AWS

Slide 52

Slide 52 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 52 TokenFilter • ྫɿkuromoji_baseform – ೖྗɿ – ग़ྗɿ • ྫɿkuromoji_readingform – ೖྗɿ – ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ ण࢘ ͕ ඒຯ͍͠ ͨ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ sushi ga oishika ta

Slide 53

Slide 53 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 53 Mapping • ΠϯσοΫεͷߏ଄ʢschemaʣΛఆٛ • Schema less!͚ͩͲ… • ܗࣜΛࢦఆʢanalyze͢Δ/͠ͳ͍ɺ੔਺/গ਺ɺ೔෇ͳͲʣ

Slide 54

Slide 54 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 54 ྫɿWikipediaͷMappingͷҰ෦ "mappings": { "page": { "properties": { "link": { "type": "string", "fields": { "raw": {"type": "string", "index": "not_analyzed"} } }, "text": { "type": "string"}, "title": { "type": "string", "fields": { "raw": {"type": "string", "index": "not_analyzed"} }…

Slide 55

Slide 55 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 55 WebϑΥʔϥϜ͸͡Ί·ͨ͠ • ໰୊఺ – Google GroupsͰ͸ElasticsearchɺLogstashͳͲ͕ผʑͷML – ֤ࠃޠͷML΋ผʑ – ϝʔϧͩͱຒ΋Εͯ͠·͏͜ͱ΋ଟʑ • ͱ͍͏͜ͱͰ – WebϑΥʔϥϜΛ্ཱͪ͛·ͨ͠ɻ – https://discuss.elastic.co – ೔ຊޠͷΧςΰϦ΋ɻ – ͥͻొ࿥࣭ͯ͠໰ɾճ౴Λʂ

Slide 56

Slide 56 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 56 ࢀߟจݙ • Elasticsearch - The Definitive guide – http://www.elastic.co/guide/en/elasticsearch/guide/current/ index.html • ॻ੶ʢ೔ຊޠʣ – ElasticSearchServer೔ຊޠ൛  αʔό/ΠϯϑϥΤϯδχΞ  ɹཆ੒ಡຊɹϩάऩू

Slide 57

Slide 57 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 57 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/2015/sf/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions • Elasticsearch as a Service, Found.no • https://www.found.no/

Slide 58

Slide 58 text

Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://purchases.elastic.co