Luceneと日本語の検索

Slide 1

Slide 1 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Jun Ohtani @johtani 2014/12/17 - JJUG φΠτɾηϛφʔ Luceneͱ೔ຊޠͷݕࡧ

Slide 2

Slide 2 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΞδΣϯμ • Luceneʁ • సஔΠϯσοΫεɺN-Gramɺܗଶૉղੳ • ೔ຊޠݕࡧʁ • KuromojiͳͲ • Elasticsearch࢖͏ͱʁ

Slide 3

Slide 3 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited about • Me, Jun Ohtani / Technical Adovocate lucene-gosenίϛολʔ ElasticSearch Server೔ຊޠ൛ͷ຋༁ elasticsearch-extended-analysisͷ։ൃ http://blog.johtani.info • Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel  Professional services: Support & development subscriptions Trainings

Slide 4

Slide 4 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Luceneͱ͸ʁ

Slide 5

Slide 5 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Luceneͷಛ௃ • ߴ଎ͳݕࡧϥΠϒϥϦ • Javaϕʔε • Apache License 2.0 • ࠷৽൛ɿ4.10.2ʢ΋͏͙͢ɺ4.10.3ϦϦʔεʣ • JDK 7ʢu25͔Βu55ͷؒʹϦϦʔε͞ΕͨJDK͸ό ά͕͋ΔͨΊ࢖༻ෆՄʣ

Slide 6

Slide 6 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Luceneͷओͳػೳ • సஔΠϯσοΫεͷ࡞੒ͱݕࡧ • ݕࡧ݁ՌͷείΞϦϯά • ๛෋ͳΫΤϦʢۙ๣ɺϑϨʔζɺൣғͳͲʣ • ϑΟʔϧυࢦఆʹΑΔݕࡧ

Slide 7

Slide 7 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited సஔΠϯσοΫεʁ

Slide 8

Slide 8 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ υΩϡϝϯτʹؚ·ΕΔଐੑ

Slide 9

Slide 9 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ༻ޠ • ΫΤϦ ݕࡧ৚݅ɺݕࡧࣜ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ

Slide 10

Slide 10 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥

Slide 11

Slide 11 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ

Slide 14

Slide 14 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited https://github.com/johtani/jjug- example αϯϓϧίʔυ

Slide 22

Slide 22 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷநग़

Slide 23

Slide 23 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ

Slide 24

Slide 24 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I speak about Lucene in JJUG night seminar. εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸JJUGφΠτηϛφʔͰLuceneʹ͍ͭͯ࿩͢ɻ   Ͳ͜Ͱ۠੾Ε͹Α͍ʁ

Slide 25

Slide 25 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ

Slide 26

Slide 26 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ܗଶૉղੳ • ϝϦοτɿ ҙຯͷ͋Δ୯ޠͷ੾Ε໨  ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ ग़ෆೳɻ ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ

Slide 27

Slide 27 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited N-Gram • ϝϦοτɿ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ΠϯσοΫεංେԽ ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

Slide 28

Slide 28 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analyzerͷ֓ཁ Analyzer υΩϡϝϯτ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ... ΠϯσοΫε

Slide 29

Slide 29 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analyzerͷ֓ཁ Analyzer ݕࡧ  Ωʔϫʔυ ΠϯσοΫε ΠϯσοΫε  ݕࡧॲཧ ݕࡧ݁Ռ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ...

Slide 30

Slide 30 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analyzer Analyzerͷߏ੒ char_filter tokenizer char_filter token_  filter token_  filter Text Tokens

Slide 31

Slide 31 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Char Filter • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ • ྫɿhtml_strip ೖྗɿ ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS

Slide 32

Slide 32 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Tokenizer • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹ ґଘ • ྫɿstandard ೖྗɿ ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS

Slide 33

Slide 33 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Tokenizer • ྫɿkeyword ग़ྗɿ • ྫɿkuromoji_tokenizer ೖྗɿ ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ Elasticsearch is not a service of AWS ण͕࢘ඒຯ͔ͬͨ͠

Slide 34

Slide 34 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited TokenFilter • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ • ྫɿlowercase ೖྗɿ ग़ྗɿ • ྫɿstop ೖྗɿ ग़ྗɿ Elasticsearch is not a service of AWS elasticsearch is not a service of aws Elasticsearch is not a service of AWS Elasticsearch service AWS

Slide 35

Slide 35 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited TokenFilter • ྫɿkuromoji_baseform ೖྗɿ ग़ྗɿ • ྫɿkuromoji_readingform ೖྗɿ ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ ण࢘ ͕ ඒຯ͍͠ ͨ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ sushi ga oishika ta

Slide 36

Slide 36 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analysis - ೔ຊޠͩͱʁ

Slide 37

Slide 37 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited kuromoji • Christian Moen͞ΜʹΑΓLuceneʹدଃ • ࣙॻʹ͸MeCabͷIPADICࣙॻ/ϞσϧΛར༻ • LuceneͷAnalyzerʹؚ·ΕΔ

Slide 38

Slide 38 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited lucene-gosen • LGPL v2 • MeCabͷࣙॻσʔλΛར༻Մೳ • NAISTɺIPADicΛબ୒Մೳ • όΠφϦࣙॻͷϏϧυπʔϧ͋Γ • https://github.com/lucene-gosen/lucene-gosen

Slide 39

Slide 39 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ͦͷ΄͔ͷLuceneͷػೳ

Slide 40

Slide 40 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ͦͷ΄͔ͷओͳػೳ • GEO • ϋΠϥΠτ • ϑΝηοτ • άϧʔϐϯά

Slide 41

Slide 41 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Luceneͷόʔδϣϯ • εςʔϒϧɿ4ܥ • ։ൃɿ5ܥ - ࣍ظόʔδϣϯ • Ϛελɿ6ܥ - Java 8ඞਢ

Slide 42

Slide 42 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited LuceneͷϞδϡʔϧ • core جຊύοέʔδ • analysis ݴޠ΍υϝΠϯͷ௥ՃAnalysisύοέʔδ • codec σʔλܗࣜʢcodecɺpostings formatʣʹؔ͢Δύοέʔδ • queries ௥ՃͷΫΤϦɺϑΟϧλ

Slide 43

Slide 43 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited LuceneͷϞδϡʔϧ • queryparser ΫΤϦύʔαʔ͓ΑͼϑϨʔϜϫʔΫ • classification LuceneͷΠϯσοΫε৘ใΛ༻͍ͨClassificationϞδϡʔϧ • expressions ಈతͳ஋ͷܭࢉΛϓϥΨϒϧʹมߋՄೳʹ͢ΔϞδϡʔϧ • facet ϑΝηοτʹؔ͢ΔϞδϡʔϧ

Slide 44

Slide 44 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited LuceneͷϞδϡʔϧ • grouping ݕࡧ݁Ռͷάϧʔϐϯά༻Ϟδϡʔϧ • highlighter ݕࡧ݁ՌͷϋΠϥΠτ༻Ϟδϡʔϧ • join ΠϯσοΫεɺݕࡧ࣌ͷδϣΠϯ༻Ϟδϡʔϧ • spatial • ஍ཧ৘ใݕࡧ༻Ϟδϡʔϧ

Slide 45

Slide 45 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited LuceneͷϞδϡʔϧ • suggest ΦʔταδΣετɺεϖϧνΣοΫ༻Ϟδϡʔϧ • test-framework Luceneʹؔ࿈͢ΔΞϓϦέʔγϣϯͷͨΊͷςετϑϨʔϜ ϫʔΫ • demo LuceneͷػೳΛ঺հ͢ΔσϞʢϑΝηοτͱ͔ʣ

Slide 46

Slide 46 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearchͱ͸ʁ

Slide 47

Slide 47 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϑϦʔϫʔυݕࡧ

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ

Slide 55

Slide 55 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ElasticsearchͰ೔ຊޠ • elasticsearch-analysis-kuromoji  https://github.com/elasticsearch/elasticsearch- analysis-kuromoji • ΠϯετʔϧʢElasticsearch 1.4.xܥͷ৔߹ʣ bin/plugin -i elasticsearch/elasticsearch-analysis-kuromoji/2.4.1

Slide 56

Slide 56 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ࢀߟจݙ - Lucene • ॻ੶ - Lucene in Action 2nd EditionʢLucene 3ʣ  http://www.manning.com/hatcher3/ • ެࣜαΠτ  http://lucene.apache.org/core/ • ೔ຊޠͷࢀߟϒϩά  http://lucene.jugem.jp    http://mocobeta-backup.tumblr.com/tagged/ lucene

Slide 57

Slide 57 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ࢀߟจݙ - Elasticsearch • Github https://github.com/elasticsearch • ΨΠυ http://www.elasticsearch.org/guide/ • ॻ੶ ElasticSearchServer೔ຊޠ൛