Introduction Elasticsearch / Full Text Search at TMU

Slide 1

Slide 1 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Jun Ohtani @johtani ೖ໳Elasticsearch

Slide 2

Slide 2 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΞδΣϯμ • શจݕࡧͱ͸ʁʢೖ໳͢Δલʹʣ • సஔΠϯσοΫεɺN-Gramɺܗଶૉղੳ • Elasticsearchͱ͸ʁ • ͲΜͳ΋ͷ͔ • ΠϯσοΫεͷ෼ࢄͷ࢓૊Έ • ࢀߟจݙ

Slide 3

Slide 3 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited about • Me, Jun Ohtani / Technical Adovocate lucene-gosenίϛολʔ ElasticSearch Server೔ຊޠ൛ͷ຋༁ elasticsearch-extended-analysisͷ։ൃ http://blog.johtani.info • Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel  Professional services: Support & development subscriptions Trainings

Slide 4

Slide 4 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited શจݕࡧͱ͸ʁ

Slide 5

Slide 5 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited શจݕࡧͱ͸ʁ • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ ͓͍ͯɺෳ਺ͷจॻʢϑΝΠϧʣ͔Βಛఆͷจࣈ ྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ʮ୯Ұ ϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻ ʹ·͕ͨͬͯɺจॻʹؚ·ΕΔશจΛର৅ͱͨ͠ ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ  ʢWikipediaΑΓʣ

Slide 6

Slide 6 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ݕࡧ৚݅ɺݕࡧࣜ

Slide 7

Slide 7 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ༻ޠ • εΩʔϚ υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ

Slide 8

Slide 8 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥

Slide 9

Slide 9 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ

Slide 12

Slide 12 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch at MTU. ! • ೔ຊޠͷ৔߹ ࢲ͸MTUͰೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ

Slide 20

Slide 20 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch at MTU.     εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸MTUͰೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ    Ͳ͜Ͱ۠੾Ε͹Α͍ʁ

Slide 21

Slide 21 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ

Slide 22

Slide 22 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ܗଶૉղੳ ! ! • ϝϦοτɿ ҙຯͷ͋Δ୯ޠͷ੾Ε໨  ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ ग़ෆೳɻ ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ

Slide 23

Slide 23 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited N-Gram ! ! • ϝϦοτɿ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ΠϯσοΫεංେԽ ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

Slide 24

Slide 24 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearchͱ͸ʁ

Slide 25

Slide 25 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϑϦʔϫʔυݕࡧ

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ

Slide 33

Slide 33 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΠϯσοΫεͷߏ੒  ࿦ཧߏ੒ͱ෺ཧߏ੒

Slide 34

Slide 34 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΠϯσοΫεͷ࿦ཧߏ੒

Slide 35

Slide 35 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΠϯσοΫεͷ࿦ཧߏ੒ bookstore dvd book 1 4 1 2 2 2 • 1/2/3/…ɿυΩϡϝϯτ ݕࡧ͍ͨ͠σʔλʢྫɿ1࡭ͷຊͳͲʣ • dvd/bookɿλΠϓ υΩϡϝϯτͷ࿦ཧతͳू߹ɺΧςΰϦ  ʢྫɿbookʹຊͷू߹ʣ • bookstoreɿΠϯσοΫε ݕࡧ͍ͨ͠σʔλ

Slide 36

Slide 36 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XPUT localhost:9200/bookstore/book/1 -d ' { "title" : "ElasticSearch Server೔ຊޠ൛", "authors" : “Jun Ohtani", "started" : "2013-07-17", "pages" : 336 }' Create…

Slide 37

Slide 37 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XPUT localhost:9200/bookstore/book/1 -d ' { "title" : "ElasticSearch Server೔ຊޠ൛", "authors" : [ “Jun Ohtani", “Genta Kaneyama" ], "started" : "2013-07-17", "pages" : 336 }' Update…

Slide 38

Slide 38 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Delete… » curl -X DELETE localhost:9200/bookstore/book/1 Realtime GET… » curl —X GET localhost:9200/bookstore/book/1 » curl —X GET localhost:9200/bookstore/book/1/_source

Slide 39

Slide 39 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Search » curl -XGET localhost:9200/bookstore/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "ElasticSearch Server೔ຊޠ൛", "authors" : [ “Jun Ohtani", “Genta Kaneyama" ], "started" : “2013-07-17", "pages" :336 } } ] } }

Slide 40

Slide 40 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XGET ‘localhost:9200/booksstore/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0 ... } }' Search - Query DSL

Slide 41

Slide 41 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΠϯσοΫεͷ෺ཧߏ੒

Slide 42

Slide 42 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ෺ཧߏ੒ node1 bookstore otheridx 1 4 1 2 2 2 • 1/2/3/…ɿγϟʔυ(shard) ΠϯσοΫεΛ෺ཧతʹ෼ׂͯ͠อ࣋͢ ΔͨΊͷ࠷খ୯Ґɻ • bookstoreɿΠϯσοΫε ΠϯσοΫε͸γϟʔυͷू߹ • node1ɿϊʔυ Elasticsearchͷϓϩηε

Slide 43

Slide 43 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΠϯσοΫεͷ෺ཧߏ੒ • Ϋϥελͱϊʔυͱγϟʔυͷؔ܎ • Elasticsearchͷϊʔυू߹ʹΫϥελ • Ϋϥελ͸nݸͷϊʔυ͔Βߏ੒  ˞cluster.name͕ಉҰͳΒಉҰΫϥελͱΈͳ͢ • ΠϯσοΫε͸nݸͷγϟʔυ͔Βߏ੒͞Εɺ  γϟʔυ୯ҐͰϊʔυʹ഑ஔʹ෼ࢄ؅ཧ • γϟʔυ਺ͷܾఆ • ΠϯσοΫε࡞੒࣌ʹͷΈγϟʔυ਺ΛࢦఆՄೳ

Slide 44

Slide 44 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Ϋϥελ node 1 bookstore otheridx 2 1 4 1 node 2 bookstore otheridx 2 2 3 3 4 1 ϨϓϦΧ ϓϥΠϚϦ

Slide 45

Slide 45 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ϓϥΠϚϦͱϨϓϦΧͷؔ܎ • ϓϥΠϚϦγϟʔυͱϨϓϦΧγϟʔυ • γϟʔυ୯ҐͰϨϓϦΧʢෳ੡ʣΛอ࣋Մೳ • σʔλ͸ϓϥΠϚϦʹొ࿥ޙɺϨϓϦΧʹొ࿥ • ಉҰϊʔυ্ʹಉҰγϟʔυͷϓϥΠϚϦͱϨϓ ϦΧ͸ଘࡏ͠ͳ͍ • ϨϓϦΧ਺͸ಈతʹมߋՄೳ

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΠϯσοΫεͷొ࿥ͱݕࡧͷ ಺෦ಈ࡞

Slide 48

Slide 48 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • υΩϡϝϯτΛొ࿥ doc1 node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1

Slide 49

Slide 49 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • υΩϡϝϯτͷIDͷϋογϡ஋Λܭࢉͯ͠γϟʔ υ਺ͰׂΓɺొ࿥͢ΔγϟʔυΛܾΊΔ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 doc1

Slide 50

Slide 50 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • υΩϡϝϯτIDͷϋογϡ஋Λܭࢉɺγϟʔυ਺ ͰׂΓɺొ࿥͢ΔγϟʔυΛܾΊΔ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 doc1 hash(1)%4=1

Slide 51

Slide 51 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • γϟʔυ͕ܾ·ͬͨΒɺϓϥΠϚϦγϟʔυ͕͋ Δϊʔυ͕Ͳ͔͜Λௐ΂Δ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 doc1

Slide 52

Slide 52 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • ϓϥΠϚϦ͔ΒϨϓϦΧ΁ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 doc1

Slide 53

Slide 53 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ݕࡧΫΤϦΛ࣮ߦ Query node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1

Slide 54

Slide 54 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ϊʔυ͕̍ड͚औͬͨΫΤϦΛશγϟʔυʹૹ৴ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 Query

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ϊʔυ͕̍ड͚औͬͨΫΤϦΛશγϟʔυʹૹ৴ • γϟʔυຖʹݕࡧ࣮ߦ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 Query Query Query Query Query

Slide 57

Slide 57 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • શγϟʔυ͔Βͷ݁ՌΛϚʔδ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 Query ݁Ռ ݁Ռ ݁Ռ ݁Ռ

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ݁ՌΛૹ৴ ݁Ռ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1

Slide 60

Slide 60 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited σϞ

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ࢀߟจݙ • Github https://github.com/elasticsearch • ΨΠυ http://www.elasticsearch.org/guide/ • ॻ੶ ElasticSearchServer೔ຊޠ൛