Elasticsearch & Kibana - Speaker Deck

Slide 1

Slide 1 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Jun Ohtani @johtani 2014/Sep. Elasticsearch & Kibana

Slide 2

Slide 2 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΞδΣϯμ • Elasticsearchͷ঺հ • ElasticsearchͷσϞ • KibanaͷσϞ

Slide 3

Slide 3 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited about • Me, Jun Ohtani / Technical Adovocate lucene-gosenίϛολʔ ElasticSearch Server೔ຊޠ൛ͷ຋༁ elasticsearch-extended-analysisͷ։ൃ http://blog.johtani.info • Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel  Professional services: Support & development subscriptions Trainings

Slide 4

Slide 4 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited શจݕࡧͱ͸ʁ

Slide 5

Slide 5 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited શจݕࡧͱ͸ʁ • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ ͓͍ͯɺෳ਺ͷจॻʢϑΝΠϧʣ͔Βಛఆͷจࣈ ྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ʮ୯Ұ ϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻ ʹ·͕ͨͬͯɺจॻʹؚ·ΕΔશจΛର৅ͱͨ͠ ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ  ʢWikipediaΑΓʣ

Slide 6

Slide 6 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ݕࡧ৚݅ɺݕࡧࣜ

Slide 7

Slide 7 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ༻ޠ • εΩʔϚ υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ

Slide 8

Slide 8 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥

Slide 9

Slide 9 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ

Slide 12

Slide 12 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch.   ɹ  ɹ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ   

Slide 20

Slide 20 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch.     εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ    Ͳ͜Ͱ۠੾Ε͹Α͍ʁ

Slide 21

Slide 21 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ

Slide 22

Slide 22 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ܗଶૉղੳ ! ! • ϝϦοτɿ ҙຯͷ͋Δ୯ޠͷ੾Ε໨  ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ ग़ෆೳɻ ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ

Slide 23

Slide 23 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited N-Gram ! ! • ϝϦοτɿ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ΠϯσοΫεංେԽ ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

Slide 24

Slide 24 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearchͱ͸ʁ

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ

Slide 33

Slide 33 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Πϯετʔϧ͔Βݕࡧ·Ͱ

Slide 34

Slide 34 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited μ΢ϯϩʔυͱىಈ $ wget https://download.elasticsearch.org/... $ tar -xf elasticsearch-1.3.2.tar.gz $ ./elasticsearch-1.3.2/bin/elasticsearch ... [2014-08-29 13:53:07,356][INFO ][node] [Araña] started ... Also puppet modules and RPM/DEB

Slide 35

Slide 35 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ىಈͷ֬ೝ » curl localhost:9200 { "status" : 200, "name" : "Araña", "version" : { "number" : "1.3.2", "build_hash" : "dee175dbe2f254f3f26992f5d7591939aaefd12f", "build_timestamp" : "2014-08-13T14:29:30Z", "build_snapshot" : false, "lucene_version" : "4.9" }, "tagline" : "You Know, for Search" }

Slide 36

Slide 36 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }' σʔλొ࿥

Slide 37

Slide 37 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }' σʔλߋ৽

Slide 38

Slide 38 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited σʔλ࡟আ » curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ » curl —X GET localhost:9200/books/book/1 » curl —X GET localhost:9200/books/book/1/_source

Slide 39

Slide 39 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ » curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ] } }

Slide 40

Slide 40 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XGET ‘localhost:9200/books/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0 ... } }' ݕࡧ - Query DSL

Slide 41

Slide 41 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ෼ࢄߏ੒ɺεέʔϧ

Slide 42

Slide 42 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Basic terms • ΠϯσοΫε σʔλͷ࿦ཧతͳू߹ɻ  RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ SPOFͷղফ • γϟʔσΟϯά ෳ਺Ϛγϯ΁σʔλΛ෼ׂ  ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্  σʔλϑϩʔ੍ޚ

Slide 43

Slide 43 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited γϟʔυͱϨϓϦΧ node 1 orders products 1 4 1 2 2 2 curl -‐X PUT localhost:9200/orders -‐d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -‐X PUT localhost:9200/products -‐d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'

Slide 44

Slide 44 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited γϟʔυͱϨϓϦΧ node 1 orders products 1 4 1 node 2 orders products 2 2 3 4 1 2 3

Slide 45

Slide 45 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ࣗಈతͳ෼ࢄ node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3

Slide 46

Slide 46 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Ϋϥελ؅ཧ • ୯ҰϚελ Ϋϥελͷঢ়ଶʢϊʔυ৘ใɺϚοϐϯάʣΛ؅ཧ • ϚϧνΩϟετʹΑΔϊʔυ୳ࡧ  ɹ (ϢχΩϟετ΋Մೳ) • ඞཁͳઃఆ Ϋϥελ໊  ࠷খϚελϊʔυ਺ • Tips: 3ͭͷϚελϊʔυΛ༻ҙ͠ɺσʔλΛอ࣋ ͠ͳ͍

Slide 47

Slide 47 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΫϥελɺϊʔυͷαΠδϯά • σʔλྔͱϦΫΤετʹґଘ υΩϡϝϯτ਺͸ʁϑΟʔϧυ਺͸ʁΫΤϦͷׂ߹͸ʁ  Facet΍AggregationɺιʔτɺΧελϜείΞͷར༻͸ʁ  ॻ͖ࠐΈͷׂ߹͸ʁυΩϡϝϯτ͸࡟আ/ߋ৽͢Δʁ  ࣌ؒʹґଘͨ͠σʔλʁ • 1ϊʔυɺϨϓϦΧແ͠Ͱςετ γϟʔυαΠζɺJVMώʔϓ࢖༻ྔɺGCස౓ɺγϟʔυ͓Α ͼϊʔυ਺ɺγϟʔυ͋ͨΓͷυΩϡϝϯτ਺ɺCPU࢖༻ ཰ɺσΟεΫ࢖༻཰ɺΠϯσοΫεͷύλʔϯ • Tips: 30 GB heap

Slide 48

Slide 48 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Ϋϥελ͓ΑͼΞϓϦͷߏ੒ • ϩʔΧϧσΟεΫ; SAN OK • Tips: ϨΠςϯγͷߴ͍WAN্ͷΫϥελ͸ඇਪ ঑ɻCross-zone in EC2 is OK. ! Your app ES Data 1 ES Data N ... ES Master; no data ! Your app ... ES Master; no data ES Master; no data High Speed Network ES Node Client ES Node Client

Slide 49

Slide 49 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΠϯσοΫεͷొ࿥ͱݕࡧͷ ಺෦ಈ࡞

Slide 50

Slide 50 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • υΩϡϝϯτΛొ࿥ doc1 node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1

Slide 51

Slide 51 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • υΩϡϝϯτͷIDͷϋογϡ஋Λܭࢉͯ͠γϟʔ υ਺ͰׂΓɺొ࿥͢ΔγϟʔυΛܾΊΔ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 doc1

Slide 52

Slide 52 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • γϟʔυ͕ܾ·ͬͨΒɺϓϥΠϚϦγϟʔυ͕͋ Δϊʔυ͕Ͳ͔͜Λௐ΂Δ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 doc1

Slide 53

Slide 53 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • ϓϥΠϚϦ͔ΒϨϓϦΧ΁ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 doc1

Slide 54

Slide 54 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ݕࡧΫΤϦΛ࣮ߦ Query node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1

Slide 55

Slide 55 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ϊʔυ͕̍ड͚औͬͨΫΤϦΛશγϟʔυʹૹ৴ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 Query

Slide 56

Slide 56 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ϊʔυ͕̍ड͚औͬͨΫΤϦΛશγϟʔυʹૹ৴ • γϟʔυຖʹݕࡧ࣮ߦ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 Query Query Query Query Query

Slide 57

Slide 57 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • શγϟʔυ͔Βͷ݁ՌΛϚʔδ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 Query ݁Ռ ݁Ռ ݁Ռ ݁Ռ

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ݁ՌΛૹ৴ ݁Ռ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1

Slide 60

Slide 60 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analysis & Mappings

Slide 61

Slide 61 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analysisͱ͸ʁ • సஔΠϯσοΫεͷ୯ޠʢTermʣΛυΩϡϝϯτ ͔Βநग़͢Δॲཧ • ϑΟʔϧυຖʹࢦఆ͞ΕͨΞφϥΠβ͕ॲཧ • ΞφϥΠβ͸CharFilterɺTokenizerɺTokenFilter͔ Βߏ੒ • ΠϯσοΫε࣌ɺݕࡧ࣌ʹ࣮ߦ

Slide 62

Slide 62 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analysisͷ֓ཁ Analysis υΩϡϝϯτ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ... ΠϯσοΫε

Slide 63

Slide 63 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analysisͷ֓ཁ Analysis ݕࡧ  Ωʔϫʔυ ΠϯσοΫε ΠϯσοΫε  ݕࡧॲཧ ݕࡧ݁Ռ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ...

Slide 64

Slide 64 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analyzer Analysisͷߏ੒ char_filter tokenizer char_filter token_  filter token_  filter Text Tokens

Slide 65

Slide 65 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analyzerͷઃఆ {"index":{ “analysis":{ "analyzer" : { "my_analyzer" : { "type" : "custom", "tokenizer" : “kuromoji_tokenizer”, “char_filter" : [“char_filter1”,…], “filter" : [“token_filter1”,…] } }

Slide 66

Slide 66 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Char Filter • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ ! • ྫɿhtml_strip ೖྗɿ ! ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS

Slide 67

Slide 67 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Tokenizer • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹ ґଘ • ྫɿstandard ೖྗɿ ! ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS

Slide 68

Slide 68 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Tokenizer • ྫɿkeyword ग़ྗɿ • ྫɿkuromoji_tokenizer ೖྗɿ ! ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ Elasticsearch is not a service of AWS ण͕࢘ඒຯ͔ͬͨ͠

Slide 69

Slide 69 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited TokenFilter • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ • ྫɿlowercase ೖྗɿ ! ग़ྗɿ • ྫɿstop ೖྗɿ ! ग़ྗɿ Elasticsearch is not a service of AWS elasticsearch is not a service of aws Elasticsearch is not a service of AWS Elasticsearch service AWS

Slide 70

Slide 70 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited TokenFilter • ྫɿkuromoji_baseform ೖྗɿ ! ग़ྗɿ • ྫɿkuromoji_readingform ೖྗɿ ! ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ ण࢘ ͕ ඒຯ͍͠ ͨ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ sushi ga oishika ta

Slide 71

Slide 71 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Mapping • ΠϯσοΫεͷߏ଄ʢschemaʣΛఆٛ • Schema less!͚ͩͲ… • ܗࣜΛࢦఆʢanalyze͢Δ/͠ͳ͍ɺ੔਺/গ਺ɺ೔ ෇ͳͲʣ

Slide 72

Slide 72 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿWikipediaͷMappingͷҰ෦ "mappings": { "page": { "properties": { "link": { "type": "string", "fields": { "raw": {"type": "string", "index": "not_analyzed"} } }, "text": { "type": "string"}, "title": { "type": "string", "fields": { "raw": {"type": "string", "index": "not_analyzed"} }…

Slide 73

Slide 73 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited aggregation

Slide 74

Slide 74 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ ֊૚తͳूܭɺάϧʔϓԽ  ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ

Slide 75

Slide 75 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿݴޠ͝ͱͷϑΝηοτ » curl -XGET twitter-2014.08.22/_search -d ' GET twitter-2014.08.22/_search { "facets": { "lang_facet": { "terms": {"field": "lang"} } } }

Slide 76

Slide 76 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿݴޠ͝ͱͷϑΝηοτ "facets": { "lang_facet": { "_type": "terms", "missing": 0, "total": 4521075, "other": 320228, "terms": [ { "term": "en", "count": 1471819 }, { "term": "ja", "count": 980145 },

Slide 77

Slide 77 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿݴޠ͓Αͼ஍Ҭͷूܭ » curl -XGET twitter-2014.08.22/_search -d ' { "aggs": { "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }

Slide 78

Slide 78 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿݴޠ͓Αͼ஍Ҭͷूܭ "aggregations": { "lang": { "buckets": [{…}, { "key": "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…

Slide 79

Slide 79 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited bucket • filter • filters (>=1.4.0) • missing • nested • reverse nested (>=1.2.0) • children • terms • significant terms (>=1.1.0) • range • date range • ipv4 range • histogram • date histogram • geo distance • geohash grid

Slide 80

Slide 80 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited metric • min • max • sum • avg • stats • extended_stats • value count • percentiles (>=1.1.0) • percentile ranks (>=1.3) • cardinality (>=1.1.0) • geo bounds (>=1.3.0) • top_hits (>=1.3.0) • scripted metric   (>= 1.4.0)

Slide 81

Slide 81 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ͦͷଞͷػೳ

Slide 82

Slide 82 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Geo • ͞·͟·ͳܗࣜͷσʔλͰGeoݕࡧՄೳ • σʔλɿ Ң౓ܦ౓ɺGeoHashɺGeoShape • ݕࡧɿ geo shape queryɺgeo shape filterɺgeo distance filterɺ geo distance range filterɺgeo bounding box filterɺ geo polygon filter

Slide 83

Slide 83 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿGeoσʔλొ࿥ ## Ң౓ܦ౓ { "pin" : { "location" : "41.12,-71.34" } } ! ## GeoHash { "pin" : { "location" : "drm3btev3e86" } }

Slide 84

Slide 84 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿGeoσʔλొ࿥ ## GeoShape(Polygon) { "location" : { "type" : "polygon", "coordinates" : [[ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ]] } } ## GeoShape(Circle) { "location" : { "type" : "circle", "coordinates" : [-45.0, 45.0], "radius" : "100m" } }

Slide 85

Slide 85 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿGeoσʔλݕࡧ { "query": { "geo_shape": { "location": { "shape": { "type": "envelope", "coordinates": [[13, 53],[14, 52]] } } } } }

Slide 86

Slide 86 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿڑ཭ʹΑΔϑΟϧλ { "filtered" : { "query" : {"match_all" : {}}, "filter" : { "geo_distance" : { "distance" : "200km", "pin.location" : { "lat" : 40, "lon" : -70 } } } } }

Slide 87

Slide 87 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Percolator • υΩϡϝϯτͰ͸ͳ͘ΫΤϦΛొ࿥ • _percolate APIʹυΩϡϝϯτΛૹ৴ • ొ࿥͞ΕͨΫΤϦʹϚονͨ͠৔߹ɺ  Ϛονͨ͠ΫΤϦΛฦ͢

Slide 88

Slide 88 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿΫΤϦొ࿥ curl -XPUT 'localhost:9200/my-index/.percolator/1' -d '{ "query" : { "match" : { "message" : "bonsai tree" } } }'

Slide 89

Slide 89 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿσʔλνΣοΫ curl -XGET 'localhost:9200/my-index/message/_percolate' -d '{ "doc" : { "message" : "A new bonsai tree in the office" } }'

Slide 90

Slide 90 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿ݁Ռ { "took" : 19, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "total" : 1, "matches" : [ { "_index" : "my-index", "_id" : "1" } ] }

Slide 91

Slide 91 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot/Restore • 1.0͔Βಋೖ • ΠϯσοΫε୯ҐͰSnapshot/RestoreՄೳ • อଘઌʢRepositoryʣ Shard FSɺS3ɺHDFSɺAzure…

Slide 92

Slide 92 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿϦϙδτϦ࡞੒ $ curl -XPUT 'http://localhost:9200/_snapshot/ my_backup' -d '{ "type": "fs", "settings": { "location": "/mount/backups/my_backup", "compress": true } }'

Slide 93

Slide 93 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿSnapshotͱϦετΞ ## Snapshot  $ curl -XPUT “localhost:9200/_snapshot/my_backup/ snapshot_1?wait_for_completion=true" ! ! ## Restore $ curl -XPOST "localhost:9200/_snapshot/my_backup/ snapshot_1/_restore"

Slide 94

Slide 94 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Ecosystem • Plugins ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ Ruby, python, php, perl, javascript, .NET Scala, clojure, go • Kibana • Logstash

Slide 95

Slide 95 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Hadoop࿈ܞ&Kibana • ʮαʔόʗΠϯϑϥΤϯδχΞཆ੒ಡຊ ϩάऩ ूʙՄࢹԽฤʯൃചத • Hadoop࿈ܞͷ঺հ • Kibanaͷৄࡉ

Slide 96

Slide 96 text

Slide 97

Slide 97 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Tools for sys admins

Slide 98

Slide 98 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited REST-based management • Elasticsearch is full of monitoring APIs Everything is returned as JSON • Humans are not the world’s best JSON parsers • What if elasticsearch had an easy to use interface from the commandline?

Slide 99

Slide 99 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Which node is the master? $ curl "localhost:9200/_cluster/state?pretty&filter_metadata=true&! filter_routing_table=true"! {! "cluster_name" : "elasticsearch",! "master_node" : "GNf0hEXlTfaBvQXKBF300A",! "blocks" : { },! "nodes" : {! "ObdRqLHGQ6CMI5rOEstA5A" : {! "name" : "Triton",! "transport_address" : “inet[/10.0.1.11:9300]”,! "attributes" : { }! },! "4C7pKbfhTvu0slcSy_G4_w" : {! "name" : "Kid Colt",! "transport_address" : "inet[/10.0.1.12:9300]",! "attributes" : { }! },! "GNf0hEXlTfaBvQXKBF300A" : {! "name" : "Lang, Steven",! "transport_address" : "inet[/10.0.1.13:9300]",! "attributes" : { }! }! }! }

Slide 100

Slide 100 text

Slide 101

Slide 101 text

Slide 102

Slide 102 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Monitor your cluster with Marvel • Point in time views are a start • Marvel shows historical trends • Visualize cluster behavior, act before problems ! • Free for development, $500/year for up to 5 nodes

Slide 103

Slide 103 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Overview

Slide 104

Slide 104 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Node statistics

Slide 105

Slide 105 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Index statistics

Slide 106

Slide 106 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Cluster Pulse

Slide 107

Slide 107 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Sense

Slide 108

Slide 108 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited σϞ for Elasticsearch  ೔ຊޠWikipedia

Slide 109

Slide 109 text

Slide 110

Slide 110 text

Slide 111

Slide 111 text

Slide 112

Slide 112 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ࢀߟจݙ • Elasticsearch - The Definitive guide http://www.elasticsearch.com/guide/en/elasticsearch/guide/ current/index.html • ॻ੶ ElasticSearchServer೔ຊޠ൛  αʔό/ΠϯϑϥΤϯδχΞ  ɹཆ੒ಡຊɹϩάऩू

Slide 113

Slide 113 text

Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Q&A • Training http://training.elasticsearch.com • Support http://www.elasticsearch.com/support/ • Github https://github.com/elasticsearch