$30 off During Our Annual Pro Sale. View Details »

Elasticsearch & Kibana

Jun Ohtani
September 04, 2014

Elasticsearch & Kibana

Introduction Elasticsearch and Kibana.

Jun Ohtani

September 04, 2014
Tweet

More Decks by Jun Ohtani

Other Decks in Technology

Transcript

  1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Jun Ohtani
    @johtani
    2014/Sep.
    Elasticsearch & Kibana

    View Slide

  2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ΞδΣϯμ
    • Elasticsearchͷ঺հ
    • ElasticsearchͷσϞ
    • KibanaͷσϞ

    View Slide

  3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    about
    • Me, Jun Ohtani / Technical Adovocate
    lucene-gosenίϛολʔ
    ElasticSearch Server೔ຊޠ൛ͷ຋༁
    elasticsearch-extended-analysisͷ։ൃ
    http://blog.johtani.info
    • Elasticsearch, founded in 2012
    Products: Elasticsearch, Logstash, Kibana, Marvel

    Professional services: Support & development subscriptions
    Trainings

    View Slide

  4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    શจݕࡧͱ͸ʁ

    View Slide

  5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    શจݕࡧͱ͸ʁ
    • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ
    ͓͍ͯɺෳ਺ͷจॻʢϑΝΠϧʣ͔Βಛఆͷจࣈ
    ྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ʮ୯Ұ
    ϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻ
    ʹ·͕ͨͬͯɺจॻʹؚ·ΕΔશจΛର৅ͱͨ͠
    ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ

    ʢWikipediaΑΓʣ

    View Slide

  6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ༻ޠ
    • ΠϯσοΫε
    ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ
    • υΩϡϝϯτʢจॻʣ
    ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ
    • ϑΟʔϧυ
    υΩϡϝϯτʹؚ·ΕΔଐੑ
    • ΫΤϦ
    ݕࡧ৚݅ɺݕࡧࣜ

    View Slide

  7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ༻ޠ
    • εΩʔϚ
    υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ
    • λʔϜʢTermʣɺτʔΫϯʢTokenʣ
    ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ
    จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ
    ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ

    View Slide

  8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    υΩϡϝϯτͷొ࿥

    View Slide

  9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ


    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ

    View Slide

  10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ


    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ
    ୯ޠ͔Βidͷ഑ྻ͕
    Ҿ͚ΔΑ͏ʹ

    View Slide

  11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ݕࡧ৚݅ೖྗ
    ΧπΦɹαβΤ

    View Slide

  12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ୯ޠͷ۠੾Γํ
    • ӳޠͷ৔߹
    I am speaking Introduction Elasticsearch.

    ɹ

    ɹ
    • ೔ຊޠͷ৔߹
    ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ


    View Slide

  20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ୯ޠͷ۠੾Γํ
    • ӳޠͷ৔߹
    I am speaking Introduction Elasticsearch.


    εϖʔε͕੾Ε໨ͱΘ͔Δ
    • ೔ຊޠͷ৔߹
    ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ


    Ͳ͜Ͱ۠੾Ε͹Α͍ʁ

    View Slide

  21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    N-Gramͱܗଶૉղੳ
    • సஔΠϯσοΫεͷΩʔͷ࡞Γํ
    ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ
    Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒
    • N-Gram
    NจࣈͣͭจষΛ۠੾Δ
    • ܗଶૉղੳ
    ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ

    View Slide

  22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ܗଶૉղੳ
    !
    !
    • ϝϦοτɿ
    ҙຯͷ͋Δ୯ޠͷ੾Ε໨

    ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ
    • σϝϦοτɿ
    ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ
    ग़ෆೳɻ
    ΧπΦ͸αβΤͷఋ
    ΧπΦ ͸ ͷ
    αβΤ ఋ

    View Slide

  23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    N-Gram
    !
    !
    • ϝϦοτɿ
    ະ஌ޠʹରԠՄೳ
    • σϝϦοτɿ
    ΠϯσοΫεංେԽ
    ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ
    ΧπΦ͸αβΤͷఋ
    Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

    View Slide

  24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearchͱ͸ʁ

    View Slide

  25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϑϦʔϫʔυݕࡧ

    View Slide

  26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ߜΓࠐΈ

    View Slide

  27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϋΠϥΠτ

    View Slide

  28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ιʔτ

    View Slide

  29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϖʔδϯά

    View Slide

  30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ूܭ

    View Slide

  31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    αδΣετ

    View Slide

  32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch in 10 seconds
    • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST
    & JSON
    • Φʔϓϯιʔε: Apache License 2.0
    • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ
    • JavaͰ࣮૷ɻ֦ு΋༰қ

    View Slide

  33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Πϯετʔϧ͔Βݕࡧ·Ͱ

    View Slide

  34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    μ΢ϯϩʔυͱىಈ
    $ wget https://download.elasticsearch.org/...
    $ tar -xf elasticsearch-1.3.2.tar.gz
    $ ./elasticsearch-1.3.2/bin/elasticsearch
    ...
    [2014-08-29 13:53:07,356][INFO ][node] [Araña] started
    ...
    Also puppet modules and RPM/DEB

    View Slide

  35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ىಈͷ֬ೝ
    » curl localhost:9200
    {
    "status" : 200,
    "name" : "Araña",
    "version" : {
    "number" : "1.3.2",
    "build_hash" : "dee175dbe2f254f3f26992f5d7591939aaefd12f",
    "build_timestamp" : "2014-08-13T14:29:30Z",
    "build_snapshot" : false,
    "lucene_version" : "4.9"
    },
    "tagline" : "You Know, for Search"
    }

    View Slide

  36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : "Clinton Gormley",
    "started" : "2013-02-04",
    "pages" : 230
    }'
    σʔλొ࿥

    View Slide

  37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : "2013-02-04",
    "pages" : 230
    }'
    σʔλߋ৽

    View Slide

  38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    σʔλ࡟আ
    » curl -X DELETE localhost:9200/books/book/1
    σʔλͷऔಘ
    » curl —X GET localhost:9200/books/book/1
    » curl —X GET localhost:9200/books/book/1/_source

    View Slide

  39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    » curl -XGET localhost:9200/books/_search?q=elasticsearch
    {
    "took" : 2, "timed_out" : false,
    "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
    "hits" : {
    "total" : 1, "max_score" : 0.076713204,
    "hits" : [ {
    "_index" : “books", "_type" : “book", "_id" : "1",
    "_score" : 0.076713204, "_source" : {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : “2013-02-04", "pages" : 230
    }
    } ]
    }
    }

    View Slide

  40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XGET ‘localhost:9200/books/book/_search' -d '{
    "query": {
    "filtered" : {
    "query" : {
    "match": {
    "text" : {
    "query" : “To Be Or Not To Be",
    "cutoff_frequency" : 0.01
    }
    }
    },
    "filter" : {
    "range": {
    "price": {
    "gte": 20.0
    "lte": 50.0
    ...
    }
    }'
    ݕࡧ - Query DSL

    View Slide

  41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ෼ࢄߏ੒ɺεέʔϧ

    View Slide

  42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Basic terms
    • ΠϯσοΫε
    σʔλͷ࿦ཧతͳू߹ɻ

    RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical
    • ϨϓϦέʔγϣϯ
    ಡΈࠐΈͷεέʔϥϏϦςΟ޲্
    SPOFͷղফ
    • γϟʔσΟϯά
    ෳ਺Ϛγϯ΁σʔλΛ෼ׂ

    ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্

    σʔλϑϩʔ੍ޚ

    View Slide

  43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    γϟʔυͱϨϓϦΧ
    node 1
    orders
    products
    1
    4
    1 2
    2
    2
    curl  -­‐X  PUT  localhost:9200/orders  -­‐d  '{  
       "settings.index.number_of_shards"  :  4  
       "settings.index.number_of_replicas"  :  1  
    }'
    curl  -­‐X  PUT  localhost:9200/products  -­‐d  '{  
       "settings.index.number_of_shards"  :  2  
       "settings.index.number_of_replicas"  :  0  
    }'

    View Slide

  44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    γϟʔυͱϨϓϦΧ
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    2
    3 4
    1 2
    3

    View Slide

  45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ࣗಈతͳ෼ࢄ
    node 1
    orders
    products
    2
    1
    4
    1
    node 2
    orders
    products
    2
    2
    node 3
    orders
    products
    3 4
    1
    3

    View Slide

  46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Ϋϥελ؅ཧ
    • ୯ҰϚελ
    Ϋϥελͷঢ়ଶʢϊʔυ৘ใɺϚοϐϯάʣΛ؅ཧ
    • ϚϧνΩϟετʹΑΔϊʔυ୳ࡧ

    ɹ (ϢχΩϟετ΋Մೳ)
    • ඞཁͳઃఆ
    Ϋϥελ໊

    ࠷খϚελϊʔυ਺
    • Tips: 3ͭͷϚελϊʔυΛ༻ҙ͠ɺσʔλΛอ࣋
    ͠ͳ͍

    View Slide

  47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ΫϥελɺϊʔυͷαΠδϯά
    • σʔλྔͱϦΫΤετʹґଘ
    υΩϡϝϯτ਺͸ʁϑΟʔϧυ਺͸ʁΫΤϦͷׂ߹͸ʁ

    Facet΍AggregationɺιʔτɺΧελϜείΞͷར༻͸ʁ

    ॻ͖ࠐΈͷׂ߹͸ʁυΩϡϝϯτ͸࡟আ/ߋ৽͢Δʁ

    ࣌ؒʹґଘͨ͠σʔλʁ
    • 1ϊʔυɺϨϓϦΧແ͠Ͱςετ
    γϟʔυαΠζɺJVMώʔϓ࢖༻ྔɺGCස౓ɺγϟʔυ͓Α
    ͼϊʔυ਺ɺγϟʔυ͋ͨΓͷυΩϡϝϯτ਺ɺCPU࢖༻
    ཰ɺσΟεΫ࢖༻཰ɺΠϯσοΫεͷύλʔϯ
    • Tips: 30 GB heap

    View Slide

  48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Ϋϥελ͓ΑͼΞϓϦͷߏ੒
    • ϩʔΧϧσΟεΫ; SAN OK
    • Tips: ϨΠςϯγͷߴ͍WAN্ͷΫϥελ͸ඇਪ
    ঑ɻCross-zone in EC2 is OK.
    !
    Your app
    ES Data 1 ES Data N
    ...
    ES Master; no data
    !
    Your app ...
    ES Master; no data
    ES Master; no data High Speed Network
    ES Node Client ES Node Client

    View Slide

  49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ΠϯσοΫεͷొ࿥ͱݕࡧͷ
    ಺෦ಈ࡞

    View Slide

  50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτొ࿥γʔέϯε
    • υΩϡϝϯτΛొ࿥
    doc1 node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1

    View Slide

  51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτొ࿥γʔέϯε
    • υΩϡϝϯτͷIDͷϋογϡ஋Λܭࢉͯ͠γϟʔ
    υ਺ͰׂΓɺొ࿥͢ΔγϟʔυΛܾΊΔ
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    doc1

    View Slide

  52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτొ࿥γʔέϯε
    • γϟʔυ͕ܾ·ͬͨΒɺϓϥΠϚϦγϟʔυ͕͋
    Δϊʔυ͕Ͳ͔͜Λௐ΂Δ
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    doc1

    View Slide

  53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτొ࿥γʔέϯε
    • ϓϥΠϚϦ͔ΒϨϓϦΧ΁
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    doc1

    View Slide

  54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • ݕࡧΫΤϦΛ࣮ߦ
    Query node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1

    View Slide

  55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • ϊʔυ͕̍ड͚औͬͨΫΤϦΛશγϟʔυʹૹ৴
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    Query

    View Slide

  56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • ϊʔυ͕̍ड͚औͬͨΫΤϦΛશγϟʔυʹૹ৴
    • γϟʔυຖʹݕࡧ࣮ߦ
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    Query
    Query
    Query
    Query
    Query

    View Slide

  57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • શγϟʔυ͔Βͷ݁ՌΛϚʔδ
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    Query
    ݁Ռ
    ݁Ռ
    ݁Ռ
    ݁Ռ

    View Slide

  58. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • શγϟʔυ͔Βͷ݁ՌΛϚʔδ
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    ݁Ռ

    View Slide

  59. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • ݁ՌΛૹ৴
    ݁Ռ node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1

    View Slide

  60. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysis & Mappings

    View Slide

  61. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysisͱ͸ʁ
    • సஔΠϯσοΫεͷ୯ޠʢTermʣΛυΩϡϝϯτ
    ͔Βநग़͢Δॲཧ
    • ϑΟʔϧυຖʹࢦఆ͞ΕͨΞφϥΠβ͕ॲཧ
    • ΞφϥΠβ͸CharFilterɺTokenizerɺTokenFilter͔
    Βߏ੒
    • ΠϯσοΫε࣌ɺݕࡧ࣌ʹ࣮ߦ

    View Slide

  62. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysisͷ֓ཁ
    Analysis
    υΩϡϝϯτ
    Term Id
    ΧπΦ 1
    αβΤ 1ɺ2
    ϫΧϝ 2
    … ...
    ΠϯσοΫε

    View Slide

  63. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysisͷ֓ཁ
    Analysis
    ݕࡧ

    Ωʔϫʔυ
    ΠϯσοΫε
    ΠϯσοΫε

    ݕࡧॲཧ
    ݕࡧ݁Ռ
    Term Id
    ΧπΦ 1
    αβΤ 1ɺ2
    ϫΧϝ 2
    … ...

    View Slide

  64. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analyzer
    Analysisͷߏ੒
    char_filter tokenizer
    char_filter
    token_

    filter
    token_

    filter
    Text Tokens

    View Slide

  65. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analyzerͷઃఆ
    {"index":{
    “analysis":{
    "analyzer" : {
    "my_analyzer" : {
    "type" : "custom",
    "tokenizer" : “kuromoji_tokenizer”,
    “char_filter" : [“char_filter1”,…],
    “filter" : [“token_filter1”,…]
    }
    }

    View Slide

  66. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Char Filter
    • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ
    !
    • ྫɿhtml_strip
    ೖྗɿ
    !
    ग़ྗɿ
    Elasticsearch is not a service of AWS
    Elasticsearch is not a service of AWS

    View Slide

  67. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Tokenizer
    • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ
    • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹ
    ґଘ
    • ྫɿstandard
    ೖྗɿ
    !
    ग़ྗɿ Elasticsearch is not a service of AWS
    Elasticsearch is not a service of AWS

    View Slide

  68. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Tokenizer
    • ྫɿkeyword
    ग़ྗɿ
    • ྫɿkuromoji_tokenizer
    ೖྗɿ
    !
    ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    Elasticsearch is not a service of AWS
    ण͕࢘ඒຯ͔ͬͨ͠

    View Slide

  69. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    TokenFilter
    • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ
    • ྫɿlowercase
    ೖྗɿ
    !
    ग़ྗɿ
    • ྫɿstop
    ೖྗɿ
    !
    ग़ྗɿ
    Elasticsearch is not a service of AWS
    elasticsearch is not a service of aws
    Elasticsearch is not a service of AWS
    Elasticsearch service AWS

    View Slide

  70. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    TokenFilter
    • ྫɿkuromoji_baseform
    ೖྗɿ
    !
    ग़ྗɿ
    • ྫɿkuromoji_readingform
    ೖྗɿ
    !
    ग़ྗɿ
    ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    ण࢘ ͕ ඒຯ͍͠ ͨ
    ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    sushi ga oishika ta

    View Slide

  71. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Mapping
    • ΠϯσοΫεͷߏ଄ʢschemaʣΛఆٛ
    • Schema less!͚ͩͲ…
    • ܗࣜΛࢦఆʢanalyze͢Δ/͠ͳ͍ɺ੔਺/গ਺ɺ೔
    ෇ͳͲʣ

    View Slide

  72. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿWikipediaͷMappingͷҰ෦
    "mappings": {
    "page": {
    "properties": {
    "link": {
    "type": "string",
    "fields": {
    "raw": {"type": "string",
    "index": "not_analyzed"} } },
    "text": { "type": "string"},
    "title": {
    "type": "string",
    "fields": {
    "raw": {"type": "string",
    "index": "not_analyzed"} }…

    View Slide

  73. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    aggregation

    View Slide

  74. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregationͱ͸
    • 1.0͔Βಋೖ
    • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ
    ֊૚తͳूܭɺάϧʔϓԽ

    ಈతͳूܭɺάϧʔϓԽ
    • େ͖͘2छྨ
    BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά
    Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ

    View Slide

  75. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿݴޠ͝ͱͷϑΝηοτ
    » curl -XGET twitter-2014.08.22/_search -d '
    GET twitter-2014.08.22/_search
    {
    "facets": {
    "lang_facet": {
    "terms": {"field": "lang"}
    }
    }
    }

    View Slide

  76. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿݴޠ͝ͱͷϑΝηοτ
    "facets": {
    "lang_facet": {
    "_type": "terms",
    "missing": 0,
    "total": 4521075,
    "other": 320228,
    "terms": [
    {
    "term": "en",
    "count": 1471819
    },
    {
    "term": "ja",
    "count": 980145
    },

    View Slide

  77. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿݴޠ͓Αͼ஍Ҭͷूܭ
    » curl -XGET twitter-2014.08.22/_search -d '
    {
    "aggs": {
    "lang": {
    "terms": {"field": "lang" },
    "aggs": {
    "place": {
    "terms": {
    "field": “place.full_name", "size": 10
    }
    }
    }
    }
    }
    }

    View Slide

  78. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿݴޠ͓Αͼ஍Ҭͷूܭ
    "aggregations": {
    "lang": {
    "buckets": [{…}, {
    "key": "ja",
    "doc_count": 980145,
    "place": {
    "buckets": [
    { "key": "ژ౎ࢢ෬ݟ۠, ژ౎",
    "doc_count":252 },
    { "key": "ઍ୅ా۠, ౦ژ",
    "doc_count": 39 },…

    View Slide

  79. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    bucket
    • filter
    • filters (>=1.4.0)
    • missing
    • nested
    • reverse nested
    (>=1.2.0)
    • children
    • terms
    • significant terms
    (>=1.1.0)
    • range
    • date range
    • ipv4 range
    • histogram
    • date histogram
    • geo distance
    • geohash grid

    View Slide

  80. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    metric
    • min
    • max
    • sum
    • avg
    • stats
    • extended_stats
    • value count
    • percentiles (>=1.1.0)
    • percentile ranks (>=1.3)
    • cardinality (>=1.1.0)
    • geo bounds (>=1.3.0)
    • top_hits (>=1.3.0)
    • scripted metric 

    (>= 1.4.0)

    View Slide

  81. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ͦͷଞͷػೳ

    View Slide

  82. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Geo
    • ͞·͟·ͳܗࣜͷσʔλͰGeoݕࡧՄೳ
    • σʔλɿ
    Ң౓ܦ౓ɺGeoHashɺGeoShape
    • ݕࡧɿ
    geo shape queryɺgeo shape filterɺgeo distance filterɺ
    geo distance range filterɺgeo bounding box filterɺ
    geo polygon filter

    View Slide

  83. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿGeoσʔλొ࿥
    ## Ң౓ܦ౓
    {
    "pin" : {
    "location" : "41.12,-71.34"
    }
    }
    !
    ## GeoHash
    {
    "pin" : {
    "location" : "drm3btev3e86"
    }
    }

    View Slide

  84. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿGeoσʔλొ࿥
    ## GeoShape(Polygon)
    {
    "location" : {
    "type" : "polygon",
    "coordinates" : [[ [100.0, 0.0], [101.0, 0.0],
    [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ]]
    }
    }
    ## GeoShape(Circle)
    {
    "location" : {
    "type" : "circle",
    "coordinates" : [-45.0, 45.0],
    "radius" : "100m"
    }
    }

    View Slide

  85. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿGeoσʔλݕࡧ
    {
    "query": {
    "geo_shape": {
    "location": {
    "shape": {
    "type": "envelope",
    "coordinates": [[13, 53],[14, 52]]
    }
    }
    }
    }
    }

    View Slide

  86. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿڑ཭ʹΑΔϑΟϧλ
    {
    "filtered" : {
    "query" : {"match_all" : {}},
    "filter" : {
    "geo_distance" : {
    "distance" : "200km",
    "pin.location" : {
    "lat" : 40,
    "lon" : -70
    }
    }
    }
    }
    }

    View Slide

  87. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Percolator
    • υΩϡϝϯτͰ͸ͳ͘ΫΤϦΛొ࿥
    • _percolate APIʹυΩϡϝϯτΛૹ৴
    • ొ࿥͞ΕͨΫΤϦʹϚονͨ͠৔߹ɺ

    Ϛονͨ͠ΫΤϦΛฦ͢

    View Slide

  88. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿΫΤϦొ࿥
    curl -XPUT 'localhost:9200/my-index/.percolator/1' -d
    '{
    "query" : {
    "match" : {
    "message" : "bonsai tree"
    }
    }
    }'

    View Slide

  89. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿσʔλνΣοΫ
    curl -XGET 'localhost:9200/my-index/message/_percolate'
    -d '{
    "doc" : {
    "message" : "A new bonsai tree in the office"
    }
    }'

    View Slide

  90. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿ݁Ռ
    {
    "took" : 19,
    "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
    },
    "total" : 1,
    "matches" : [
    {
    "_index" : "my-index",
    "_id" : "1"
    }
    ]
    }

    View Slide

  91. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Snapshot/Restore
    • 1.0͔Βಋೖ
    • ΠϯσοΫε୯ҐͰSnapshot/RestoreՄೳ
    • อଘઌʢRepositoryʣ
    Shard FSɺS3ɺHDFSɺAzure…

    View Slide

  92. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿϦϙδτϦ࡞੒
    $ curl -XPUT 'http://localhost:9200/_snapshot/
    my_backup' -d '{
    "type": "fs",
    "settings": {
    "location": "/mount/backups/my_backup",
    "compress": true
    }
    }'

    View Slide

  93. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿSnapshotͱϦετΞ
    ## Snapshot

    $ curl -XPUT “localhost:9200/_snapshot/my_backup/
    snapshot_1?wait_for_completion=true"
    !
    !
    ## Restore
    $ curl -XPOST "localhost:9200/_snapshot/my_backup/
    snapshot_1/_restore"

    View Slide

  94. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Ecosystem
    • Plugins
    ϓϥάΠϯʹΑΔػೳͷ௥Ճ
    • ΫϥΠΞϯτϥΠϒϥϦ
    Ruby, python, php, perl, javascript, .NET
    Scala, clojure, go
    • Kibana
    • Logstash

    View Slide

  95. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Hadoop࿈ܞ&Kibana
    • ʮαʔόʗΠϯϑϥΤϯδχΞཆ੒ಡຊ ϩάऩ
    ूʙՄࢹԽฤʯൃചத
    • Hadoop࿈ܞͷ঺հ
    • Kibanaͷৄࡉ

    View Slide

  96. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited

    View Slide

  97. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Tools for sys admins

    View Slide

  98. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    REST-based management
    • Elasticsearch is full of monitoring APIs
    Everything is returned as JSON
    • Humans are not the world’s best JSON parsers
    • What if elasticsearch had an easy to use interface
    from the commandline?

    View Slide

  99. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Which node is the master?
    $ curl "localhost:9200/_cluster/state?pretty&filter_metadata=true&!
    filter_routing_table=true"!
    {!
    "cluster_name" : "elasticsearch",!
    "master_node" : "GNf0hEXlTfaBvQXKBF300A",!
    "blocks" : { },!
    "nodes" : {!
    "ObdRqLHGQ6CMI5rOEstA5A" : {!
    "name" : "Triton",!
    "transport_address" : “inet[/10.0.1.11:9300]”,!
    "attributes" : { }!
    },!
    "4C7pKbfhTvu0slcSy_G4_w" : {!
    "name" : "Kid Colt",!
    "transport_address" : "inet[/10.0.1.12:9300]",!
    "attributes" : { }!
    },!
    "GNf0hEXlTfaBvQXKBF300A" : {!
    "name" : "Lang, Steven",!
    "transport_address" : "inet[/10.0.1.13:9300]",!
    "attributes" : { }!
    }!
    }!
    }

    View Slide

  100. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Which one is the master? (v1.0)
    $ curl localhost:9200/_cat/master
    GNf0hEXlTfaBvQXKBF300A 10.0.1.13 Lang, Steven

    View Slide

  101. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    _cat/* api
    • /_cat/allocation
    • /_cat/count
    • /_cat/health
    • /_cat/master
    • /_cat/aliases
    • /_cat/nodes
    • /_cat/recovery
    • /_cat/shards
    • /_cat/indices
    • /_cat/thread_pool

    View Slide

  102. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Monitor your cluster with Marvel
    • Point in time views are a start
    • Marvel shows historical trends
    • Visualize cluster behavior, act before problems
    !
    • Free for development, $500/year for up to 5
    nodes

    View Slide

  103. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Overview

    View Slide

  104. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Node statistics

    View Slide

  105. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Index statistics

    View Slide

  106. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Cluster Pulse

    View Slide

  107. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sense

    View Slide

  108. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    σϞ for Elasticsearch

    ೔ຊޠWikipedia

    View Slide

  109. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    σϞ for Elasticsearch

    Twitter Stream

    View Slide

  110. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    σϞ for Kibana

    Twitter Stream

    View Slide

  111. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    σϞ for Kibana

    Git log

    View Slide

  112. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ࢀߟจݙ
    • Elasticsearch - The Definitive guide
    http://www.elasticsearch.com/guide/en/elasticsearch/guide/
    current/index.html
    • ॻ੶
    ElasticSearchServer೔ຊޠ൛

    αʔό/ΠϯϑϥΤϯδχΞ

    ɹཆ੒ಡຊɹϩάऩू

    View Slide

  113. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Q&A
    • Training
    http://training.elasticsearch.com
    • Support
    http://www.elasticsearch.com/support/
    • Github
    https://github.com/elasticsearch

    View Slide