Elasticsearch & Kibana

Introduction Elasticsearch and Kibana.

Transcript

1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   Jun Ohtani
   @johtani
   2014/Sep.
   Elasticsearch & Kibana
   

   View Slide

2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   ΞδΣϯμ
   • Elasticsearchͷ঺հ
   • ElasticsearchͷσϞ
   • KibanaͷσϞ
   

   View Slide

3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   about
   • Me, Jun Ohtani / Technical Adovocate
   lucene-gosenίϛολʔ
   ElasticSearch Server೔ຊޠ൛ͷ຋༁
   elasticsearch-extended-analysisͷ։ൃ
   http://blog.johtani.info
   • Elasticsearch, founded in 2012
   Products: Elasticsearch, Logstash, Kibana, Marvel

   Professional services: Support & development subscriptions
   Trainings
   

   View Slide

4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   શจݕࡧͱ͸ʁ
   

   View Slide

5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   શจݕࡧͱ͸ʁ
   • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ
   ͓͍ͯɺෳ਺ͷจॻʢϑΝΠϧʣ͔Βಛఆͷจࣈ
   ྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ʮ୯Ұ
   ϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻ
   ʹ·͕ͨͬͯɺจॻʹؚ·ΕΔશจΛର৅ͱͨ͠
   ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ

   ʢWikipediaΑΓʣ
   

   View Slide

6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   ༻ޠ
   • ΠϯσοΫε
   ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ
   • υΩϡϝϯτʢจॻʣ
   ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ
   • ϑΟʔϧυ
   υΩϡϝϯτʹؚ·ΕΔଐੑ
   • ΫΤϦ
   ݕࡧ৚݅ɺݕࡧࣜ
   

   View Slide

7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   ༻ޠ
   • εΩʔϚ
   υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ
   • λʔϜʢTermʣɺτʔΫϯʢTokenʣ
   ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ
   จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ
   ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ
   

   View Slide

8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   υΩϡϝϯτͷొ࿥
   1
   2
   ΧπΦ͸αβΤͷఋ
   αβΤ͸ϫΧϝͷ࢞
   υΩϡϝϯτͷొ࿥
   

   View Slide

9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   υΩϡϝϯτͷొ࿥
   1
   2
   ΧπΦ͸αβΤͷఋ
   αβΤ͸ϫΧϝͷ࢞
   1
   2
   ΧπΦ
   αβΤ
   ͸
   ͸
   ͷ
   ͷ
   αβΤ
   ϫΧϝ
   ఋ
   ࢞
   υΩϡϝϯτͷొ࿥
   ୯ޠʹ෼ׂ
   

   View Slide

10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ
    ఋ
    ࢞
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ
    ୯ޠ͔Βidͷ഑ྻ͕
    Ҿ͚ΔΑ͏ʹ
    

    View Slide

11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ݕࡧ৚݅ೖྗ
    ΧπΦɹαβΤ
    

    View Slide

12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ୯ޠͷ۠੾Γํ
    • ӳޠͷ৔߹
    I am speaking Introduction Elasticsearch.
    

    ɹ

    ɹ
    • ೔ຊޠͷ৔߹
    ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ

    

    

    View Slide

20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ୯ޠͷ۠੾Γํ
    • ӳޠͷ৔߹
    I am speaking Introduction Elasticsearch.
    

    

    εϖʔε͕੾Ε໨ͱΘ͔Δ
    • ೔ຊޠͷ৔߹
    ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ

    

    Ͳ͜Ͱ۠੾Ε͹Α͍ʁ
    

    View Slide

21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    N-Gramͱܗଶૉղੳ
    • సஔΠϯσοΫεͷΩʔͷ࡞Γํ
    ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ
    Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒
    • N-Gram
    NจࣈͣͭจষΛ۠੾Δ
    • ܗଶૉղੳ
    ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ
    

    View Slide

22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ܗଶૉղੳ
    !
    !
    • ϝϦοτɿ
    ҙຯͷ͋Δ୯ޠͷ੾Ε໨

    ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ
    • σϝϦοτɿ
    ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ
    ग़ෆೳɻ
    ΧπΦ͸αβΤͷఋ
    ΧπΦ ͸ ͷ
    αβΤ ఋ
    

    View Slide

23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    N-Gram
    !
    !
    • ϝϦοτɿ
    ະ஌ޠʹରԠՄೳ
    • σϝϦοτɿ
    ΠϯσοΫεංେԽ
    ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ
    ΧπΦ͸αβΤͷఋ
    Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
    

    View Slide

24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearchͱ͸ʁ
    

    View Slide

25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϑϦʔϫʔυݕࡧ
    

    View Slide

26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ߜΓࠐΈ
    

    View Slide

27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϋΠϥΠτ
    

    View Slide

28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ιʔτ
    

    View Slide

29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ϖʔδϯά
    

    View Slide

30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    ूܭ
    

    View Slide

31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    αδΣετ
    

    View Slide

32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Elasticsearch in 10 seconds
    • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST
    & JSON
    • Φʔϓϯιʔε: Apache License 2.0
    • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ
    • JavaͰ࣮૷ɻ֦ு΋༰қ
    

    View Slide

33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Πϯετʔϧ͔Βݕࡧ·Ͱ
    

    View Slide

34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    μ΢ϯϩʔυͱىಈ
    $ wget https://download.elasticsearch.org/...
    $ tar -xf elasticsearch-1.3.2.tar.gz
    $ ./elasticsearch-1.3.2/bin/elasticsearch
    ...
    [2014-08-29 13:53:07,356][INFO ][node] [Araña] started
    ...
    Also puppet modules and RPM/DEB
    

    View Slide

35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ىಈͷ֬ೝ
    » curl localhost:9200
    {
    "status" : 200,
    "name" : "Araña",
    "version" : {
    "number" : "1.3.2",
    "build_hash" : "dee175dbe2f254f3f26992f5d7591939aaefd12f",
    "build_timestamp" : "2014-08-13T14:29:30Z",
    "build_snapshot" : false,
    "lucene_version" : "4.9"
    },
    "tagline" : "You Know, for Search"
    }
    

    View Slide

36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : "Clinton Gormley",
    "started" : "2013-02-04",
    "pages" : 230
    }'
    σʔλొ࿥
    

    View Slide

37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : "2013-02-04",
    "pages" : 230
    }'
    σʔλߋ৽
    

    View Slide

38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    σʔλ࡟আ
    » curl -X DELETE localhost:9200/books/book/1
    σʔλͷऔಘ
    » curl —X GET localhost:9200/books/book/1
    » curl —X GET localhost:9200/books/book/1/_source
    

    View Slide

39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧ
    » curl -XGET localhost:9200/books/_search?q=elasticsearch
    {
    "took" : 2, "timed_out" : false,
    "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
    "hits" : {
    "total" : 1, "max_score" : 0.076713204,
    "hits" : [ {
    "_index" : “books", "_type" : “book", "_id" : "1",
    "_score" : 0.076713204, "_source" : {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : “2013-02-04", "pages" : 230
    }
    } ]
    }
    }
    

    View Slide

40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XGET ‘localhost:9200/books/book/_search' -d '{
    "query": {
    "filtered" : {
    "query" : {
    "match": {
    "text" : {
    "query" : “To Be Or Not To Be",
    "cutoff_frequency" : 0.01
    }
    }
    },
    "filter" : {
    "range": {
    "price": {
    "gte": 20.0
    "lte": 50.0
    ...
    }
    }'
    ݕࡧ - Query DSL
    

    View Slide

41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ෼ࢄߏ੒ɺεέʔϧ
    

    View Slide

42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Basic terms
    • ΠϯσοΫε
    σʔλͷ࿦ཧతͳू߹ɻ

    RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical
    • ϨϓϦέʔγϣϯ
    ಡΈࠐΈͷεέʔϥϏϦςΟ޲্
    SPOFͷղফ
    • γϟʔσΟϯά
    ෳ਺Ϛγϯ΁σʔλΛ෼ׂ

    ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্

    σʔλϑϩʔ੍ޚ
    

    View Slide

43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    γϟʔυͱϨϓϦΧ
    node 1
    orders
    products
    1
    4
    1 2
    2
    2
    curl  -­‐X  PUT  localhost:9200/orders  -­‐d  '{  
       "settings.index.number_of_shards"  :  4  
       "settings.index.number_of_replicas"  :  1  
    }'
    curl  -­‐X  PUT  localhost:9200/products  -­‐d  '{  
       "settings.index.number_of_shards"  :  2  
       "settings.index.number_of_replicas"  :  0  
    }'
    

    View Slide

44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    γϟʔυͱϨϓϦΧ
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    2
    3 4
    1 2
    3
    

    View Slide

45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ࣗಈతͳ෼ࢄ
    node 1
    orders
    products
    2
    1
    4
    1
    node 2
    orders
    products
    2
    2
    node 3
    orders
    products
    3 4
    1
    3
    

    View Slide

46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Ϋϥελ؅ཧ
    • ୯ҰϚελ
    Ϋϥελͷঢ়ଶʢϊʔυ৘ใɺϚοϐϯάʣΛ؅ཧ
    • ϚϧνΩϟετʹΑΔϊʔυ୳ࡧ

    ɹ (ϢχΩϟετ΋Մೳ)
    • ඞཁͳઃఆ
    Ϋϥελ໊

    ࠷খϚελϊʔυ਺
    • Tips: 3ͭͷϚελϊʔυΛ༻ҙ͠ɺσʔλΛอ࣋
    ͠ͳ͍
    

    View Slide

47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ΫϥελɺϊʔυͷαΠδϯά
    • σʔλྔͱϦΫΤετʹґଘ
    υΩϡϝϯτ਺͸ʁϑΟʔϧυ਺͸ʁΫΤϦͷׂ߹͸ʁ

    Facet΍AggregationɺιʔτɺΧελϜείΞͷར༻͸ʁ

    ॻ͖ࠐΈͷׂ߹͸ʁυΩϡϝϯτ͸࡟আ/ߋ৽͢Δʁ

    ࣌ؒʹґଘͨ͠σʔλʁ
    • 1ϊʔυɺϨϓϦΧແ͠Ͱςετ
    γϟʔυαΠζɺJVMώʔϓ࢖༻ྔɺGCස౓ɺγϟʔυ͓Α
    ͼϊʔυ਺ɺγϟʔυ͋ͨΓͷυΩϡϝϯτ਺ɺCPU࢖༻
    ཰ɺσΟεΫ࢖༻཰ɺΠϯσοΫεͷύλʔϯ
    • Tips: 30 GB heap
    

    View Slide

48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Ϋϥελ͓ΑͼΞϓϦͷߏ੒
    • ϩʔΧϧσΟεΫ; SAN OK
    • Tips: ϨΠςϯγͷߴ͍WAN্ͷΫϥελ͸ඇਪ
    ঑ɻCross-zone in EC2 is OK.
    !
    Your app
    ES Data 1 ES Data N
    ...
    ES Master; no data
    !
    Your app ...
    ES Master; no data
    ES Master; no data High Speed Network
    ES Node Client ES Node Client
    

    View Slide

49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ΠϯσοΫεͷొ࿥ͱݕࡧͷ
    ಺෦ಈ࡞
    

    View Slide

50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτొ࿥γʔέϯε
    • υΩϡϝϯτΛొ࿥
    doc1 node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    

    View Slide

51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτొ࿥γʔέϯε
    • υΩϡϝϯτͷIDͷϋογϡ஋Λܭࢉͯ͠γϟʔ
    υ਺ͰׂΓɺొ࿥͢ΔγϟʔυΛܾΊΔ
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    doc1
    

    View Slide

52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτొ࿥γʔέϯε
    • γϟʔυ͕ܾ·ͬͨΒɺϓϥΠϚϦγϟʔυ͕͋
    Δϊʔυ͕Ͳ͔͜Λௐ΂Δ
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    doc1
    

    View Slide

53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    υΩϡϝϯτొ࿥γʔέϯε
    • ϓϥΠϚϦ͔ΒϨϓϦΧ΁
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    doc1
    

    View Slide

54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • ݕࡧΫΤϦΛ࣮ߦ
    Query node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    

    View Slide

55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • ϊʔυ͕̍ड͚औͬͨΫΤϦΛશγϟʔυʹૹ৴
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    Query
    

    View Slide

56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • ϊʔυ͕̍ड͚औͬͨΫΤϦΛશγϟʔυʹૹ৴
    • γϟʔυຖʹݕࡧ࣮ߦ
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    Query
    Query
    Query
    Query
    Query
    

    View Slide

57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • શγϟʔυ͔Βͷ݁ՌΛϚʔδ
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    Query
    ݁Ռ
    ݁Ռ
    ݁Ռ
    ݁Ռ
    

    View Slide

58. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • શγϟʔυ͔Βͷ݁ՌΛϚʔδ
    node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    ݁Ռ
    

    View Slide

59. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ݕࡧγʔέϯε
    • ݁ՌΛૹ৴
    ݁Ռ node 1
    bookstore
    2
    1
    4
    node 2
    bookstore
    2
    3
    3 4
    1
    

    View Slide

60. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysis & Mappings
    

    View Slide

61. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysisͱ͸ʁ
    • సஔΠϯσοΫεͷ୯ޠʢTermʣΛυΩϡϝϯτ
    ͔Βநग़͢Δॲཧ
    • ϑΟʔϧυຖʹࢦఆ͞ΕͨΞφϥΠβ͕ॲཧ
    • ΞφϥΠβ͸CharFilterɺTokenizerɺTokenFilter͔
    Βߏ੒
    • ΠϯσοΫε࣌ɺݕࡧ࣌ʹ࣮ߦ
    

    View Slide

62. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysisͷ֓ཁ
    Analysis
    υΩϡϝϯτ
    Term Id
    ΧπΦ 1
    αβΤ 1ɺ2
    ϫΧϝ 2
    … ...
    ΠϯσοΫε
    

    View Slide

63. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analysisͷ֓ཁ
    Analysis
    ݕࡧ

    Ωʔϫʔυ
    ΠϯσοΫε
    ΠϯσοΫε

    ݕࡧॲཧ
    ݕࡧ݁Ռ
    Term Id
    ΧπΦ 1
    αβΤ 1ɺ2
    ϫΧϝ 2
    … ...
    

    View Slide

64. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analyzer
    Analysisͷߏ੒
    char_filter tokenizer
    char_filter
    token_

    filter
    token_

    filter
    Text Tokens
    

    View Slide

65. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Analyzerͷઃఆ
    {"index":{
    “analysis":{
    "analyzer" : {
    "my_analyzer" : {
    "type" : "custom",
    "tokenizer" : “kuromoji_tokenizer”,
    “char_filter" : [“char_filter1”,…],
    “filter" : [“token_filter1”,…]
    }
    }
    

    View Slide

66. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Char Filter
    • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ
    !
    • ྫɿhtml_strip
    ೖྗɿ
    !
    ग़ྗɿ
    Elasticsearch is not a service of AWS
    Elasticsearch is not a service of AWS
    

    View Slide

67. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Tokenizer
    • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ
    • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹ
    ґଘ
    • ྫɿstandard
    ೖྗɿ
    !
    ग़ྗɿ Elasticsearch is not a service of AWS
    Elasticsearch is not a service of AWS
    

    View Slide

68. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Tokenizer
    • ྫɿkeyword
    ग़ྗɿ
    • ྫɿkuromoji_tokenizer
    ೖྗɿ
    !
    ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    Elasticsearch is not a service of AWS
    ण͕࢘ඒຯ͔ͬͨ͠
    

    View Slide

69. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    TokenFilter
    • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ
    • ྫɿlowercase
    ೖྗɿ
    !
    ग़ྗɿ
    • ྫɿstop
    ೖྗɿ
    !
    ग़ྗɿ
    Elasticsearch is not a service of AWS
    elasticsearch is not a service of aws
    Elasticsearch is not a service of AWS
    Elasticsearch service AWS
    

    View Slide

70. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    TokenFilter
    • ྫɿkuromoji_baseform
    ೖྗɿ
    !
    ग़ྗɿ
    • ྫɿkuromoji_readingform
    ೖྗɿ
    !
    ग़ྗɿ
    ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    ण࢘ ͕ ඒຯ͍͠ ͨ
    ण࢘ ͕ ඒຯ͔ͬ͠ ͨ
    sushi ga oishika ta
    

    View Slide

71. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Mapping
    • ΠϯσοΫεͷߏ଄ʢschemaʣΛఆٛ
    • Schema less!͚ͩͲ…
    • ܗࣜΛࢦఆʢanalyze͢Δ/͠ͳ͍ɺ੔਺/গ਺ɺ೔
    ෇ͳͲʣ
    

    View Slide

72. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿWikipediaͷMappingͷҰ෦
    "mappings": {
    "page": {
    "properties": {
    "link": {
    "type": "string",
    "fields": {
    "raw": {"type": "string",
    "index": "not_analyzed"} } },
    "text": { "type": "string"},
    "title": {
    "type": "string",
    "fields": {
    "raw": {"type": "string",
    "index": "not_analyzed"} }…
    

    View Slide

73. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    aggregation
    

    View Slide

74. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregationͱ͸
    • 1.0͔Βಋೖ
    • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ
    ֊૚తͳूܭɺάϧʔϓԽ

    ಈతͳूܭɺάϧʔϓԽ
    • େ͖͘2छྨ
    BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά
    Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ
    

    View Slide

75. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿݴޠ͝ͱͷϑΝηοτ
    » curl -XGET twitter-2014.08.22/_search -d '
    GET twitter-2014.08.22/_search
    {
    "facets": {
    "lang_facet": {
    "terms": {"field": "lang"}
    }
    }
    }
    

    View Slide

76. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿݴޠ͝ͱͷϑΝηοτ
    "facets": {
    "lang_facet": {
    "_type": "terms",
    "missing": 0,
    "total": 4521075,
    "other": 320228,
    "terms": [
    {
    "term": "en",
    "count": 1471819
    },
    {
    "term": "ja",
    "count": 980145
    },
    

    View Slide

77. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿݴޠ͓Αͼ஍Ҭͷूܭ
    » curl -XGET twitter-2014.08.22/_search -d '
    {
    "aggs": {
    "lang": {
    "terms": {"field": "lang" },
    "aggs": {
    "place": {
    "terms": {
    "field": “place.full_name", "size": 10
    }
    }
    }
    }
    }
    }
    

    View Slide

78. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿݴޠ͓Αͼ஍Ҭͷूܭ
    "aggregations": {
    "lang": {
    "buckets": [{…}, {
    "key": "ja",
    "doc_count": 980145,
    "place": {
    "buckets": [
    { "key": "ژ౎ࢢ෬ݟ۠, ژ౎",
    "doc_count":252 },
    { "key": "ઍ୅ా۠, ౦ژ",
    "doc_count": 39 },…
    

    View Slide

79. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    bucket
    • filter
    • filters (>=1.4.0)
    • missing
    • nested
    • reverse nested
    (>=1.2.0)
    • children
    • terms
    • significant terms
    (>=1.1.0)
    • range
    • date range
    • ipv4 range
    • histogram
    • date histogram
    • geo distance
    • geohash grid
    

    View Slide

80. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    metric
    • min
    • max
    • sum
    • avg
    • stats
    • extended_stats
    • value count
    • percentiles (>=1.1.0)
    • percentile ranks (>=1.3)
    • cardinality (>=1.1.0)
    • geo bounds (>=1.3.0)
    • top_hits (>=1.3.0)
    • scripted metric 

    (>= 1.4.0)
    

    View Slide

81. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ͦͷଞͷػೳ
    

    View Slide

82. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Geo
    • ͞·͟·ͳܗࣜͷσʔλͰGeoݕࡧՄೳ
    • σʔλɿ
    Ң౓ܦ౓ɺGeoHashɺGeoShape
    • ݕࡧɿ
    geo shape queryɺgeo shape filterɺgeo distance filterɺ
    geo distance range filterɺgeo bounding box filterɺ
    geo polygon filter
    

    View Slide

83. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿGeoσʔλొ࿥
    ## Ң౓ܦ౓
    {
    "pin" : {
    "location" : "41.12,-71.34"
    }
    }
    !
    ## GeoHash
    {
    "pin" : {
    "location" : "drm3btev3e86"
    }
    }
    

    View Slide

84. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿGeoσʔλొ࿥
    ## GeoShape(Polygon)
    {
    "location" : {
    "type" : "polygon",
    "coordinates" : [[ [100.0, 0.0], [101.0, 0.0],
    [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ]]
    }
    }
    ## GeoShape(Circle)
    {
    "location" : {
    "type" : "circle",
    "coordinates" : [-45.0, 45.0],
    "radius" : "100m"
    }
    }
    

    View Slide

85. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿGeoσʔλݕࡧ
    {
    "query": {
    "geo_shape": {
    "location": {
    "shape": {
    "type": "envelope",
    "coordinates": [[13, 53],[14, 52]]
    }
    }
    }
    }
    }
    

    View Slide

86. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿڑ཭ʹΑΔϑΟϧλ
    {
    "filtered" : {
    "query" : {"match_all" : {}},
    "filter" : {
    "geo_distance" : {
    "distance" : "200km",
    "pin.location" : {
    "lat" : 40,
    "lon" : -70
    }
    }
    }
    }
    }
    

    View Slide

87. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Percolator
    • υΩϡϝϯτͰ͸ͳ͘ΫΤϦΛొ࿥
    • _percolate APIʹυΩϡϝϯτΛૹ৴
    • ొ࿥͞ΕͨΫΤϦʹϚονͨ͠৔߹ɺ

    Ϛονͨ͠ΫΤϦΛฦ͢
    

    View Slide

88. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿΫΤϦొ࿥
    curl -XPUT 'localhost:9200/my-index/.percolator/1' -d
    '{
    "query" : {
    "match" : {
    "message" : "bonsai tree"
    }
    }
    }'
    

    View Slide

89. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿσʔλνΣοΫ
    curl -XGET 'localhost:9200/my-index/message/_percolate'
    -d '{
    "doc" : {
    "message" : "A new bonsai tree in the office"
    }
    }'
    

    View Slide

90. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿ݁Ռ
    {
    "took" : 19,
    "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
    },
    "total" : 1,
    "matches" : [
    {
    "_index" : "my-index",
    "_id" : "1"
    }
    ]
    }
    

    View Slide

91. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Snapshot/Restore
    • 1.0͔Βಋೖ
    • ΠϯσοΫε୯ҐͰSnapshot/RestoreՄೳ
    • อଘઌʢRepositoryʣ
    Shard FSɺS3ɺHDFSɺAzure…
    

    View Slide

92. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿϦϙδτϦ࡞੒
    $ curl -XPUT 'http://localhost:9200/_snapshot/
    my_backup' -d '{
    "type": "fs",
    "settings": {
    "location": "/mount/backups/my_backup",
    "compress": true
    }
    }'
    

    View Slide

93. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    ྫɿSnapshotͱϦετΞ
    ## Snapshot

    $ curl -XPUT “localhost:9200/_snapshot/my_backup/
    snapshot_1?wait_for_completion=true"
    !
    !
    ## Restore
    $ curl -XPOST "localhost:9200/_snapshot/my_backup/
    snapshot_1/_restore"
    

    View Slide

94. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Ecosystem
    • Plugins
    ϓϥάΠϯʹΑΔػೳͷ௥Ճ
    • ΫϥΠΞϯτϥΠϒϥϦ
    Ruby, python, php, perl, javascript, .NET
    Scala, clojure, go
    • Kibana
    • Logstash
    

    View Slide

95. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Hadoop࿈ܞ&Kibana
    • ʮαʔόʗΠϯϑϥΤϯδχΞཆ੒ಡຊ ϩάऩ
    ूʙՄࢹԽฤʯൃചத
    • Hadoop࿈ܞͷ঺հ
    • Kibanaͷৄࡉ
    

    View Slide

96. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    

    View Slide

97. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Tools for sys admins
    

    View Slide

98. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    REST-based management
    • Elasticsearch is full of monitoring APIs
    Everything is returned as JSON
    • Humans are not the world’s best JSON parsers
    • What if elasticsearch had an easy to use interface
    from the commandline?
    

    View Slide

99. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Which node is the master?
    $ curl "localhost:9200/_cluster/state?pretty&filter_metadata=true&!
    filter_routing_table=true"!
    {!
    "cluster_name" : "elasticsearch",!
    "master_node" : "GNf0hEXlTfaBvQXKBF300A",!
    "blocks" : { },!
    "nodes" : {!
    "ObdRqLHGQ6CMI5rOEstA5A" : {!
    "name" : "Triton",!
    "transport_address" : “inet[/10.0.1.11:9300]”,!
    "attributes" : { }!
    },!
    "4C7pKbfhTvu0slcSy_G4_w" : {!
    "name" : "Kid Colt",!
    "transport_address" : "inet[/10.0.1.12:9300]",!
    "attributes" : { }!
    },!
    "GNf0hEXlTfaBvQXKBF300A" : {!
    "name" : "Lang, Steven",!
    "transport_address" : "inet[/10.0.1.13:9300]",!
    "attributes" : { }!
    }!
    }!
    }
    

    View Slide

100. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
     Which one is the master? (v1.0)
     $ curl localhost:9200/_cat/master
     GNf0hEXlTfaBvQXKBF300A 10.0.1.13 Lang, Steven
     

     View Slide

101. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
     _cat/* api
     • /_cat/allocation
     • /_cat/count
     • /_cat/health
     • /_cat/master
     • /_cat/aliases
     • /_cat/nodes
     • /_cat/recovery
     • /_cat/shards
     • /_cat/indices
     • /_cat/thread_pool
     

     View Slide

102. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Monitor your cluster with Marvel
     • Point in time views are a start
     • Marvel shows historical trends
     • Visualize cluster behavior, act before problems
     !
     • Free for development, $500/year for up to 5
     nodes
     

     View Slide

103. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Overview
     

     View Slide

104. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Node statistics
     

     View Slide

105. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Index statistics
     

     View Slide

106. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Cluster Pulse
     

     View Slide

107. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Sense
     

     View Slide

108. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     σϞ for Elasticsearch

     ೔ຊޠWikipedia
     

     View Slide

109. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     σϞ for Elasticsearch

     Twitter Stream
     

     View Slide

110. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     σϞ for Kibana

     Twitter Stream
     

     View Slide

111. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     σϞ for Kibana

     Git log
     

     View Slide

112. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     ࢀߟจݙ
     • Elasticsearch - The Definitive guide
     http://www.elasticsearch.com/guide/en/elasticsearch/guide/
     current/index.html
     • ॻ੶
     ElasticSearchServer೔ຊޠ൛

     αʔό/ΠϯϑϥΤϯδχΞ

     ɹཆ੒ಡຊɹϩάऩू
     

     View Slide

113. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
     Q&A
     • Training
     http://training.elasticsearch.com
     • Support
     http://www.elasticsearch.com/support/
     • Github
     https://github.com/elasticsearch
     

     View Slide