様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics

657aeeff3fc467567dacebf8a1ea0b23?s=47 Jun Ohtani
October 27, 2018

様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics

OSC 2018 Tokyo/Fall での発表資料になります。

657aeeff3fc467567dacebf8a1ea0b23?s=128

Jun Ohtani

October 27, 2018
Tweet

Transcript

  1. 2.

    !2 ΞδΣϯμ • ϝτϦοΫʗϩάͱ͸ʁ • γεςϜϝτϦΫεղੳɺϩάղੳΛࢼ͠ʹ΍ͬͯΈΑ͏ • Beats - Elasticsearch

    - KibanaͰղੳ • ຊ֨తʹղੳΛ΍Δʹ͸ʁ • LogstashͰϩά΍ϝτϦΫεΛதܧɾू໿ • ͞Βʹ৭ʑࢼͯ͠ΈΔʹ͸ʁ
  2. 3.

    !3 about • Me, Jun Ohtani / Community Engineer ‒

    lucene-gosenίϛολʔ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ڞஶ ‒ http://blog.johtani.info
 • Elastic, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 Elastic APM, 
 Elastic Cloud, Swiftype 
 Professional services: Support & development subscriptions
 Trainings, Consulting, SaaS
  3. 6.
  4. 10.

    Metrics Logging APM Site
 Search Application Search Business
 Analytics Enterprise


    Search Security
 Analytics Future ιϦϡʔγϣϯ อଘɺݕࡧɺ෼ੳ ՄࢹԽɺ؅ཧ ΠϯδΣετ Kibana Elasticsearch Beats Logstash Elastic Stack
  5. 11.

    Metrics Logging APM Site
 Search App
 Search Business
 Analytics Enterprise


    Search Security
 Analytics Future ιϦϡʔγϣϯ SaaS Elastic Cloud Self Managed Elastic Cloud
 Enterprise Standalone σϓϩΠ อଘɺݕࡧɺ෼ੳ ՄࢹԽɺ؅ཧ ΠϯδΣετ Kibana Elasticsearch Beats Logstash Elastic Stack
  6. 12.

    อଘɺݕࡧɺ෼ੳ Elasticsearch ՄࢹԽɺ؅ཧ Kibana Beats ΠϯδΣετ Logstash Metrics Logging APM

    Site
 Search Application Search Business
 Analytics Enterprise
 Search Security
 Analytics Future ιϦϡʔγϣϯ SaaS Elastic Cloud Self Managed Elastic Cloud
 Enterprise Standalone σϓϩΠ Elastic Stack
  7. 15.

    !15

  8. 17.

    The Beats family Heartbeat Uptime monitoring Filebeat Log files Winlogbeat

    Windows Event Logs Packetbeat Network data +40 community Beats Metricbeat Metrics Auditbeat Audit data
  9. 20.

    !20 Metricbeat Ϟδϡʔϧ • Aerospike module • Apache module •

    Ceph module • Couchbase module • Docker module • Dropwizard module • Elasticsearch module • Etcd module • Golang module • Graphite module • HAProxy module • HTTP module • Jolokia module • Kafka module • Kibana module • Kubernetes module • kvm module • Logstash module • Memcached module • MongoDB module • Munin module • MySQL module • Nginx module • • PHP_FPM module • PostgreSQL module • Prometheus module • RabbitMQ module • Redis module • System module • uwsgi module • vSphere module • Windows module • ZooKeeper module

  10. 23.

    Filebeat modules - v6.4.2 • Apache2 module • Auditd module

    • Icinga module • IIS module • Kafka module • Logstash module • MongoDB module • MySQL module • Nginx module • Osquery module • PostgreSQL module • Redis module • System module • Traefik module
  11. 28.

    !28

  12. 29.

    29 Elasticsearch Heart of the Elastic Stack ෼ࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ

    ։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ
  13. 36.
  14. 38.

    Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:

    Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ !38
  15. 40.

    σʔλొ࿥ 40 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
  16. 41.

    σʔλߋ৽ 41 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
  17. 42.

    σʔλ࡟আ !42 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET

    localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
  18. 43.

    ݕࡧ - Query DSL !43 curl -XGET ‘localhost:9200/books/doc/_search' -d '{

    "query": { "bool": { "must": [ { "match": { "title": "Search" }}, { "match": { "content": "Elasticsearch" }} ], "filter": [ { "term": { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ] } } }'
  19. 45.

    Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ •

    ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ !45
  20. 46.

    γϟʔυͱϨϓϦΧ !46 node 1 orders products 1 4 1 2

    2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
  21. 47.
  22. 48.

    ࣗಈతͳ෼ࢄ !48 node 1 orders products 2 1 4 1

    node 2 orders products 2 2 node 3 orders products 3 4 1 3
  23. 54.

    υΩϡϝϯτͷొ࿥ !54 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ

    ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ
  24. 55.

    υΩϡϝϯτͷొ࿥ !55 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ

    ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ
  25. 56.

    ݕࡧ !56 ΧπΦ αβΤ 2 ͸ ͷ ࢞ ϫΧϝ 2

    1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ 1 1
  26. 57.

    ݕࡧ !57 ΧπΦ αβΤ 2 ͸ ͷ ࢞ ϫΧϝ 2

    1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ 1 1
  27. 58.

    ݕࡧ !58 ΧπΦ αβΤ 2 ͸ ͷ ࢞ ϫΧϝ 2

    1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ 1 1
  28. 59.

    ݕࡧ !59 ΧπΦ αβΤ 2 ͸ ͷ ࢞ ϫΧϝ 2

    1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ 1 1
  29. 60.

    ݕࡧ !60 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  30. 61.

    ݕࡧ !61 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  31. 62.

    ݕࡧ !62 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
  32. 63.

    ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 


    • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 
 !63
  33. 64.

    ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 


    εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ 64
  34. 67.

    N-Gram • ϝϦοτɿ ‒ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒

    ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ !67 ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
  35. 70.

    Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby,

    python, php, perl, javascript, .NET • Scala, clojure, go !70
  36. 72.

    !72

  37. 73.

    73 Kibana Window into the Elastic Stack ՄࢹԽͱ෼ੳ ஍ཧۭؒ ΧελϚΠζͱ

    Ϩϙʔτͷڞ༗ άϥϑ୳ࡧ Elastic Stack΁ͷ ηΩϡΞͳΞΫηεͱ؅ཧ ΧελϜAppsͷ࡞੒
  38. 78.

    !78 Elastic Stackͷߏ੒ Beats Log Files Metrics Wire Data your{beat}

    Kibana Instances Kafka Distributed Message Queue Notification Queues Storage Metrics Data Store Web APIs Social Sensors Elasticsearch Nodes Logstash Nodes
  39. 79.

    !79

  40. 82.

    Logstash architecture !82 Input Output Filter ? ? collect and

    split alter and enrich store and visualize
  41. 85.

    1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 85
  42. 86.

    ઃఆɿfilter 86 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  43. 87.

    ύʔε !87 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
  44. 88.

    ઃఆɿfilter !88 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  45. 89.

    ೔෇ͷύʔε 89 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", …

    } {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }
  46. 90.

    ઃఆɿfilter !90 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  47. 91.

    IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 91 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,

    … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
  48. 92.

    ઃఆɿfilter !92 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  49. 93.

    ϢʔβΤʔδΣϯτͷύʔε 93 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101

    Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"
  50. 94.
  51. 96.

    !96 elasticsearch-hadoop - •  D E H •  PD ecd

    ER •  g D •  CH •  Ca M DMS D FERC
  52. 97.

    !97

  53. 98.

    !98

  54. 99.

    !99

  55. 101.

    !101 σʔλͷొ࿥ํ๏ • Kibanaͷαϯϓϧσʔλʢ6.4͔Βʣ • LogstashͰJDBC input • LogstashͰCSV •

    FilebeatͰΞΫηεϩά • MetricbeatͰϝτϦοΫ • PacketbeatͰMySQL/PostgreSQLͷύέοτղੳ
  56. 113.

    !113 ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/

    index.html • ॻ੶ʢ೔ຊޠʣ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ‒ Elasticsearch࣮ફΨΠυ
  57. 114.

    !114 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co

    • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions