Elastic Stackでマイクロサービス運用を
楽にするには? / Monitoring Microservices with Elastic Stack

657aeeff3fc467567dacebf8a1ea0b23?s=47 Jun Ohtani
January 25, 2019

Elastic Stackでマイクロサービス運用を
楽にするには? / Monitoring Microservices with Elastic Stack

OSC 大阪 2019でのセッションで使用した資料です。
「マイクロサービス」とありますが、アプリケーションの監視の話がメインですので、アプリケーションの監視やログ解析などに興味のある方にも読んでいただける内容となっています。

657aeeff3fc467567dacebf8a1ea0b23?s=128

Jun Ohtani

January 25, 2019
Tweet

Transcript

  1. !1 2019/01/25 Community Engineer @Elastic
 Jun Ohtani @johtani Elastic StackͰϚΠΫϩαʔϏεӡ༻Λ


    ָʹ͢Δʹ͸ʁ - Elastic Stackͷೖ໳ͱ׆༻ -
  2. !2 about • Me, Jun Ohtani / Community Engineer ‒

    lucene-gosenίϛολʔ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ڞஶ ‒ http://blog.johtani.info
 • Elastic, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 Elastic APM, 
 Elastic Cloud, Swiftype 
 Professional services: Support & development subscriptions
 Trainings, Consulting, SaaS
  3. !3 ΞδΣϯμ • ϚΠΫϩαʔϏεͱ͸ʁ • Elastic Stackͱ͸ʁ • ༷ʑͳ؍఺͔ΒͷΞϓϦέʔγϣϯͷ؂ࢹ •

    ͞Βʹ৭ʑࢼͯ͠ΈΔʹ͸ʁ
  4. !4 ϚΠΫϩαʔϏεͱ͸

  5. !5 ϚΠΫϩαʔϏε (Wikipedia) https://ja.wikipedia.org/wiki/ϚΠΫϩαʔϏε

  6. !6 ϞϊϦγοΫ v.s. ϚΠΫϩαʔϏε

  7. !7 ϚΠΫϩαʔϏεʁʁʁ

  8. !8 Elastic Stackͱ͸ʁ

  9. Elastic Stack อଘɺݕࡧɺ෼ੳ Elasticsearch ՄࢹԽɺ؅ཧ Kibana Beats ΠϯδΣετ Logstash

  10. Metrics Logging APM Site
 Search Application Search Business
 Analytics Enterprise


    Search Security
 Analytics Future ιϦϡʔγϣϯ อଘɺݕࡧɺ෼ੳ ՄࢹԽɺ؅ཧ ΠϯδΣετ Kibana Elasticsearch Beats Logstash Elastic Stack
  11. Metrics Logging APM Site
 Search App
 Search Business
 Analytics Enterprise


    Search Security
 Analytics Future ιϦϡʔγϣϯ SaaS Elastic Cloud Self Managed Elastic Cloud
 Enterprise Standalone σϓϩΠ อଘɺݕࡧɺ෼ੳ ՄࢹԽɺ؅ཧ ΠϯδΣετ Kibana Elasticsearch Beats Logstash Elastic Stack
  12. อଘɺݕࡧɺ෼ੳ Elasticsearch ՄࢹԽɺ؅ཧ Kibana Beats ΠϯδΣετ Logstash Metrics Logging APM

    Site
 Search Application Search Business
 Analytics Enterprise
 Search Security
 Analytics Future ιϦϡʔγϣϯ SaaS Elastic Cloud Self Managed Elastic Cloud
 Enterprise Standalone σϓϩΠ Elastic Stack
  13. !13

  14. 14 Beats ܰྔσʔλγούʔ ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू໿ ม׵ͱύʔεͷͨΊ Logstashʹసૹ Elastic Cloudʹసૹ Libbeat:

    ΧελϜbeatsͷͨ ΊͷAPIϑϨʔϜϫʔΫ 30Ҏ্ͷίϛϡχςΟbeats
  15. The Beats family Heartbeat Uptime monitoring Filebeat Log files Winlogbeat

    Windows Event Logs Packetbeat Network data +40 community Beats Metricbeat Metrics Auditbeat Audit data
  16. !16

  17. 17 Logstash σʔλՃ޻ύΠϓϥΠϯ શͯͷܗࣜɺαΠζͱσʔλιʔ εͷ౤ೖ ύʔεͱಈతͳ σʔλม׵ ͋ΒΏΔग़ྗʹ σʔλసૹ ҆શͰ҉߸Խ͞Εͨ


    σʔλೖྗ ಠࣗͷύΠϓϥΠϯॲཧ ͷ࡞੒ 200Ҏ্ͷϓϥάΠϯ
  18. !18

  19. 19 Elasticsearch Heart of the Elastic Stack ෼ࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ

    ։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ
  20. Elasticsearchͱ͸ʁ

  21. ϑϦʔϫʔυݕࡧ !21

  22. ߜΓࠐΈ !22

  23. ϋΠϥΠτ !23

  24. ιʔτ !24

  25. ϖʔδϯά !25

  26. ूܭ !26

  27. αδΣετ !27

  28. Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:

    Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ !28
  29. ؆୯ͳCRUD

  30. σʔλొ࿥ 30 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
  31. σʔλߋ৽ 31 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
  32. σʔλ࡟আ !32 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET

    localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
  33. ݕࡧ - Query DSL !33 curl -XGET ‘localhost:9200/books/doc/_search' -d '{

    "query": { "bool": { "must": [ { "match": { "title": "Search" }}, { "match": { "content": "Elasticsearch" }} ], "filter": [ { "term": { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ] } } }'
  34. ෼ࢄߏ੒ɺ
 εέʔϧ

  35. γϟʔυͱϨϓϦΧ !35 node 1 orders products 1 4 1 2

    2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
  36. γϟʔυͱϨϓϦΧ !36 node 1 orders products 1 4 1 node

    2 orders products 2 2 3 4 1 2 3
  37. ࣗಈతͳ෼ࢄ !37 node 1 orders products 2 1 4 1

    node 2 orders products 2 2 node 3 orders products 3 4 1 3
  38. !38

  39. 39 Kibana Window into the Elastic Stack ՄࢹԽͱ෼ੳ ஍ཧۭؒ ΧελϚΠζͱ

    Ϩϙʔτͷڞ༗ άϥϑ୳ࡧ Elastic Stack΁ͷ ηΩϡΞͳΞΫηεͱ؅ཧ ΧελϜAppsͷ࡞੒
  40. !40 Kibana 6

  41. !41 Elastic Stackͷߏ੒ Beats Log Files Metrics Wire Data your{beat}

    Kibana Instances Kafka Distributed Message Queue Notification Queues Storage Metrics Data Store Web APIs Social Sensors Elasticsearch Nodes Logstash Nodes
  42. !42 ΞϓϦέʔγϣϯͷ
 ؂ࢹϙΠϯτ

  43. !43 ؂ࢹϙΠϯτ • ֎ܗ؂ࢹ • ϝτϦοΫʢϝτϦΫεʣ • αʔόʔɺΞϓϦέʔγϣϯ • ϩά

    • ΞϓϦέʔγϣϯͷϦϦʔελΠϛϯά • ෼ࢄτϨʔγϯά
  44. !44 ֎ܗ؂ࢹ • ࢮ׆؂ࢹ • ϓϩηε • HTTPαʔόʔ • TCP

    • ICMP
  45. Lightweight Shipper for Uptime Monitoring Heartbeat

  46. !46 ϝτϦοΫ • αʔόʔ • CPUɺϝϞϦɺσΟεΫɺωοτϫʔΫI/Oɺϓϩηε਺ • ΞϓϦέʔγϣϯ • ϦΫΤετ਺ɺίωΫγϣϯ਺ɺॲཧ࣌ؒ

    • ίϯςφʔ • ίϯςφ਺
  47. Collect system and application metrics Metricbeat

  48. lots of modules Metricbeat

  49. !49 Metricbeat Ϟδϡʔϧ • Aerospike module • Apache module •

    Ceph module • Couchbase module • Docker module • Dropwizard module • Elasticsearch module • Etcd module • Golang module • Graphite module • HAProxy module • HTTP module • Jolokia module • Kafka module • Kibana module • Kubernetes module • kvm module • Logstash module • Memcached module • MongoDB module • Munin module • MySQL module • Nginx module • • PHP_FPM module • PostgreSQL module • Prometheus module • RabbitMQ module • Redis module • System module • uwsgi module • vSphere module • Windows module • ZooKeeper module

  50. !50 ϩά • ΞΫηεϩά • γεςϜϩά • ೝূϩά • εϩʔϩά

    • ΞϓϦέʔγϣϯϩά
  51. tail log from file Filebeat

  52. many modules Filebeat

  53. Filebeat modules - v6.4.2 • Apache2 module • Auditd module

    • Icinga module • IIS module • Kafka module • Logstash module • MongoDB module • MySQL module • Nginx module • Osquery module • PostgreSQL module • Redis module • System module • Traefik module
  54. Welcome to 1998 winlogbeat

  55. Now winlogbeat

  56. Capture the Packet Packetbeat

  57. Capture the Packet Packetbeat

  58. !58 ΞϓϦέʔγϣϯͷϦϦʔελΠϛϯά • όάϑΟοΫεϦϦʔε • ৽ػೳϦϦʔε • ৽αʔϏε։࢝ • αʔόʔ૿ڧ

  59. !59 ෼ࢄτϨʔγϯά • ϚΠΫϩαʔϏε • 1ͭͷϦΫΤετʹରͯ͠ෳ਺ͷϓϩηε͕ؔ܎ • ΞϓϦέʔγϣϯύϑΥʔϚϯεϞχλϦϯάͷ1ͭ

  60. !60

  61. !61 Elastic APM

  62. Distributed Tracing Beta | Basic (free) શͯͷܭଌ͞ΕͨαʔϏεΛݟΔͨΊͷ ౷߹͞ΕͨϏϡʔ αϒίϯςΩετ಺ͷτϨʔεʹભҠ OpenTracing

    ޓ׵
  63. !63 ͦͷ΄͔ͷศརͳػೳ • Infra UI • Logs UI • Machine

    Learning • Alerting
  64. Infrastructure Solution Beta | Basic (free) ΠϯϑϥΦϖϨʔλʔ޲͚ʹಛԽ Λ௒͑ΔΠϯϑϥͷߏ੒Λ၆ᛌ ,VCFSOFUFTɺ%PDLFSͷωΠςΟϒαϙʔ τ

    ϝτϦοΫɺϩάɺ"1.Ϗϡʔ΁ͷ
 υϦϧɾμ΢ϯ ΞυϗοΫ͓Αͼߏ଄Խݕࡧ
  65. Logs Solution Beta | Basic (free) ϥΠϒͰϩάͷτϥϒϧγϡʔςΟϯά Λॿ͚ΔܰྔͳϩάϏϡʔΞʔ ίϯιʔϧͷΑ͏ͳදࣔ UBJMGͷΑ͏ͳ

    ϥΠϒɾϩάɾετϦʔ ϛϯά ཤྺϩάͷແݶεΫϩʔϧ ΞυϗοΫ͓Αͼߏ଄Խݕࡧ
  66. !66

  67. !67

  68. !68

  69. !69 ͞Βʹ׆༻͢Δʹ͸ʁ

  70. !70

  71. !71

  72. ͦͷଞͷ࢖͍ํ !72

  73. !73 σʔλͷొ࿥ํ๏ • Kibanaͷαϯϓϧσʔλʢ6.4͔Βʣ • LogstashͰJDBC input • LogstashͰCSV •

    FilebeatͰΞΫηεϩά • MetricbeatͰϝτϦοΫ • PacketbeatͰMySQL/PostgreSQLͷύέοτղੳ
  74. !74 Kibanaͷαϯϓϧσʔλʢ>= 6.4.0ʣ

  75. !75 ϫϯΫϦοΫͰσʔλొ࿥

  76. !76 LogstashͰJDBC Input Kibana Instances Data Store Elasticsearch Nodes Logstash

    Nodes
  77. !77 JDBC Input

  78. !78 LogstashͰCSV Kibana Instances CSV
 File Elasticsearch Nodes Logstash Nodes

  79. !79 CSV filter

  80. !80 FilebeatͰΞΫηεϩά Beats Log Files Kibana Instances Elasticsearch Nodes

  81. • 2ͭͷElasticsearchϓϥάΠϯΛΠϯετʔϧͯ͠ElasticsearchΛىಈ • Filebeatͷapache2ϞδϡʔϧΛ༗ޮԽ • modules.d/apache2.ymlʹΞΫηεϩάͷύεΛઃఆ • setupίϚϯυΛ࣮ߦ͔ͯ͠ΒFilebeatΛىಈ !81 FilebeatͰΞΫηεϩά

  82. MetricbeatͰϝτϦοΫ Beats Metrics Kibana Instances Elasticsearch Nodes

  83. • MetricbeatͷsystemϞδϡʔϧΛ༗ޮԽ • setupίϚϯυΛ࣮ߦ͔ͯ͠ΒFilebeatΛىಈ !83 MetricbeatͰϝτϦοΫ

  84. !84 PacketbeatͰMySQLɺPostgreSQLͷύέοτղੳ Beats Wire Data Kibana Instances Elasticsearch Nodes

  85. !85 ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/

    index.html • ॻ੶ʢ೔ຊޠʣ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ‒ Elasticsearch࣮ફΨΠυ
  86. !86 ࢀߟจݙ • ೖ໳ ؂ࢹ
 ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯ
 Mike JulianɹஶɺদӜ ൏ਓɹ༁
 


    https://www.oreilly.co.jp/books/9784873118642/
  87. !87 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co

    • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions
  88. Thank you! • Web : https://www.elastic.co/jp/ • Forums : https://discuss.elastic.co/

    • Twitter : @johtani