Elastic Stackでマイクロサービス運用を 楽にするには？ / Monitoring Microservices with Elastic Stack

OSC 大阪 2019でのセッションで使用した資料です。
「マイクロサービス」とありますが、アプリケーションの監視の話がメインですので、アプリケーションの監視やログ解析などに興味のある方にも読んでいただける内容となっています。

Transcript

1. !1 2019/01/25 Community Engineer @Elastic
 Jun Ohtani @johtani Elastic StackͰϚΠΫϩαʔϏεӡ༻Λ
 ָʹ͢Δʹ͸ʁ - Elastic Stackͷೖ໳ͱ׆༻ -
2. !2 about • Me, Jun Ohtani / Community Engineer ‒ lucene-gosenίϛολʔ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ڞஶ ‒ http://blog.johtani.info
 • Elastic, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 Elastic APM, 
 Elastic Cloud, Swiftype 
 Professional services: Support & development subscriptions
 Trainings, Consulting, SaaS
3. !3 ΞδΣϯμ • ϚΠΫϩαʔϏεͱ͸ʁ • Elastic Stackͱ͸ʁ • ༷ʑͳ؍఺͔ΒͷΞϓϦέʔγϣϯͷ؂ࢹ • ͞Βʹ৭ʑࢼͯ͠ΈΔʹ͸ʁ
4. !4 ϚΠΫϩαʔϏεͱ͸
5. !5 ϚΠΫϩαʔϏε (Wikipedia) https://ja.wikipedia.org/wiki/ϚΠΫϩαʔϏε
6. !6 ϞϊϦγοΫ v.s. ϚΠΫϩαʔϏε
7. !7 ϚΠΫϩαʔϏεʁʁʁ
8. !8 Elastic Stackͱ͸ʁ
9. Elastic Stack อଘɺݕࡧɺ෼ੳ Elasticsearch ՄࢹԽɺ؅ཧ Kibana Beats ΠϯδΣετ Logstash
10. Metrics Logging APM Site
 Search Application Search Business
 Analytics Enterprise
 Search Security
 Analytics Future ιϦϡʔγϣϯ อଘɺݕࡧɺ෼ੳ ՄࢹԽɺ؅ཧ ΠϯδΣετ Kibana Elasticsearch Beats Logstash Elastic Stack
11. Metrics Logging APM Site
 Search App
 Search Business
 Analytics Enterprise
 Search Security
 Analytics Future ιϦϡʔγϣϯ SaaS Elastic Cloud Self Managed Elastic Cloud
 Enterprise Standalone σϓϩΠ อଘɺݕࡧɺ෼ੳ ՄࢹԽɺ؅ཧ ΠϯδΣετ Kibana Elasticsearch Beats Logstash Elastic Stack
12. อଘɺݕࡧɺ෼ੳ Elasticsearch ՄࢹԽɺ؅ཧ Kibana Beats ΠϯδΣετ Logstash Metrics Logging APM Site
 Search Application Search Business
 Analytics Enterprise
 Search Security
 Analytics Future ιϦϡʔγϣϯ SaaS Elastic Cloud Self Managed Elastic Cloud
 Enterprise Standalone σϓϩΠ Elastic Stack
13. !13
14. 14 Beats ܰྔσʔλγούʔ ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू໿ ม׵ͱύʔεͷͨΊ Logstashʹసૹ Elastic Cloudʹసૹ Libbeat: ΧελϜbeatsͷͨ ΊͷAPIϑϨʔϜϫʔΫ 30Ҏ্ͷίϛϡχςΟbeats
15. The Beats family Heartbeat Uptime monitoring Filebeat Log files Winlogbeat Windows Event Logs Packetbeat Network data +40 community Beats Metricbeat Metrics Auditbeat Audit data
16. !16
17. 17 Logstash σʔλՃ޻ύΠϓϥΠϯ શͯͷܗࣜɺαΠζͱσʔλιʔ εͷ౤ೖ ύʔεͱಈతͳ σʔλม׵ ͋ΒΏΔग़ྗʹ σʔλసૹ ҆શͰ҉߸Խ͞Εͨ
 σʔλೖྗ ಠࣗͷύΠϓϥΠϯॲཧ ͷ࡞੒ 200Ҏ্ͷϓϥάΠϯ
18. !18
19. 19 Elasticsearch Heart of the Elastic Stack ෼ࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ ։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ
20. Elasticsearchͱ͸ʁ
21. ϑϦʔϫʔυݕࡧ !21
22. ߜΓࠐΈ !22
23. ϋΠϥΠτ !23
24. ιʔτ !24
25. ϖʔδϯά !25
26. ूܭ !26
27. αδΣετ !27
28. Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ !28
29. ؆୯ͳCRUD
30. σʔλొ࿥ 30 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
31. σʔλߋ৽ 31 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
32. σʔλ࡟আ !32 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
33. ݕࡧ - Query DSL !33 curl -XGET ‘localhost:9200/books/doc/_search' -d '{ "query": { "bool": { "must": [ { "match": { "title": "Search" }}, { "match": { "content": "Elasticsearch" }} ], "filter": [ { "term": { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ] } } }'
34. ෼ࢄߏ੒ɺ
 εέʔϧ
35. γϟʔυͱϨϓϦΧ !35 node 1 orders products 1 4 1 2 2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
36. γϟʔυͱϨϓϦΧ !36 node 1 orders products 1 4 1 node 2 orders products 2 2 3 4 1 2 3
37. ࣗಈతͳ෼ࢄ !37 node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3
38. !38
39. 39 Kibana Window into the Elastic Stack ՄࢹԽͱ෼ੳ ஍ཧۭؒ ΧελϚΠζͱ Ϩϙʔτͷڞ༗ άϥϑ୳ࡧ Elastic Stack΁ͷ ηΩϡΞͳΞΫηεͱ؅ཧ ΧελϜAppsͷ࡞੒
40. !40 Kibana 6
41. !41 Elastic Stackͷߏ੒ Beats Log Files Metrics Wire Data your{beat} Kibana Instances Kafka Distributed Message Queue Notification Queues Storage Metrics Data Store Web APIs Social Sensors Elasticsearch Nodes Logstash Nodes
42. !42 ΞϓϦέʔγϣϯͷ
 ؂ࢹϙΠϯτ
43. !43 ؂ࢹϙΠϯτ • ֎ܗ؂ࢹ • ϝτϦοΫʢϝτϦΫεʣ • αʔόʔɺΞϓϦέʔγϣϯ • ϩά • ΞϓϦέʔγϣϯͷϦϦʔελΠϛϯά • ෼ࢄτϨʔγϯά
44. !44 ֎ܗ؂ࢹ • ࢮ׆؂ࢹ • ϓϩηε • HTTPαʔόʔ • TCP • ICMP
45. Lightweight Shipper for Uptime Monitoring Heartbeat
46. !46 ϝτϦοΫ • αʔόʔ • CPUɺϝϞϦɺσΟεΫɺωοτϫʔΫI/Oɺϓϩηε਺ • ΞϓϦέʔγϣϯ • ϦΫΤετ਺ɺίωΫγϣϯ਺ɺॲཧ࣌ؒ • ίϯςφʔ • ίϯςφ਺
47. Collect system and application metrics Metricbeat
48. lots of modules Metricbeat
49. !49 Metricbeat Ϟδϡʔϧ • Aerospike module • Apache module • Ceph module • Couchbase module • Docker module • Dropwizard module • Elasticsearch module • Etcd module • Golang module • Graphite module • HAProxy module • HTTP module • Jolokia module • Kafka module • Kibana module • Kubernetes module • kvm module • Logstash module • Memcached module • MongoDB module • Munin module • MySQL module • Nginx module • • PHP_FPM module • PostgreSQL module • Prometheus module • RabbitMQ module • Redis module • System module • uwsgi module • vSphere module • Windows module • ZooKeeper module

50. !50 ϩά • ΞΫηεϩά • γεςϜϩά • ೝূϩά • εϩʔϩά • ΞϓϦέʔγϣϯϩά
51. tail log from file Filebeat
52. many modules Filebeat
53. Filebeat modules - v6.4.2 • Apache2 module • Auditd module • Icinga module • IIS module • Kafka module • Logstash module • MongoDB module • MySQL module • Nginx module • Osquery module • PostgreSQL module • Redis module • System module • Traefik module
54. Welcome to 1998 winlogbeat
55. Now winlogbeat
56. Capture the Packet Packetbeat
57. Capture the Packet Packetbeat
58. !58 ΞϓϦέʔγϣϯͷϦϦʔελΠϛϯά • όάϑΟοΫεϦϦʔε • ৽ػೳϦϦʔε • ৽αʔϏε։࢝ • αʔόʔ૿ڧ
59. !59 ෼ࢄτϨʔγϯά • ϚΠΫϩαʔϏε • 1ͭͷϦΫΤετʹରͯ͠ෳ਺ͷϓϩηε͕ؔ܎ • ΞϓϦέʔγϣϯύϑΥʔϚϯεϞχλϦϯάͷ1ͭ
60. !60
61. !61 Elastic APM
62. Distributed Tracing Beta | Basic (free) શͯͷܭଌ͞ΕͨαʔϏεΛݟΔͨΊͷ ౷߹͞ΕͨϏϡʔ αϒίϯςΩετ಺ͷτϨʔεʹભҠ OpenTracing ޓ׵
63. !63 ͦͷ΄͔ͷศརͳػೳ • Infra UI • Logs UI • Machine Learning • Alerting
64. Infrastructure Solution Beta | Basic (free) ΠϯϑϥΦϖϨʔλʔ޲͚ʹಛԽ
    Λ௒͑ΔΠϯϑϥͷߏ੒Λ၆ᛌ
    ,VCFSOFUFTɺ%PDLFS
    ͷωΠςΟϒαϙʔ τ
    ϝτϦοΫɺϩάɺ"1.
    Ϗϡʔ΁ͷ
 υϦϧɾμ΢ϯ
    ΞυϗοΫ͓Αͼߏ଄Խݕࡧ
    
65. Logs Solution Beta | Basic (free) ϥΠϒͰϩάͷτϥϒϧγϡʔςΟϯά Λॿ͚ΔܰྔͳϩάϏϡʔΞʔ
    ίϯιʔϧͷΑ͏ͳදࣔ
    UBJM
    GͷΑ͏ͳ ϥΠϒɾϩάɾετϦʔ ϛϯά
    
    ཤྺϩάͷແݶεΫϩʔϧ
    ΞυϗοΫ͓Αͼߏ଄Խݕࡧ
    
66. !66
67. !67
68. !68
69. !69 ͞Βʹ׆༻͢Δʹ͸ʁ
70. !70
71. !71
72. ͦͷଞͷ࢖͍ํ !72
73. !73 σʔλͷొ࿥ํ๏ • Kibanaͷαϯϓϧσʔλʢ6.4͔Βʣ • LogstashͰJDBC input • LogstashͰCSV • FilebeatͰΞΫηεϩά • MetricbeatͰϝτϦοΫ • PacketbeatͰMySQL/PostgreSQLͷύέοτղੳ
74. !74 Kibanaͷαϯϓϧσʔλʢ>= 6.4.0ʣ
75. !75 ϫϯΫϦοΫͰσʔλొ࿥
76. !76 LogstashͰJDBC Input Kibana Instances Data Store Elasticsearch Nodes Logstash Nodes
77. !77 JDBC Input
78. !78 LogstashͰCSV Kibana Instances CSV
 File Elasticsearch Nodes Logstash Nodes
79. !79 CSV filter
80. !80 FilebeatͰΞΫηεϩά Beats Log Files Kibana Instances Elasticsearch Nodes
81. • 2ͭͷElasticsearchϓϥάΠϯΛΠϯετʔϧͯ͠ElasticsearchΛىಈ • Filebeatͷapache2ϞδϡʔϧΛ༗ޮԽ • modules.d/apache2.ymlʹΞΫηεϩάͷύεΛઃఆ • setupίϚϯυΛ࣮ߦ͔ͯ͠ΒFilebeatΛىಈ !81 FilebeatͰΞΫηεϩά
82. MetricbeatͰϝτϦοΫ Beats Metrics Kibana Instances Elasticsearch Nodes
83. • MetricbeatͷsystemϞδϡʔϧΛ༗ޮԽ • setupίϚϯυΛ࣮ߦ͔ͯ͠ΒFilebeatΛىಈ !83 MetricbeatͰϝτϦοΫ
84. !84 PacketbeatͰMySQLɺPostgreSQLͷύέοτղੳ Beats Wire Data Kibana Instances Elasticsearch Nodes
85. !85 ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/ index.html • ॻ੶ʢ೔ຊޠʣ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ‒ Elasticsearch࣮ફΨΠυ
86. !86 ࢀߟจݙ • ೖ໳ ؂ࢹ
 ―ϞμϯͳϞχλϦϯάͷͨΊͷσβΠϯύλʔϯ
 Mike JulianɹஶɺদӜ ൏ਓɹ༁
 
 https://www.oreilly.co.jp/books/9784873118642/
87. !87 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions
88. Thank you! • Web : https://www.elastic.co/jp/ • Forums : https://discuss.elastic.co/ • Twitter : @johtani