July Tech Festa 2021 winter https://techfesta.connpass.com/event/193966/
Life with DatadogTakeshi Kondo / @chaspy2021/01/24July Tech Festa Winter 2021
View Slide
Who am Ichaspy chaspy_Lead Software EngineerSite Reliability at QuipperTakeshi Kondo
Japan Datadog User Group Organizer• https://datadog-jp.connpass.com/
Software Design 2݄߸ʹDatadog ϋϯζΦϯهࣄΛدߘ• ʲୈ2ಛूʳγεςϜࢹͷ࢝Ίํɾଓ͚ํ ୈ3ষ DatadogͰ࣮ફ͢ΔSaaSࢹhttps://twitter.com/chaspy_/status/1350270946767081472?s=20Datadog ॳ৺ऀͷํͥͻʂ
TγϟπͱۺԼ͍͖ͨͩ·ͨ͠https://twitter.com/chaspy_/status/1352577570278043649?s=20ཉ͍͠ͻͱ meetup Ͱͨ͠ΓϞϒϓϩࢀՃ͢ΔͱΒ͑Δ͔ʂ
ࠓ Datadog ͷѪΛޠΓ·͢https://www.datadoghq.com/about/resources/Monitoring SaaS
ࠓͷൃද• ର• γεςϜӡ༻ऀ• Monitoring SaaS ʹڵຯ͕͋Δͻͱ• Datadog ʹڵຯ͕͋Δͻͱ• ΰʔϧ• Datadog ͷྑ͞ΛΔ• Datadog ͷ༷ʑͳԠ༻ػೳΛΔ͜ͱͰ׆༻ͷώϯτΛಘΔ
Agenda• Datadog ͷ͖ͳͱ͜Ζ• ϓϩμΫτѪᷓΕΔαϙʔτ• ϦονͰΩϡʔτͳ GUI• SLO Summary & Alert• Datadog ͱͷੜ׆• Monitor Summary• Weekly Summary• Alfred ͱͷΈ߹Θͤ• Datadog ͱͷੜ׆ Ԡ༻ฤ• Datadog Cost ͱͷ͖߹͍ํ• ৽نαʔϏεͱ Datadog• Custom Metrics ͷ׆༻
Datadog ͷ͖ͳͱ͜Ζɿαϙʔτ͕͍͢͝• Ϩεϙϯε͕ૣ͍(΄΅࣮֬ʹ24hҎʹؼͬͯ͘Δʣ• ճ͕త֬• Ͱ͖ͳ͍͜ͱͰϫʔΫΞϥϯυΛҊͯ͘͠ΕΔ• ϓϩμΫτΛྑ͘͠Α͏ͱ͍͏͕࢟ݟ͑Δ
αϙʔτʹ͓ئ͍͢Δํ๏• جຊνέοτΛىථͯ͠Δʢӳޠʣ• Help -> resource -> Tickets & Email Support• Ͳ͏ͯ͠ӳޠແཧͳΒຊޠͰͷճΛ͓ئ͍͢ΕຊαϙʔτʹӦۀ࣌ؒͯ͘͠ΕΔΒ͍͠• Event Stream ʹ @support-datadog Λଧͯ live chat Ͱ͖ΔΒ͍͠ʢͬͨ͜ͱͳ͍͕ʣ
աڈͷ࣭ྫ• Terraform Ͱ SLO ͷ Apply ͕ Invalid Query ͰౖΒΕΔ• Monitor Summary Λ metrics ͱͯ͠औΕͳ͍͔૬ஊ• EKS Control Plane ͷ metrics औಘํ๏• SLO ҧͨ͠ͱ͖ʹΞϥʔτΛඈ͢ํ๏• API Key ͷΛ૿ͤͳ͍͔૬ஊ
Monitor Summary Λ metrics ͱͯ͠औΕͳ͍͔૬ஊ• Monitor Summary ͬͯ·͔͢ʁ• िʹҰϝʔϧདྷͯΔͭ• https://app.datadoghq.com/report/monitor• ࣮ GUI ͔Βಋઢ͕ͳ͍ʢϝʔϧ͔Β͔͠ඈͳ͍ʣ
Monitor Summary
Monitor SummaryҙͷνϟϯωϧͷΞϥʔτ͚ͩϑΟϧλ͍ͨ͠
Raw data(CSV) ͕ api ͰऔΕΔΒ͍͠• CSV ͕μϯϩʔυͰ͖Δ• ͜ͷ CSV ͔Β custom metrics ΛૹΔίʔυΛॻ͍ͨhttps://app.datadoghq.com/report/hourly_data/monitorhttps://docs.datadoghq.com/monitors/faq/how-can-i-export-alert-history/?tab=us
SLO ҧͨ͠ͱ͖ʹΞϥʔτΛඈ͢ํ๏• ࣌·ͩ SLO Error Budget Alert ͷػೳͳ͔ͬͨ• Product Manager ͕ग़ͯ͘Δɻͦͷػೳઈࢍ։ൃதͰɺϢʔβΠϯλϏϡʔΛͤͯ͘͞Εͳ͍͔ͱݴͬͯ͘Δ• Support Ticket ্Ͱ͍͔࣭ͭ͘ʹ͑Δ• ͦͷޙແࣄ SLO Alert Beta ϦϦʔε!
Τϯήʔδϝϯτͷߴ·Γ
ϦονͰΩϡʔτͳGUI
ϦονͰΩϡʔτͳGUI• ՄࢹԽͷํ๏͕ଟ͍• ͙͢ʹө͞ΕΔ• ͕ؔଟ͍• ίϐϖָ͕
࣮ࡍʹάϥϑΛ͍ͬͯ͡ΈΑ͏
SLO Summary & Alert
Datadog ͷ SLO ػೳͳͯ͘ͳΒͳ͍૬https://quipper.hatenablog.com/entry/2020/01/30/slo-review
Terraform ཧ & generator Ͱ৽نαʔϏε࡞࣌ʹ؆୯ʹ࡞Ͱ͖Δ࣮ࡍʹݟͯΈΑ͏
Monitor Summary Λຖே Daily Standup Ͱݟ͍ͯΔ
Monitor Summary• >muted:false tag:(("severity:alert" OR"severity:emergency") AND "team:sre")• ͯ͢ͷ Alert ʹ severity tag ͱ team tag ͕͍͍ͭͯΔ• ࣗͨͪͷ୲ͷΞϥʔτͷΈΛ Daily Standup Ͱ Check• NG ͩͬͨΒຖͷ୲ऀ͕ରॲ͢Δ
Weekly Summary• ͜͏͍͏ϝʔϧ͕དྷ͍ͯ·͢• [Monitor Report] You received xxx alerts
Weekly Summary• ࣮͜͜ͰݟΕΔ• https://app.datadoghq.com/report/monitor• ΫϦοΫ͢Δͱि࣌ؒଳΞϥʔτνϟϯωϧɺmonitorͰߜΓࠐΈ͕Ͱ͖Δ• ͜ΕΛݟͯΞϥʔτͷࣗମͷਪҠΛ͍ͬͯΔ
Alfred ͱͷΈ߹Θͤ• Dashboard, monitor, SLO Λαοͱ։͚ΔΑ͏ʹ• Web Search ొ͍ͯ͠Δ• ddd https://app.datadoghq.com/dashboard/lists?q={query}• ddm https://app.datadoghq.com/monitors/manage?q={query}• ddslo https://app.datadoghq.com/slo?query={query}
Datadog Cost ͷܭࢉࣜ• ݄ؒͷϗετͷ 99 percentile (max Ͱͳ͍)• ϗετΛݮ͢Δͱ Datadog Cost ݮΔ• ϗετΛݮΒ͔͢͠ͳ͍ʢԦಓͳ͠ʣ
৽نαʔϏεͱ Datadog• ৽نαʔϏεߏங࣌ʹ"Production Readiness Checklist"͕͋Δhttps://quipper.hatenablog.com/entry/2020/01/30/production-readiness-with-all
Production Readiness Checklist - Monitoring / Logging
Template Dashboard
࣮ࡍʹ Template Dashboard ΛݟͯΈΑ͏
Production Readiness Checklist - SLI/SLO• Generator Λͬͯ࡞ͬͯ Dashboard ʹՃ͢Δ͚ͩʂ• SLO ࣗମΛԿʹ͢Δ͔ Design Doc ϑΣʔζͰࡦఆ͢Δ
Custom metrics ͷ׆༻ྫ• CI Ͱ͔͔ͬͨ࣌ؒΛଌఆ• Docker build ͷ࣌ؒɺ֤ςετͷ࣌ؒͳͲ• CircleCI ͷίετΛଌఆɺՄࢹԽ• Dependabot ʹΑͬͯ open ʹͳͬͯΔ PR ΛՄࢹԽ• εύΠΫΞΫηεΛ͙ͨΊʹෛՙใΛ custom metrics ͱͯ͠ૹΓɺKubernetes HPA ͔Βར༻
Kubernetes meetup tokyo #38 Ͱ͠·ͨ͠https://twitter.com/chaspy_/status/1352194206962388995?s=20https://speakerdeck.com/chaspy/v2beta2-and-examples-of-using-hpa-external-metrics-with-datadoghttps://quipper.hatenablog.com/entry/2020/11/30/scheduled-scaling-with-hpa
Custom metric ͱͯ͠ͷճΓͷใΛૹΔϝϦοτ• ݱঢ়ΛՄࢹԽͰ͖Δ• ҙͷλάͰϑΟϧλ/আ֎Ͱ͖Δ• Metric ͔ΒΞϥʔτΛൃ๒Ͱ͖Δ
Custom metrics ΛૹΔ design pattern• Kubernetes ڥͩͱ Autodiscovery ػೳ͕ศརhttps://docs.datadoghq.com/agent/kubernetes/integrations/?tab=kubernetesAPI ͕͋Δ֎෦γεςϜCustom metric ੜ͢Δ܅(Kubernetes Deployment)API ͰσʔλऔಘDatadog-agenthttp Ͱ prometheus ܗࣜͰmetric Λ export
Custom metrics ΛૹΔ design pattern• Kubernetes ڥͩͱ Autodiscovery ػೳ͕ศརhttps://docs.datadoghq.com/agent/kubernetes/integrations/?tab=kubernetesAPI ͕͋Δ֎෦γεςϜCustom metric ੜ͢Δ܅(Kubernetes Deployment)API ͰσʔλऔಘDatadog-agent1 annotations:2 ad.datadoghq.com/timed-exam-schedule-exporter.check_names: |3 ["prometheus"]4 ad.datadoghq.com/timed-exam-schedule-exporter.init_configs:|5 [{}]6 ad.datadoghq.com/timed-exam-schedule-exporter.instances: |7 [8 {9 "prometheus_url": "http://%%host%%:8080/metrics",10 "namespace": "namespace",11 "metrics": ["*"]12 }13 ]
Custom metrics ΛૹΔ design pattern• Kubernetes ڥͩͱ Autodiscovery ػೳ͕ศརhttps://docs.datadoghq.com/agent/kubernetes/integrations/?tab=kubernetesAPI ͕͋Δ֎෦γεςϜCustom metric ੜ͢Δ܅(Kubernetes Deployment)API ͰσʔλऔಘDatadog-agent• GitHub ͷ PR issue• AWS Security Event(GuardDuty / ECRImage Scan)• OS ͝ͱͷ EC2 Horst • RDS ͷ version ͝ͱͷΫϥελ
·ͱΊ• Datadog Ͱ͔Βͳ͍͜ͱ͕͋ͬͨΒ...• ͙͢ʹ Support ʹฉ͜͏ʂૣͯ͘ஸೡʂ• Japan Datadog User Group ͷ Slack Ͱฉ͍ͯ okʂ• Monitor Summary ͱ Weekly Report Ͱ Monitor ࣗମݮΒ͍ͯ͜͠͏• Custom metrics ͦ͜ Datadog ͷޣຯɺͲΜͲΜ׆༻͠Α͏ʂ
More• ࣮ࡍͷάϥϑͱ͔ݟ͍ͨͻͱըͷͳ͍࣌ʹདྷͯʂ• ͜ͷ͋ͱͷ zoom breakout room ͰΑ͠• ݸผʹ Twitter Ͱ࿈བྷ͘ΕΕ Datadog ૬ஊΓ·͢
એ
Thank you!chaspy chaspy_Lead Software EngineerSite Reliability at QuipperTakeshi Kondo
chaspy
tomzoh
mochico
jnytnai0613
mixi_engineers
prathamesh
yusuke
iwamot
kyonmm
ritou
opeonikute
bengo4com
pauli
eileencodes
nonsquared
mthomps
aarron
hursman
dougneiner
swwweet
bkeepers
carmenintech
addyosmani
productmarketing