Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Life with Datadog

Takeshi Kondo
January 24, 2021

Life with Datadog

July Tech Festa 2021 winter
https://techfesta.connpass.com/event/193966/

Takeshi Kondo

January 24, 2021
Tweet

More Decks by Takeshi Kondo

Other Decks in Technology

Transcript

  1. Life with Datadog
    Takeshi Kondo / @chaspy
    2021/01/24
    July Tech Festa Winter 2021

    View Slide

  2. Who am I
    chaspy chaspy_
    Lead Software Engineer

    Site Reliability at Quipper
    Takeshi Kondo

    View Slide

  3. Japan Datadog User Group Organizer
    • https://datadog-jp.connpass.com/

    View Slide

  4. Software Design 2݄߸ʹDatadog ϋϯζΦϯهࣄΛدߘ
    • ʲୈ2ಛूʳγεςϜ؂ࢹͷ࢝Ίํɾଓ͚ํ ୈ3ষ Datadog
    Ͱ࣮ફ͢ΔSaaS؂ࢹ
    https://twitter.com/chaspy_/status/1350270946767081472?s=20
    Datadog ॳ৺ऀͷํ͸ͥͻʂ

    View Slide

  5. TγϟπͱۺԼ͍͖ͨͩ·ͨ͠
    https://twitter.com/chaspy_/status/1352577570278043649?s=20
    ཉ͍͠ͻͱ͸ meetup Ͱ࿩ͨ͠Γ
    ϞϒϓϩࢀՃ͢Δͱ΋Β͑Δ͔΋ʂ

    View Slide

  6. ࠓ೔͸ Datadog ΁ͷѪΛޠΓ·͢
    https://www.datadoghq.com/about/resources/
    Monitoring SaaS

    View Slide

  7. ࠓ೔ͷൃද
    • ର৅
    • γεςϜӡ༻ऀ
    • Monitoring SaaS ʹڵຯ͕͋Δͻͱ
    • Datadog ʹڵຯ͕͋Δͻͱ
    • ΰʔϧ
    • Datadog ͷྑ͞Λ஌Δ
    • Datadog ͷ༷ʑͳԠ༻ػೳΛ஌Δ͜ͱͰ׆༻ͷώϯτΛಘΔ

    View Slide

  8. Agenda
    • Datadog ͷ޷͖ͳͱ͜Ζ
    • ϓϩμΫτѪᷓΕΔαϙʔτ
    • ϦονͰΩϡʔτͳ GUI
    • SLO Summary & Alert
    • Datadog ͱͷੜ׆
    • Monitor Summary
    • Weekly Summary
    • Alfred ͱͷ૊Έ߹Θͤ
    • Datadog ͱͷੜ׆ Ԡ༻ฤ
    • Datadog Cost ͱͷ޲͖߹͍ํ
    • ৽نαʔϏεͱ Datadog
    • Custom Metrics ͷ׆༻

    View Slide

  9. Agenda
    • Datadog ͷ޷͖ͳͱ͜Ζ
    • ϓϩμΫτѪᷓΕΔαϙʔτ
    • ϦονͰΩϡʔτͳ GUI
    • SLO Summary & Alert
    • Datadog ͱͷੜ׆
    • Monitor Summary
    • Weekly Summary
    • Alfred ͱͷ૊Έ߹Θͤ
    • Datadog ͱͷੜ׆ Ԡ༻ฤ
    • Datadog Cost ͱͷ޲͖߹͍ํ
    • ৽نαʔϏεͱ Datadog
    • Custom Metrics ͷ׆༻

    View Slide

  10. Datadog ͷ޷͖ͳͱ͜Ζɿαϙʔτ͕͍͢͝
    • Ϩεϙϯε͕ૣ͍(΄΅࣮֬ʹ24hҎ಺ʹؼͬͯ͘Δʣ
    • ճ౴͕త֬
    • Ͱ͖ͳ͍͜ͱͰ΋ϫʔΫΞϥ΢ϯυΛҊ಺ͯ͘͠ΕΔ
    • ϓϩμΫτΛྑ͘͠Α͏ͱ͍͏࢟੎͕ݟ͑Δ

    View Slide

  11. αϙʔτʹ͓ئ͍͢Δํ๏
    • جຊνέοτΛىථͯ͠Δʢӳޠʣ
    • Help -> resource -> Tickets & Email Support
    • Ͳ͏ͯ͠΋ӳޠແཧͳΒ೔ຊޠͰͷճ౴Λ͓ئ͍͢Ε͹೔ຊ
    αϙʔτʹӦۀ࣌ؒͯ͘͠ΕΔΒ͍͠
    • Event Stream ʹ @support-datadog Λଧͯ͹ live chat ΋Ͱ
    ͖ΔΒ͍͠ʢ΍ͬͨ͜ͱͳ͍͕ʣ

    View Slide

  12. աڈͷ࣭໰ྫ
    • Terraform Ͱ SLO ͷ Apply ͕ Invalid Query ͰౖΒΕΔ
    • Monitor Summary Λ metrics ͱͯ͠औΕͳ͍͔૬ஊ
    • EKS Control Plane ͷ metrics औಘํ๏
    • SLO ҧ൓ͨ͠ͱ͖ʹΞϥʔτΛඈ͹͢ํ๏
    • API Key ͷ਺Λ૿΍ͤͳ͍͔૬ஊ

    View Slide

  13. աڈͷ࣭໰ྫ
    • Terraform Ͱ SLO ͷ Apply ͕ Invalid Query ͰౖΒΕΔ
    • Monitor Summary Λ metrics ͱͯ͠औΕͳ͍͔૬ஊ
    • EKS Control Plane ͷ metrics औಘํ๏
    • SLO ҧ൓ͨ͠ͱ͖ʹΞϥʔτΛඈ͹͢ํ๏
    • API Key ͷ਺Λ૿΍ͤͳ͍͔૬ஊ

    View Slide

  14. Monitor Summary Λ metrics ͱͯ͠औΕͳ͍͔૬ஊ
    • Monitor Summary ஌ͬͯ·͔͢ʁ
    • िʹҰ౓ϝʔϧདྷͯΔ΍ͭ
    • https://app.datadoghq.com/report/monitor
    • ࣮͸ GUI ͔Βಋઢ͕ͳ͍ʢϝʔϧ͔Β͔͠ඈ΂ͳ͍ʣ

    View Slide

  15. Monitor Summary

    View Slide

  16. Monitor Summary
    ೚ҙͷνϟϯωϧͷΞϥʔτ
    ͚ͩϑΟϧλ͍ͨ͠

    View Slide

  17. Raw data(CSV) ͕ api ͰऔΕΔΒ͍͠
    • CSV ͕μ΢ϯϩʔυͰ͖Δ
    • ͜ͷ CSV ͔Β custom metrics ΛૹΔίʔυΛॻ͍ͨ
    https://app.datadoghq.com/report/hourly_data/monitor
    https://docs.datadoghq.com/monitors/faq/how-can-i-export-alert-history/?tab=us

    View Slide

  18. SLO ҧ൓ͨ͠ͱ͖ʹΞϥʔτΛඈ͹͢ํ๏
    • ౰࣌͸·ͩ SLO Error Budget Alert ͷػೳ͸ͳ͔ͬͨ
    • Product Manager ͕ग़ͯ͘Δɻͦͷػೳ͸ઈࢍ։ൃதͰɺ
    ϢʔβΠϯλϏϡʔΛͤͯ͘͞Εͳ͍͔ͱݴͬͯ͘Δ
    • Support Ticket ্Ͱ͍͔࣭ͭ͘໰ʹ౴͑Δ
    • ͦͷޙແࣄ SLO Alert Beta ϦϦʔε!

    View Slide

  19. Τϯήʔδϝϯτͷߴ·Γ

    View Slide

  20. ϦονͰΩϡʔτͳGUI

    View Slide

  21. ϦονͰΩϡʔτͳGUI
    • ՄࢹԽͷํ๏͕ଟ͍
    • ͙͢ʹ൓ө͞ΕΔ
    • ؔ਺͕ଟ͍
    • ίϐϖָ͕

    View Slide

  22. ࣮ࡍʹάϥϑΛ͍ͬͯ͡ΈΑ͏

    View Slide

  23. SLO Summary & Alert

    View Slide

  24. SLO Summary & Alert

    View Slide

  25. Datadog ͷ SLO ػೳ͸ͳͯ͘͸ͳΒͳ͍૬๮
    https://quipper.hatenablog.com/entry/2020/01/30/slo-review

    View Slide

  26. Terraform ؅ཧ & generator Ͱ৽نαʔϏε࡞
    ੒࣌ʹ΋؆୯ʹ࡞੒Ͱ͖Δ
    ࣮ࡍʹݟͯΈΑ͏

    View Slide

  27. Agenda
    • Datadog ͷ޷͖ͳͱ͜Ζ
    • ϓϩμΫτѪᷓΕΔαϙʔτ
    • ϦονͰΩϡʔτͳ GUI
    • SLO Summary & Alert
    • Datadog ͱͷੜ׆
    • Monitor Summary
    • Weekly Summary
    • Alfred ͱͷ૊Έ߹Θͤ
    • Datadog ͱͷੜ׆ Ԡ༻ฤ
    • Datadog Cost ͱͷ޲͖߹͍ํ
    • ৽نαʔϏεͱ Datadog
    • Custom Metrics ͷ׆༻

    View Slide

  28. Monitor Summary Λຖே Daily Standup Ͱݟ͍ͯΔ

    View Slide

  29. Monitor Summary
    • >muted:false tag:(("severity:alert" OR
    "severity:emergency") AND "team:sre")
    • ͢΂ͯͷ Alert ʹ severity tag ͱ team tag ͕͍͍ͭͯΔ
    • ࣗ෼ͨͪͷ୲౰ͷΞϥʔτͷΈΛ Daily Standup Ͱ Check
    • NG ͩͬͨΒ೔ຖͷ୲౰ऀ͕ରॲ͢Δ

    View Slide

  30. Weekly Summary
    • ͜͏͍͏ϝʔϧ͕དྷ͍ͯ·͢
    • [Monitor Report] You received xxx alerts

    View Slide

  31. Weekly Summary
    • ࣮͸͜͜ͰݟΕΔ
    • https://app.datadoghq.com/report/monitor
    • ΫϦοΫ͢Δͱि΍࣌ؒଳ΍Ξϥʔτνϟϯωϧɺmonitor
    ͰߜΓࠐΈ͕Ͱ͖Δ
    • ͜ΕΛݟͯΞϥʔτͷ਺ࣗମͷਪҠΛ௥͍ͬͯΔ

    View Slide

  32. Alfred ͱͷ૊Έ߹Θͤ
    • Dashboard, monitor, SLO Λαοͱ։͚ΔΑ͏ʹ
    • Web Search ొ࿥͍ͯ͠Δ
    • ddd https://app.datadoghq.com/dashboard/lists?q={query}
    • ddm https://app.datadoghq.com/monitors/manage?q={query}
    • ddslo https://app.datadoghq.com/slo?query={query}

    View Slide

  33. Agenda
    • Datadog ͷ޷͖ͳͱ͜Ζ
    • ϓϩμΫτѪᷓΕΔαϙʔτ
    • ϦονͰΩϡʔτͳ GUI
    • SLO Summary & Alert
    • Datadog ͱͷੜ׆
    • Monitor Summary
    • Weekly Summary
    • Alfred ͱͷ૊Έ߹Θͤ
    • Datadog ͱͷੜ׆ Ԡ༻ฤ
    • Datadog Cost ͱͷ޲͖߹͍ํ
    • ৽نαʔϏεͱ Datadog
    • Custom Metrics ͷ׆༻

    View Slide

  34. Datadog Cost ͷܭࢉࣜ
    • ݄ؒͷϗετ਺ͷ 99 percentile (max Ͱ͸ͳ͍)
    • ϗετ਺Λ࡟ݮ͢Δͱ Datadog Cost ΋ݮΔ
    • ϗετ਺ΛݮΒ͔͢͠ͳ͍ʢԦಓͳ͠ʣ

    View Slide

  35. ৽نαʔϏεͱ Datadog
    • ৽نαʔϏεߏங࣌ʹ͸"Production Readiness Checklist"
    ͕͋Δ
    https://quipper.hatenablog.com/entry/2020/01/30/production-readiness-with-all

    View Slide

  36. Production Readiness Checklist - Monitoring / Logging

    View Slide

  37. Template Dashboard

    View Slide

  38. ࣮ࡍʹ Template Dashboard ΛݟͯΈΑ͏

    View Slide

  39. Production Readiness Checklist - SLI/SLO
    • Generator Λ࢖ͬͯ࡞ͬͯ Dashboard ʹ௥Ճ͢Δ͚ͩʂ
    • SLO ࣗମΛԿʹ͢Δ͔͸ Design Doc ϑΣʔζͰࡦఆ͢Δ

    View Slide

  40. Custom metrics ͷ׆༻ྫ
    • CI Ͱ͔͔ͬͨ࣌ؒΛଌఆ
    • Docker build ͷ࣌ؒɺ֤ςετͷ࣌ؒͳͲ
    • CircleCI ͷίετΛଌఆɺՄࢹԽ
    • Dependabot ʹΑͬͯ open ʹͳͬͯΔ PR ਺ΛՄࢹԽ
    • εύΠΫΞΫηεΛ๷͙ͨΊʹෛՙ৘ใΛ custom metrics ͱͯ͠ૹ
    ΓɺKubernetes HPA ͔Βར༻

    View Slide

  41. Kubernetes meetup tokyo #38 Ͱ࿩͠·ͨ͠
    https://twitter.com/chaspy_/status/1352194206962388995?s=20
    https://speakerdeck.com/chaspy/v2beta2-and-examples-of-using-hpa-external-metrics-with-datadog
    https://quipper.hatenablog.com/entry/2020/11/30/scheduled-scaling-with-hpa

    View Slide

  42. Custom metric ͱͯ͠਎ͷճΓͷ৘ใΛૹΔϝϦοτ
    • ݱঢ়ΛՄࢹԽͰ͖Δ
    • ೚ҙͷλά౳ͰϑΟϧλ/আ֎Ͱ͖Δ
    • Metric ͔ΒΞϥʔτΛൃ๒Ͱ͖Δ

    View Slide

  43. Custom metrics ΛૹΔ design pattern
    • Kubernetes ؀ڥͩͱ Autodiscovery ػೳ͕ศར
    https://docs.datadoghq.com/agent/kubernetes/integrations/?tab=kubernetes
    API ͕͋Δ֎෦γεςϜ
    Custom metric ੜ੒͢Δ܅
    (Kubernetes Deployment)
    API Ͱσʔλऔಘ
    Datadog-agent
    http Ͱ prometheus ܗࣜͰ
    metric Λ export

    View Slide

  44. Custom metrics ΛૹΔ design pattern
    • Kubernetes ؀ڥͩͱ Autodiscovery ػೳ͕ศར
    https://docs.datadoghq.com/agent/kubernetes/integrations/?tab=kubernetes
    API ͕͋Δ֎෦γεςϜ
    Custom metric ੜ੒͢Δ܅
    (Kubernetes Deployment)
    API Ͱσʔλऔಘ
    Datadog-agent
    1 annotations:
    2 ad.datadoghq.com/timed-exam-schedule-exporter.check_names: |
    3 ["prometheus"]
    4 ad.datadoghq.com/timed-exam-schedule-exporter.init_configs:
    |
    5 [{}]
    6 ad.datadoghq.com/timed-exam-schedule-exporter.instances: |
    7 [
    8 {
    9 "prometheus_url": "http://%%host%%:8080/metrics",
    10 "namespace": "namespace",
    11 "metrics": ["*"]
    12 }
    13 ]

    View Slide

  45. Custom metrics ΛૹΔ design pattern
    • Kubernetes ؀ڥͩͱ Autodiscovery ػೳ͕ศར
    https://docs.datadoghq.com/agent/kubernetes/integrations/?tab=kubernetes
    API ͕͋Δ֎෦γεςϜ
    Custom metric ੜ੒͢Δ܅
    (Kubernetes Deployment)
    API Ͱσʔλऔಘ
    Datadog-agent
    • GitHub ͷ PR ΍ issue
    • AWS Security Event(GuardDuty / ECR
    Image Scan)
    • OS ͝ͱͷ EC2 Horst ਺
    • RDS ͷ version ͝ͱͷΫϥελ਺

    View Slide

  46. ·ͱΊ
    • Datadog Ͱ෼͔Βͳ͍͜ͱ͕͋ͬͨΒ...
    • ͙͢ʹ Support ʹฉ͜͏ʂૣͯ͘ஸೡʂ
    • Japan Datadog User Group ͷ Slack Ͱฉ͍ͯ΋ okʂ
    • Monitor Summary ͱ Weekly Report Ͱ Monitor ࣗମ΋ݮΒͯ͠
    ͍͜͏
    • Custom metrics ͦ͜ Datadog ͷ୉ޣຯɺͲΜͲΜ׆༻͠Α͏ʂ

    View Slide

  47. More
    • ࣮ࡍͷάϥϑͱ͔ݟ͍ͨͻͱ͸࿥ըͷͳ͍࣌ʹདྷͯʂ
    • ͜ͷ͋ͱͷ zoom breakout room Ͱ΋Α͠
    • ݸผʹ Twitter Ͱ࿈བྷ͘ΕΕ͹ Datadog ૬ஊ৐Γ·͢

    View Slide

  48. એ఻

    View Slide

  49. Thank you!
    chaspy chaspy_
    Lead Software Engineer

    Site Reliability at Quipper
    Takeshi Kondo

    View Slide