$30 off During Our Annual Pro Sale. View Details »

マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog

Daiki Matsui
November 13, 2019

マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog

Datadog Summit Tokyoでの登壇資料です。
動画はhttps://www.youtube.com/watch?v=_Z4HBiwODbc

Daiki Matsui

November 13, 2019
Tweet

More Decks by Daiki Matsui

Other Decks in Technology

Transcript

  1. 1
    ϚϧνΫϥ΢υͳେن໛αʔϏε
    KARTEΛࢧ͑ΔDatadog
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    PLAID, Inc.
    Daiki Matsui

    View Slide

  2. 2
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ࣗݾ঺հ
    দҪ େٍ גࣜձࣾϓϨΠυ
    SRE Senior Architect
    2012೥-2015೥·ͰגࣜձࣾαΠόʔυʹͯήʔϜ։ൃʹैࣄ
    2015೥ΑΓϓϨΠυʹࢀը͠ɺSREͱͯ͠ϚϧνΫϥ΢υΠϯϑϥͷߏஙɾӡ༻౳Λ୲౰
    2015೥ೖࣾ

    View Slide

  3. ɹɹʛɹɹ© 2019 PLAID Inc. 3
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    גࣜձࣾϓϨΠυʹ͍ͭͯ
    גࣜձࣾϓϨΠυ
    ౦ژ౎தԝ۠ۜ࠲6-10-1 GINZA SIX 10F
    ઃཱɿ2011೥10݄
    ैۀһɿ130໊ ࢿຊۚɿ149,987,878ԁ ※ࢿຊ४උؚۚΉ

    View Slide

  4. 4
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    KARTEʹ͍ͭͯ
    ֬ೝ͢Δ
    جຊαʔϏε΍͝ར༻ํ๏Λ͝঺հ͠·͢ɻ
    ͓ಘͳಛయ΍໾ཱͭ৘ใ͕ຬࡌͰ͢ɻ
    ॳΊͯͷํ΁
    ְתּׅ然钠ׅ׷
    6*رؠ؎ش٦׌ֽוזַזַ♳麦׃זְծ➙
    ״׶إٔؗ،حف׃׋ְהְֲ倯䗳铣דׅկ
    UIرؠ؎ش٦ָ濼׏גֶֻץֹ
    7אךرؠ؎ٕٝ٦ٕ
    DESIGN
    RULES
    » CHECK
    ׆׏ה⢪ִ׷،؎ذي׌ֽ׾䲧ִת׃׋կ
    ֿך堣⠓׾ֶ鋅鷕׃זֻկ
    窫㼎ծ
    妜׃ְ
    ౙͷओ໾ɺ
    Ξ΢λʔ
    COLLECTION
    OUTER
    嗚稊勴⟝׾㼰׃㢌刿ׅ׷׌ֽדծ֮ז׋ך椚
    䟝ך暟⟝ח⳿⠓ִ׷〳腉䚍ָ넝ֻז׶תׅկ
    勴⟝׾㢌ִג嗚稊׃ג׫גֻ׌ְׁկ
    ׀䋞劄ך暟⟝כ
    鋅אַ׶תׇ׿ד׃׋ַ
    ꟗׄ׷
    ҰਓҰਓʹ߹Θͤͨ
    ސ٬ମݧΛఏڙ
    WebαΠτͷ๚໰ऀͷߦಈΛ

    ސ٬͝ͱʹϦΞϧλΠϜʹղੳ
    $9 ސ٬ମݧ
    ϓϥοτϑΥʔϜ

    View Slide

  5. 5
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ࿩͢͜ͱ
    KARTEͷಛ௃
    KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ
    ࠓޙ΍Γ͍ͨ͜ͱ
    1
    2
    3

    View Slide

  6. 6
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ࿩͢͜ͱ
    KARTEͷಛ௃
    KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ
    ࠓޙ΍Γ͍ͨ͜ͱ

    View Slide

  7. 7
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEͷಛ௃
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    ࿩͢͜ͱ

    View Slide

  8. 8
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEͷಛ௃
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    ࿩͢͜ͱ

    View Slide

  9. 9
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    Ϛϧνςφϯτ

    View Slide

  10. ɹɹʛɹɹ© 2019 PLAID Inc. 10
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɾ༷ʑͳෛՙͷ͓٬͞Μ͕͍Δ
    ɾΞΫηεύλʔϯ΋৭ʑ
    Ϛϧνςφϯτ

    View Slide

  11. ɹɹʛɹɹ© 2019 PLAID Inc. 11
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɾෛՙ͕1೔Ͱ໿10ഒมԽ͢Δ
    ɾηʔϧ౳Ͱෛՙ͕ٸ૿͢Δ
    Ϛϧνςφϯτ
    1೔ͷෛՙਪҠ

    View Slide

  12. 12
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEͷಛ௃
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    ࿩͢͜ͱ

    View Slide

  13. 13
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ௿ϨΠςϯγʔɺߴෛՙ
    sec to real-time
    actions
    0.x
    events per day
    3 billion
    stored in BigQuery
    2+ PB
    events (rows) / sec
    65,000
    streaming insert per
    month
    1 PB
    analysis per month
    60+ PB
    SRE
    3
    ௿ϨΠςϯγʔ/ߴෛՙ

    View Slide

  14. 14
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ௿ϨΠςϯγʔɺߴෛՙ
    sec to real-time
    actions
    0.x
    events per day
    3 billion
    stored in BigQuery
    2+ PB
    events (rows) / sec
    65,000
    streaming insert per
    month
    1 PB
    analysis per month
    60+ PB
    SRE
    3
    ௿ϨΠςϯγʔ/ߴෛՙ

    View Slide

  15. 15
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ௿ϨΠςϯγʔɺߴෛՙ

    View Slide

  16. 16
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ௿ϨΠςϯγʔɺߴෛՙ
    sec to real-time
    actions
    0.x
    events per day
    3 billion
    stored in BigQuery
    2+ PB
    events (rows) / sec
    65,000
    streaming insert per
    month
    1 PB
    analysis per month
    60+ PB
    SRE
    3
    ௿ϨΠςϯγʔ/ߴෛՙ

    View Slide

  17. 17
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEͷಛ௃
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    ࿩͢͜ͱ

    View Slide

  18. 18
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    େن໛γεςϜ

    View Slide

  19. 19
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEͷಛ௃
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    ࿩͢͜ͱ

    View Slide

  20. 20
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ϚϧνΫϥ΢υߏ੒

    View Slide

  21. ɹɹʛɹɹ© 2019 PLAID Inc. 21
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    Pros
    1. ֤ϓϩόΠμʔͷྑ͍ͱ͜औΓ͕Ͱ͖Δ
    2. ϓϩόΠμʔͷো֐ΛผϓϩόΠμʔͰΧ
    όʔͰ͖Δ
    Pros/Cons
    ϚϧνΫϥ΢υߏ੒
    Cons
    1. ӡ༻ͷख͕ؒ૿͑Δ

    View Slide

  22. ɹɹʛɹɹ© 2019 PLAID Inc. 22
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ৄࡉΛ஌Γ͍ͨํ͸
    ʮσϒαϛ 2018ʯͷࢿྉͰʂ
    ϚϧνΫϥ΢υߏ੒
    https://speakerdeck.com/tikson/zhan-lue-topointonituite

    View Slide

  23. 23
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ࿩͢͜ͱ
    KARTEͷಛ௃
    KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ
    ࠓޙ΍Γ͍ͨ͜ͱ


    View Slide

  24. KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ
    24
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɾϦΞϧλΠϜੑ
    ɾॊೈੑ
    ɾεέʔϥϏϦςΟ
    ɾ҆ఆੑ

    View Slide

  25. 25
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEͷಛ௃ →
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒

    View Slide

  26. 26
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEΛࢧ͑Δ
    ؂ࢹͷϙΠϯτ
    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    ࿩͢͜ͱ

    View Slide

  27. 27
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒

    View Slide

  28. ɹɹʛɹɹ© 2019 PLAID Inc. 28
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ো֐ରԠ࣌ʹ͸DatadogΛݟΕ͹
    ͙͢ʹঢ়گΛ೺ѲͰ͖ΔΑ͏ʹ
    Datadogʹશͯू໿

    View Slide

  29. 29
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEΛࢧ͑Δ
    ؂ࢹͷϙΠϯτ
    ࿩͢͜ͱ

    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard

    View Slide

  30. 30
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒

    View Slide

  31. ɹɹʛɹɹ© 2019 PLAID Inc. 31
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    32M/mon ΄Ͳར༻
    Custom metricsͷੵۃ׆༻

    View Slide

  32. Custom metricsͷੵۃ׆༻
    32
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    https://github.com/makinoy/libs-dogstatsd
    ɾlibs-dogstatsdͰɺ؆୯ʹॲཧͷ࣮ߦճ਺΍
    ࣮ߦ࣌ؒΛଌఆͰ͖ΔΑ͏ʹ
    ɾؔ਺ΑΓ΋ࡉ͔͍୯ҐͰlatencyΛଌఆ
    ɾϢʔβ୯ҐͰͷύϑΥʔϚϯεΛଌఆ

    1 const dd = require('libs-dogstatsd');
    2 const stats = dd.start();
    3 // measure hoge
    4 hoge();
    5 stats.tick('hoge', 1, 1, [user_id:1234]);
    Node.jsͷwrapper library

    View Slide

  33. ɹɹʛɹɹ© 2019 PLAID Inc. 33
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    tagͰϢʔβࣝผ༻IDΛૹΔ
    ࠓى͍ͬͯ͜Δ໰୊͕
    ʮಛఆͷϢʔβىҼͰى͍ͬͯ͜Δʁʯ
    ʮKARTEશମͰى͍ͬͯ͜Δʁʯ
    ͔Λ੾Γ෼͚͠΍͍ͯ͘͢͠Δ
    Custom metricsͷੵۃ׆༻

    View Slide

  34. 34
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEΛࢧ͑Δ
    ؂ࢹͷϙΠϯτ
    ࿩͢͜ͱ

    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard

    View Slide

  35. 35
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒

    View Slide

  36. ɹɹʛɹɹ© 2019 PLAID Inc. 36
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    Layer 1
    ɾKARTE͕ੜ͖͍ͯΔ͔Ͳ͏͔
    LayerΛ෼͚ͨDashboard

    View Slide

  37. ɹɹʛɹɹ© 2019 PLAID Inc. 37
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    Layer 2
    ɾॏཁͳίϯϙʔωϯτ
    ɾॏཁͳΞϓϦέʔγϣϯͷϝτϦοΫ
    LayerΛ෼͚ͨDashboard

    View Slide

  38. ɹɹʛɹɹ© 2019 PLAID Inc. 38
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    Layer 3
    ɾ֤αʔό܈ͷࡉ͔͍ϝτϦοΫ
    LayerΛ෼͚ͨDashboard

    View Slide

  39. 39
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ࿩͢͜ͱ
    KARTEͷಛ௃
    KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ
    ࠓޙ΍Γ͍ͨ͜ͱ



    View Slide

  40. ɹࠓޙ΍Γ͍ͨ͜ͱ
    40
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    શͯͷΤϯδχΞ͕
    ো֐ͷҰ࣌ରԠΛͰ͖ΔΑ͏ʹ͢Δ
    ɾ؂ࢹ/ো֐ରԠͷϑϩʔͷ੔ཧ
    ɾDatadogΛ࢖͍͜ͳͯ͠΋Β͏

    View Slide

  41. ɹࠓޙ΍Γ͍ͨ͜ͱ
    41
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    Custom metrics 1ԯԁ໰୊ͷղܾ
    ɾ͋·ΓʹศརͩͬͨͷͰ࢖͍͢
    ͗ͯ͠·ͬͨ…
    ɾCustom metricsͷ੔ཧΛߦ͏

    View Slide

  42. 42
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ·ͱΊ
    ·ͱΊ

    View Slide

  43. 43
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEΛࢧ͑Δ
    ؂ࢹͷϙΠϯτ
    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard

    View Slide

  44. View Slide