マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog

Datadog Summit Tokyoでの登壇資料です。
動画はhttps://www.youtube.com/watch?v=_Z4HBiwODbc

Transcript

1. 1
   ϚϧνΫϥ΢υͳେن໛αʔϏε
   KARTEΛࢧ͑ΔDatadog
   ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
   PLAID, Inc.
   Daiki Matsui
   

   View Slide

2. 2
   2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
   ɹɹʛɹɹ© 2019 PLAID Inc.
   ࣗݾ঺հ
   দҪ େٍ גࣜձࣾϓϨΠυ
   SRE Senior Architect
   2012೥-2015೥·ͰגࣜձࣾαΠόʔυʹͯήʔϜ։ൃʹैࣄ
   2015೥ΑΓϓϨΠυʹࢀը͠ɺSREͱͯ͠ϚϧνΫϥ΢υΠϯϑϥͷߏஙɾӡ༻౳Λ୲౰
   2015೥ೖࣾ
   

   View Slide

3. ɹɹʛɹɹ© 2019 PLAID Inc. 3
   2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
   גࣜձࣾϓϨΠυʹ͍ͭͯ
   גࣜձࣾϓϨΠυ
   ౦ژ౎தԝ۠ۜ࠲6-10-1 GINZA SIX 10F
   ઃཱɿ2011೥10݄
   ैۀһɿ130໊ ࢿຊۚɿ149,987,878ԁ ※ࢿຊ४උؚۚΉ
   

   View Slide

4. 4
   2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
   ɹɹʛɹɹ© 2019 PLAID Inc.
   KARTEʹ͍ͭͯ
   ֬ೝ͢Δ
   جຊαʔϏε΍͝ར༻ํ๏Λ͝঺հ͠·͢ɻ
   ͓ಘͳಛయ΍໾ཱͭ৘ใ͕ຬࡌͰ͢ɻ
   ॳΊͯͷํ΁
   ְתּׅ然钠ׅ׷
   6*رؠ؎ش٦׌ֽוזַזַ♳麦׃זְծ➙
   ״׶إٔؗ،حف׃׋ְהְֲ倯䗳铣דׅկ
   UIرؠ؎ش٦ָ濼׏גֶֻץֹ
   7אךرؠ؎ٕٝ٦ٕ
   DESIGN
   RULES
   » CHECK
   ׆׏ה⢪ִ׷،؎ذي׌ֽ׾䲧ִת׃׋կ
   ֿך堣⠓׾ֶ鋅鷕׃זֻկ
   窫㼎ծ
   妜׃ְ
   ౙͷओ໾ɺ
   Ξ΢λʔ
   COLLECTION
   OUTER
   嗚稊勴⟝׾㼰׃㢌刿ׅ׷׌ֽדծ֮ז׋ך椚
   䟝ך暟⟝ח⳿⠓ִ׷〳腉䚍ָ넝ֻז׶תׅկ
   勴⟝׾㢌ִג嗚稊׃ג׫גֻ׌ְׁկ
   ׀䋞劄ך暟⟝כ
   鋅אַ׶תׇ׿ד׃׋ַ
   ꟗׄ׷
   ҰਓҰਓʹ߹Θͤͨ
   ސ٬ମݧΛఏڙ
   WebαΠτͷ๚໰ऀͷߦಈΛ

   ސ٬͝ͱʹϦΞϧλΠϜʹղੳ
   $9 ސ٬ମݧ
   ϓϥοτϑΥʔϜ
   

   View Slide

5. 5
   2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
   ɹɹʛɹɹ© 2019 PLAID Inc.
   ࿩͢͜ͱ
   KARTEͷಛ௃
   KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ
   ࠓޙ΍Γ͍ͨ͜ͱ
   1
   2
   3
   

   View Slide

6. 6
   2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
   ɹɹʛɹɹ© 2019 PLAID Inc.
   ࿩͢͜ͱ
   KARTEͷಛ௃
   KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ
   ࠓޙ΍Γ͍ͨ͜ͱ
   
   

   View Slide

7. 7
   ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
   KARTEͷಛ௃
   1.Ϛϧνςφϯτ
   2.௿ϨΠςϯγʔ/ߴෛՙ
   3.େن໛γεςϜ
   4.ϚϧνΫϥ΢υߏ੒
   ࿩͢͜ͱ
   →
   

   View Slide

8. 8
   ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
   KARTEͷಛ௃
   1.Ϛϧνςφϯτ
   2.௿ϨΠςϯγʔ/ߴෛՙ
   3.େن໛γεςϜ
   4.ϚϧνΫϥ΢υߏ੒
   ࿩͢͜ͱ
   →
   

   View Slide

9. 9
   2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
   ɹɹʛɹɹ© 2019 PLAID Inc.
   Ϛϧνςφϯτ
   

   View Slide

10. ɹɹʛɹɹ© 2019 PLAID Inc. 10
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɾ༷ʑͳෛՙͷ͓٬͞Μ͕͍Δ
    ɾΞΫηεύλʔϯ΋৭ʑ
    Ϛϧνςφϯτ
    

    View Slide

11. ɹɹʛɹɹ© 2019 PLAID Inc. 11
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɾෛՙ͕1೔Ͱ໿10ഒมԽ͢Δ
    ɾηʔϧ౳Ͱෛՙ͕ٸ૿͢Δ
    Ϛϧνςφϯτ
    1೔ͷෛՙਪҠ
    

    View Slide

12. 12
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEͷಛ௃
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    ࿩͢͜ͱ
    →
    

    View Slide

13. 13
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ௿ϨΠςϯγʔɺߴෛՙ
    sec to real-time
    actions
    0.x
    events per day
    3 billion
    stored in BigQuery
    2+ PB
    events (rows) / sec
    65,000
    streaming insert per
    month
    1 PB
    analysis per month
    60+ PB
    SRE
    3
    ௿ϨΠςϯγʔ/ߴෛՙ
    

    View Slide

14. 14
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ௿ϨΠςϯγʔɺߴෛՙ
    sec to real-time
    actions
    0.x
    events per day
    3 billion
    stored in BigQuery
    2+ PB
    events (rows) / sec
    65,000
    streaming insert per
    month
    1 PB
    analysis per month
    60+ PB
    SRE
    3
    ௿ϨΠςϯγʔ/ߴෛՙ
    

    View Slide

15. 15
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ௿ϨΠςϯγʔɺߴෛՙ
    

    View Slide

16. 16
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ௿ϨΠςϯγʔɺߴෛՙ
    sec to real-time
    actions
    0.x
    events per day
    3 billion
    stored in BigQuery
    2+ PB
    events (rows) / sec
    65,000
    streaming insert per
    month
    1 PB
    analysis per month
    60+ PB
    SRE
    3
    ௿ϨΠςϯγʔ/ߴෛՙ
    

    View Slide

17. 17
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEͷಛ௃
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    ࿩͢͜ͱ
    →
    

    View Slide

18. 18
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    େن໛γεςϜ
    

    View Slide

19. 19
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEͷಛ௃
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    ࿩͢͜ͱ
    →
    

    View Slide

20. 20
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ϚϧνΫϥ΢υߏ੒
    

    View Slide

21. ɹɹʛɹɹ© 2019 PLAID Inc. 21
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    Pros
    1. ֤ϓϩόΠμʔͷྑ͍ͱ͜औΓ͕Ͱ͖Δ
    2. ϓϩόΠμʔͷো֐ΛผϓϩόΠμʔͰΧ
    όʔͰ͖Δ
    Pros/Cons
    ϚϧνΫϥ΢υߏ੒
    Cons
    1. ӡ༻ͷख͕ؒ૿͑Δ
    

    View Slide

22. ɹɹʛɹɹ© 2019 PLAID Inc. 22
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ৄࡉΛ஌Γ͍ͨํ͸
    ʮσϒαϛ 2018ʯͷࢿྉͰʂ
    ϚϧνΫϥ΢υߏ੒
    https://speakerdeck.com/tikson/zhan-lue-topointonituite
    

    View Slide

23. 23
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ࿩͢͜ͱ
    KARTEͷಛ௃
    KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ
    ࠓޙ΍Γ͍ͨ͜ͱ
    
    ✓
    

    View Slide

24. KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ
    24
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɾϦΞϧλΠϜੑ
    ɾॊೈੑ
    ɾεέʔϥϏϦςΟ
    ɾ҆ఆੑ
    

    View Slide

25. 25
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEͷಛ௃ →
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    

    View Slide

26. 26
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEΛࢧ͑Δ
    ؂ࢹͷϙΠϯτ
    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    ࿩͢͜ͱ
    →
    

    View Slide

27. 27
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    →
    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    

    View Slide

28. ɹɹʛɹɹ© 2019 PLAID Inc. 28
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ো֐ରԠ࣌ʹ͸DatadogΛݟΕ͹
    ͙͢ʹঢ়گΛ೺ѲͰ͖ΔΑ͏ʹ
    Datadogʹશͯू໿
    

    View Slide

29. 29
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEΛࢧ͑Δ
    ؂ࢹͷϙΠϯτ
    ࿩͢͜ͱ
    →
    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    

    View Slide

30. 30
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    →
    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    

    View Slide

31. ɹɹʛɹɹ© 2019 PLAID Inc. 31
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    32M/mon ΄Ͳར༻
    Custom metricsͷੵۃ׆༻
    

    View Slide

32. Custom metricsͷੵۃ׆༻
    32
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    https://github.com/makinoy/libs-dogstatsd
    ɾlibs-dogstatsdͰɺ؆୯ʹॲཧͷ࣮ߦճ਺΍
    ࣮ߦ࣌ؒΛଌఆͰ͖ΔΑ͏ʹ
    ɾؔ਺ΑΓ΋ࡉ͔͍୯ҐͰlatencyΛଌఆ
    ɾϢʔβ୯ҐͰͷύϑΥʔϚϯεΛଌఆ

    1 const dd = require('libs-dogstatsd');
    2 const stats = dd.start();
    3 // measure hoge
    4 hoge();
    5 stats.tick('hoge', 1, 1, [user_id:1234]);
    Node.jsͷwrapper library
    

    View Slide

33. ɹɹʛɹɹ© 2019 PLAID Inc. 33
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    tagͰϢʔβࣝผ༻IDΛૹΔ
    ࠓى͍ͬͯ͜Δ໰୊͕
    ʮಛఆͷϢʔβىҼͰى͍ͬͯ͜Δʁʯ
    ʮKARTEશମͰى͍ͬͯ͜Δʁʯ
    ͔Λ੾Γ෼͚͠΍͍ͯ͘͢͠Δ
    Custom metricsͷੵۃ׆༻
    

    View Slide

34. 34
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEΛࢧ͑Δ
    ؂ࢹͷϙΠϯτ
    ࿩͢͜ͱ
    →
    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    

    View Slide

35. 35
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    →
    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    1.Ϛϧνςφϯτ
    2.௿ϨΠςϯγʔ/ߴෛՙ
    3.େن໛γεςϜ
    4.ϚϧνΫϥ΢υߏ੒
    

    View Slide

36. ɹɹʛɹɹ© 2019 PLAID Inc. 36
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    Layer 1
    ɾKARTE͕ੜ͖͍ͯΔ͔Ͳ͏͔
    LayerΛ෼͚ͨDashboard
    

    View Slide

37. ɹɹʛɹɹ© 2019 PLAID Inc. 37
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    Layer 2
    ɾॏཁͳίϯϙʔωϯτ
    ɾॏཁͳΞϓϦέʔγϣϯͷϝτϦοΫ
    LayerΛ෼͚ͨDashboard
    

    View Slide

38. ɹɹʛɹɹ© 2019 PLAID Inc. 38
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    Layer 3
    ɾ֤αʔό܈ͷࡉ͔͍ϝτϦοΫ
    LayerΛ෼͚ͨDashboard
    

    View Slide

39. 39
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ࿩͢͜ͱ
    KARTEͷಛ௃
    KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ
    ࠓޙ΍Γ͍ͨ͜ͱ
    
    ✓
    ✓
    

    View Slide

40. ɹࠓޙ΍Γ͍ͨ͜ͱ
    40
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    શͯͷΤϯδχΞ͕
    ো֐ͷҰ࣌ରԠΛͰ͖ΔΑ͏ʹ͢Δ
    ɾ؂ࢹ/ো֐ରԠͷϑϩʔͷ੔ཧ
    ɾDatadogΛ࢖͍͜ͳͯ͠΋Β͏
    

    View Slide

41. ɹࠓޙ΍Γ͍ͨ͜ͱ
    41
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    Custom metrics 1ԯԁ໰୊ͷղܾ
    ɾ͋·ΓʹศརͩͬͨͷͰ࢖͍͢
    ͗ͯ͠·ͬͨ…
    ɾCustom metricsͷ੔ཧΛߦ͏
    

    View Slide

42. 42
    2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    ɹɹʛɹɹ© 2019 PLAID Inc.
    ·ͱΊ
    ·ͱΊ
    

    View Slide

43. 43
    ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
    KARTEΛࢧ͑Δ
    ؂ࢹͷϙΠϯτ
    1.Datadogʹશͯू໿
    2.Custom metricsͷੵۃ׆༻
    3.LayerΛ෼͚ͨDashboard
    →
    

    View Slide

44. View Slide