Suggested Topicの質問に可能な限り答えてみた

Transcript

1. Japan Datadog User Group Meetup#1
   ABEJA, Inc.
   Shogo Muranushi
   Suggested Topicͷ࣭໰ʹՄೳͳݶΓ౴͑ͯΈͨ
   

   View Slide

2. @rwle1212
   2
   

   View Slide

3. @rwle1212
   3
   ໊લ ɿ ଜओ ૖ޛʢΉΒ͵͠ ͠ΐ͏͝ʣ
   ձࣾ ɿ ABEJA, Inc.
   ۀ຿ ɿΠϯϑϥΤϯδχΞ -> ϓϩμΫτΦʔφʔ -> SRE
   

   View Slide

4. @rwle1212
   4
   Datadogྐྵ
   2೥΄ͲɻInfra ͱ Logs ͷར༻͕ϝΠϯɻ
   Datadog͸ࡉ͔͍ػೳ͕ଟ͘ɺͿͬͪΌ͚Α͘෼͔ͬͯͳ͍ɻ
   ࠓ೔࿩͢͜ͱ΋͋͘·Ͱฐࣾͷ࢖͍ํͳͷͰਖ਼ղͰ͸ͳ͍ɻ
   ৭ʑ஌Γ͍ͨͷͰΈΜͳͰ৘ใ࣋ͪدΓͨ͘ɺ

   ·ͣ͸ొஃͯ͠Έͨɻ
   

   View Slide

5. @rwle1212
   5
   

   View Slide

6. @rwle1212
   6
   

   View Slide

7. @rwle1212
   7
   Datadogར༻ن໛ײ
   docker.containers.running Datadog Logs by weekly
   

   View Slide

8. @rwle1212
   8
   ຊ୊ʹೖΓ·͢
   

   View Slide

9. @rwle1212
   9
   ࢖͍ͬͯΔػೳ΍ͦͷλΠϛϯά

   (ʮৼΓฦΓͰ͜ΜͳμογϡϘʔυΛ࢖͍ͬͯΔʯͳͲ)
   • Α͘͞ΘΔػೳ͸LogsͱNotebooks
   • ཧ༝΍λΠϛϯά͸ޙड़
   

   View Slide

10. @rwle1212
    10
    μογϡϘʔυͲ͏΍ͬͯઃܭͯ͠·͔͢ʁ
    ΅͘ͷ͔Μ͕͍͖͑ͨ͞ΐ͏ͷμογϡϘʔυ
    ͲΜͳDashboardͰӡ༻͍ͯ͠Δ͔ɻ
    εϚʔτͳμογϡϘʔυͷ࡞Γํ
    μογϡϘʔυ͸׶͑ͯ࢖ͬͯ·ͤΜɻ
    ཧ༝͸ʮ΅͘ͷ͔Μ͕͍͖͑ͨ͞ΐ͏ͷμογϡϘʔυʯʹͳͬͯ͠·͏͔ΒͰ͢ɻ
    

    View Slide

11. @rwle1212
    11
    ࣦഊྫ1: ݁ہCloudWatchΛݟ͍ͯͨ
    ALB -> Container -> EC2 -> RDSͱ͍͏ϦΫΤετͷྲྀΕͰμογϡϘʔυΛ࡞ͬ
    ͯΈͨɻ
    μογϡϘʔυͷେ͖͞͸ܾ·͍ͬͯΔɺ׌ͭάϥϑ͕ଟ͗ͯ͢΋ෳࡶʹͳΔͷ
    ͰɺεϚʔτʹগͳΊʹάϥϑΛ഑ஔͨ͠ɻ
    ͔͠͠ো֐࣌ʹ৘ใ͕଍Γͳ͍ͷͰɺ݁ہCloudWatchΛோΊ͍ͯͨ͜ͱʹؾ෇͍
    ͨɻ
    ͔ͦ͜ΒඞཁͳάϥϑΛ଍͍ͯ͘͠ͱͲΜͲΜෳࡶʹͳΓɺଞͷϝϯόʔ͸ʮԿΛ
    Ͳ͏ݟͨΒྑ͍ͷ͔Θ͔Βͳ͍ʯঢ়ଶʹͳͬͨɻ
    ݁ہCloudWatchΛோΊ͍ͯͨ͜ͱʹؾ෇͍ͨɻ
    

    View Slide

12. @rwle1212
    12
    ࣦഊྫ2: 

    μογϡϘʔυͷछྨ͕૿͑ΔͱΑ͘Θ͔Βͳ͘ͳΔ
    

    View Slide

13. @rwle1212
    13
    ղܾྫ

    NotebookΛ࢖ͬͯΈͨ
    • ো֐࣌ɺௐࠪ࣌ʹඞཁͳಛఆͷάϥϑ
    ͕഑ஔ͞Ε͍ͯΔෛՙ֬ೝ༻ͷ
    NotebookΛ࡞੒
    • ܦա؍࡯ΛݟΔͨΊͷάϥϑ͕഑ஔ͞
    Ε͍ͯΔNotebookΛ࡞੒
    ͳͲ
    

    View Slide

14. @rwle1212
    14
    ղܾྫ

    NotebookΛ࢖ͬͯΈͨ
    • Jupyter Notebook Likeͳ࢖͍ํ͕Ͱ͖
    ͯɺίϝϯτͱάϥϑΛॻ͚Δ
    • ಛఆͷো֐ʹରͯ͠ௐ΂ͨ͜ͱɺਪଌ
    ͳͲͷίϝϯτΛॻ͖ͳ͕Βه࿥͕࢒
    ͤΔ
    • ϩάͷάϥϑԽ΋༰қ
    • Issueʹը૾ΛషΓ෇͚ΔΑΓεϜʔ
    ζʹ·ͱΊΒΕΔͷͰڞ༗ָ͕͘͢͝
    

    View Slide

15. @rwle1212
    15
    ղܾྫ

    NotebookΛ࢖ͬͯΈͨ
    • ͭ·Γ൚༻తͳμογϡϘʔυ͸࡞੒ͤͣʹɺCloudWatchͳͲͷશମΛ၆ᛌͰ
    ͖ΔϝτϦΫεΛݟͭͭɺϩάΛάϥϑԽͨ͠Γͯ͠ɺಛఆͷো֐ʹ߹Θͤͯ
    ඞཁͳάϥϑΛ഑ஔͨ͠NotebookͰௐࠪ΍ܦա؍࡯Λߦ͍ͬͯΔ
    

    View Slide

16. @rwle1212
    16
    ͦ΋ͦ΋Έͳ͞ΜͲͷػೳԡ͠?
    ͱ͍͏͜ͱͰNotebooksΛਪͯ͠·͢ɻ
    ͋ͱLogs΋ྑ͍ࢠͰ͢ɻޙड़Ͱ·ͱ·͍ͬͯ·͢ɻ
    

    View Slide

17. @rwle1212
    17
    ௿ϦΫΤετ࣌ͷrateΞϥʔτ཈੍ํ๏ʹ͍ͭͯ
    ௿ϦΫΤετͷ؂ࢹ͸ͯ͠·ͤΜ
    

    View Slide

18. @rwle1212
    18
    terraformҎ֎Ͱͷ؅ཧํ๏
    ओʹຊ൪༻͸TerraformɻͱΓ͋͑ͣͷ࣌͸खͰ࡞Δɻ
    खͰ࡞ͬͨͷͱTerraformͰ࡞ͬͨ΋ͷ͕ࠞࡏ͢Δͱɺຊ൪༻ͳͷ͔ɺΞϥʔτΛ
    ์ஔͯ͠ྑ͍ͷ͔Α͘෼͔Βͳ͘ͳΔͷͰʮCreateByTerraformʯTagΛ෇༩ͯ͠൑
    ผ͍ͯ͠Δ
    

    View Slide

19. @rwle1212
    19
    ՝ۚ؅ཧʹؔ͢Δେٕখٕ
    ಛʹେٕ͸ແ͠ɻখωλ͕͋Δͱͨ͠Β
    • Multi Org؀ڥ͸Billing͕ू໿͞Εͯ͠·͏ɻʮLast Month’s UsageʯͰར༻ྔʢྉ
    ۚͰ͸ͳ͍ʣ͕Orgຖʹग़ྗ͞ΕΔͷͰ҈෼͍ͯ͠ΔɻӦۀ͞Μʹݴ͑͹Billing΋
    ෼͚ͯ͘ΕΔ͔΋ʁखؒͳͷͰͦ͜·Ͱ΍ͬͯͳ͍
    • AWSΠϯςάϨʔγϣϯͰͨ͘͞ΜͷαʔϏεͷϝτϦΫεΛऔΕΔΑ͏ʹ͢Δ
    ͱɺCloudWatch ͷ GetMetricData API ͕ΊͪΌͪ͘Όݺͼग़͞ΕΔͷͰ՝͕ۚ
    େ͖͘૿͑Δɻ·ͣ͸ඞཁͳαʔϏε͚ͩߜͬͯऔΓࠐΉͷ͕٢ɻ

    SQSͰࢮ๢ͨ͠ܦݧ༗Γ
    

    View Slide

20. @rwle1212
    20
    ՝ۚ؅ཧʹؔ͢Δେٕখٕ
    • Logs͸ʮͲΕ͕Կ݅ʯͱ͔ݟΕΔͷͰɺͨ·ʹෆཁͳϩά͕ແ͍͔֬ೝͯ͠
    ʮIndexsʯͰআ֎ϑΟϧλΛ࡞੒ͯ͠ϩάͷ՝ۚΛ཈͑ͨΓ͢Δ

    ʢIndexed Logsͷ՝ۚ͸ࢭ·Δ͚ͲɺIngested Logsͷ՝ۚ͸ࢭ·Βͳ͍ͷͰ஫ҙɻ͍͔҆Βؾʹ͢Δ
    ΄ͲͰ͸ͳ͍͚Ͳʣ
    

    View Slide

21. @rwle1212
    21
    DBϞχλϦϯάͷTips͋Ε͜Ε
    CloudWatchΛIntegrationͯ͠ΔҐͰͦΕҎ্ͷ͜ͱ͸͓ͯ͠Γ·ͤΜ
    

    View Slide

22. @rwle1212
    22
    GoogleAnalyticsAPIར༻ͨ͠ݱࡏϢʔβʔ਺ͱ͔࠾औͯ͠
    ͍Δࣄྫ͕͋Ε͹
    GoogleAnalyticsAPI͸࢖༻͓ͯ͠Γ·ͤΜ
    

    View Slide

23. @rwle1212
    ࡢ೥ͷ12݄ʹҎԼͷهࣄΛॻ͖·ͨ͠
    23
    ଞͷ؂ࢹSaaS౳ʢࣗ෼͕஌ΔൣғͰ͸Zabbixɾ
    PrometheusɾNew RelicɾMackerelʣͱͷൺֱ
    

    View Slide

24. @rwle1212
    24
    ʲཁ໿ʳͳͥPrometheusΛࣙΊͯDatadogΛ࠾༻ͨ͠ͷ͔
    • Datadog͔ΒPrometheus΁৐Γ׵͑
    • ݩʑDatadogΛ࢖͍ͬͯͨɻ$18/host͕ߴ͔ͬͨ͠ɺKubernetesͷ؂ࢹͷσϑΝΫ
    τʹͳΓͭͭ͋ΔPrometheusΛࢼ͢໨త΋ؚΊͯ࢖͍࢝Ίͨɻ
    • ൒೥ܦաɻPrometheusࣗମͷো֐͕νϥϗϥ
    • OSSͰ৘ใগͳ͍৔߹͸ௐࠪʹ͕͔͔࣌ؒΔɻଟ͍࣌͸िʹ1,2ճ΄Ͳ໰୊͕ൃੜ
    ͠1,2ਓ͕਺࣌ؒ/ि΄ͲऔΒΕ͍ͯͨ
    • ελʔτΞοϓ͸ਓ͕͍ͳ͍ɻຊۀͷ։ൃʹूத͍ͨ͠
    • πʔϧͷ͓कΓʹ͕࣌ؒऔΒΕ͍ͯͨɻPrometheusͷӡ༻Λ֎෦ʹҕୗ͍ͨ͠ɻ
    ӡ༻Λ֎෦ʹҕୗ͢Δͱ͍͘ΒʹͳΔ͔෼͔Βͳ͍͚Ͳɺࠓͷن໛ΛDatadogʹม
    ͑Δͱ10ສ/݄͘Β͍ͩͬͨɻɾɾDatadogͰΑ͘Ͷʁͱࢥ͍࢝ΊΔ
    

    View Slide

25. @rwle1212
    25
    ʲཁ໿ʳͳͥPrometheusΛࣙΊͯDatadogΛ࠾༻ͨ͠ͷ͔
    • ϩάͷऩूͲ͏͢Δ໰୊
    • CloudWatch LogsΛ࢖͍͕ͬͯͨݕࡧੜօແɻʢ౰࣌͸ʣPrometheus͸ϩάऩू
    ͢Δػೳ͸ͳ͍ɻσϑΝΫτతʹ͸ ElasticSearch + Kibana ͩΖ͏
    • ͔͠͠ɺͦͷ࣌఺Ͱϩάྔ͕5ԯ/݄ɻঃʑʹର৅ͷΞϓϦ΍νʔϜΛ૿΍ͦ͏ͱࢥ
    ͏ͱ਺ϲ݄Ͱഒʑʹ૿͍͑ͯ͘͜ͱ͕૝ఆ͞Ε͍ͯͨ
    • ͦΕΛElasticSearchͰड͚ͳ͕Βӡ༻͢Δ͜ͱΛ૝ఆ͢Δͱɺख͕औΒΕͦ͏ײ͕
    ൒୺ͳ͔ͬͨ
    • Datadog΋ϩάͷαʔϏεΛఏڙ࢝͠Ί͍ͯͨͷͰ࢖ͬͯΈΔ͜ͱʹ
    

    View Slide

26. @rwle1212
    26
    ʲཁ໿ʳͳͥPrometheusΛࣙΊͯDatadogΛ࠾༻ͨ͠ͷ͔
    • ϩάͲ͏͢Δ͔ͷ݁࿦
    • Facets ͕ڧྗͰɺಋೖָ͕ɺίετίϯτϩʔϧ΋͠΍͘͢ɺԿ͔͋Ε͹શϩά
    Λ௥͑Δɻͦͷ্Ͱɺϩάऩू͢Δ্ͰΠϯϑϥΛؾʹ͠ͳͯ͘ྑ͍ͷ͕ؾʹೖ
    ΓɺঃʑʹϩάΛ Datadog Logs ʹدͤ·ͨ͠
    • ͦͯ͠ɺϞχλϦϯά΋ Promethes ͔Β Datadog ΁Ҡߦ
    • ͳͥPrometheusΛࣙΊͯDatadogΛ࠾༻ͨ͠ͷ͔ɺͷ·ͱΊ
    • ϩά΋ϞχλϦϯά΋ ͦΕ͕ࣄۀͷίΞͰ͋Ε͹ͪΌΜͱߏங͠ӡ༻͢Δ͜ͱࣗ
    ମ͸ԀΘͳ͍ͷͰ͕͢ɺίΞͰͳ͍ͳΒ࣌ؒΛׂ͖ͨ͘ͳ͍
    

    View Slide

27. @rwle1212
    ZabbixɺPrometheus
    ๻ͷ؍఺͔Β͢ΔͱɺػೳੑͰ͸ͳ͘ӡ༻ੑͰ൑அ͍ͯ͠ΔͷͰOSSܥ͸Α΄
    Ͳಛ௃తͳػೳ΍ཁ͕݅ͳ͍ݶΓੵۃతͳ࠾༻͸ආ͚͍ͨͰ͢
    27
    ଞͷ؂ࢹSaaS౳ʢࣗ෼͕஌ΔൣғͰ͸Zabbixɾ
    PrometheusɾNew RelicɾMackerelʣͱͷൺֱ
    

    View Slide

28. @rwle1212
    New RelicɺMackerel
    ྆ํͱ΋௚઀ݕূ͸͍ͯ͠ͳ͍ͷͰൺֱʹ͸ͳΒͳ͍Ͱ͕͢ɺݕূ͠ͳ͔ͬͨ
    ཧ༝ͷҰͭͱͯ͠ɺDatadogͰ͸ʮMonitoring Modern Infrastructureʯͱ͍͏ه
    ࣄ͕ެ։͞Ε͍ͯ·͢ɻ

    ͜͜ʹॻ͔Ε͍ͯΔ಺༰͕ͱͯ΋ཧʹ͔ͳ͍ͬͯͨͷͰɺ͜ͷΑ͏ͳࢥ૝ʹ
    ΑͬͯαʔϏε͕ઃܭ͞Εࠓޙ΋αʔϏεΛ௥Ճ͍ͯ͘͠ΜͩΖ͏ͳͱࢥ͍ɺ
    ࠾༻ͨ͠എܠ΋͋Γ·͢ɻ
    28
    ଞͷ؂ࢹSaaS౳ʢࣗ෼͕஌ΔൣғͰ͸Zabbixɾ
    PrometheusɾNew RelicɾMackerelʣͱͷൺֱ
    

    View Slide

29. @rwle1212
    29
    SLOؔ࿈ͷϞχλʔ࡞੒ݕ౼Ͱࠔͬͨ͜ͱɾ޻෉ͨ͜͠ͱ
    ͋·Γ೉͍͠ػೳͰ͸ͳ͍ͷͰಛʹͳ͠ʁ
    ڧ͍ͯ͋͛ΔͳΒʮԿΛSLOͱͯ͠ܭଌ͢Δ͔ʯͷɺͦ΋ͦ΋ͷSLOͷఆ͕ٛ೉͍͠
    ͘Β͍Ͱ͢ɻ
    

    View Slide

30. @rwle1212
    30
    Datadogͷ෍ڭ׆ಈ
    ʢͲͷ໾ׂͷΤϯδχΞ·Ͱ࢖͍ͬͯΔ͔ʁʣ
    • ޮՌ͋ͬͨ͜ͱ
    • Dev/Stg/ProdͷશΞϓϦͷϩάΛશ෦ू໿ͨ͠ɻ1ͭͷWebαΠτΛݟʹ
    ͍͘͜ͱͰશ෦ͷϩά͕ݟΕΔঢ়ଶʹͨ͠ɻࠓ·Ͱ͸؀ڥຖʹݟΔ৔ॴ͕
    ෼͔ΕͯͨΓɺग़ͯͨΓग़ͯͳ͔ͬͨΓ
    • ͔ͦ͜ΒঃʑʹDatadogͷ࢖͍ํΛೝࣝ͠ɺMonitors΍NotebooksɺAPM
    ʹϝϯόʔͷڵຯ͕޿͕ͬͯߦͬͨ
    

    View Slide

31. @rwle1212
    31
    Datadogͷ෍ڭ׆ಈ
    ʢͲͷ໾ׂͷΤϯδχΞ·Ͱ࢖͍ͬͯΔ͔ʁʣ
    • ޮՌ͋·Γͳ͔ͬͨ͜ͱ
    • Datadogͷ࢖͍ํษڧձɻʮ΁ʔɻ͜Μͳ͜ͱͰ͖ΔΜͩʯͱ͍͏͚ͩͰखΛ
    ಈ͔͞ͳ͍ͷͰฉ͍ͯ΋๨ΕΔ
    • ϩάɺϝτϦΫεΛՄೳͳݶΓू໿ͯ͠ɺඞཁʹۦΒΕͯࣗ෼Ͱௐ΂ͯ޻
    ෉ͯ͠΋Β͏ํ͕͕֮͑ྑͦ͞͏ͳؾ͕ͨ͠ɻݴ͏ͯ΋೉͘͠ͳ͍ͷͰ
    • ͋Δఔ౓ͨ͠Βʮ͓Εͷ͔Μ͕͍͖͑ͨ͞ΐ͏ͷDatadogڞ༗ձʯΛ֤ϝ
    ϯόʔͰখωλతʹ΍Δͷ΋ѱ͘͸ͳ͔ͬͨ
    

    View Slide

32. @rwle1212
    32
    Datadogͷ෍ڭ׆ಈ
    ʢͲͷ໾ׂͷΤϯδχΞ·Ͱ࢖͍ͬͯΔ͔ʁʣ
    • Ͳͷ໾ׂͷΤϯδχΞ·Ͱ࢖͍ͬͯΔ͔
    • ࢖͍ͬͯΔͷ͸ΠϯϑϥɺόοΫΤϯυ
    • ϑϩϯτΤϯυͷE2E؂ࢹܥ͸গ͠ߴ͍ͷͰ࢖͍ͬͯͳ͍
    • Real User MonitoringɺSynthetic Monitoring (Browser)
    

    View Slide

33. @rwle1212
    33
    ΠϯςάϨʔγϣϯͰ޻෉͍ͯ͠Δ͜ͱʢSlack / JIRAͱ૊
    Έ߹ΘͤͯɺΠϯγσϯτ؅ཧɾ໰୊؅ཧʹར༻ͳͲʣ
    Warning͸ʮSlack௨஌ʯ
    Alert͸ʮSlack௨஌ʯ+ʮPagerDutyͰి࿩ʯ+ʮGitHub Issue࡞੒(Zapierܦ༝)ʯ
    

    View Slide

34. @rwle1212
    34
    Ξϥʔτͷ௨஌Ϩϕϧ؅ཧ
    ʢฐࣾ͸warn / error / fatal ͳͲʣ
    • WarnͱErrorͷΈ
    • ʮαʔϏεʹ௚઀Өڹ͕Ͱ͍ͯΔϞϊɺͰ͍ͯΔՄೳੑ͕͋ΔϞϊʯΛΞϥʔτ
    ͱͯ͠ઃఆ͍ͯ͠Δ
    • ྫʣCPU΍ϝϞϦͷ࢖༻཰Λ؂ࢹͱ͔͸͍ͯ͠ͳ͍ɻͦΕΒͷӨڹʹΑͬ
    ͯALBͰ5xx͕සൃ͢Ε͹Ξϥʔτ͕ඈͿΑ͏ʹ͍ͯ͠ΔɻDynamoDBͷ
    εϩοτϦϯάͱ͔
    

    View Slide

35. @rwle1212
    35
    datadog logsʹ͍ͭͯ࢖༻ײ΍ࣄྫ
    

    View Slide

36. @rwle1212
    36
    facets
    • JSONͰૹΕ͹উखʹύʔεͯ͘͠ΕΔ
    • ύʔε͞Εͨ஋Λfacets(index)Ͱ͖ΔͷͰɺϩ
    άͷQuery΍Group ByͳͲʹར༻Ͱ͖Δ
    

    View Slide

37. @rwle1212
    37
    facets
    • Կ͕ѱ͍ͷ͔Θ͔Βͳ͍͚ͲɺindexͰ͖ͳ͍
    ͕࣌͋Δɻ୭͔ղܾํ๏Λڭ͑ͯཉ͍͠
    • ͜ͷέʔεͷ৔߹͸਺஋Ҏ֎ͷจࣈྻ͕ೖͬ
    ͪΌͬͯΔͷ͔ͳɾɾʁͬ͟ͱோΊͨײͦ͡Μ
    ͳ͜ͱͳ͍͚ͲɺͦΕΛ؆୯ʹݟΔʹ͸Ͳ͏͠
    ͨΒྑ͍ΜͩΖ͏ɾɾ
    

    View Slide

38. @rwle1212
    38
    Graph by Logs
    • ϩάΛάϥϑԽͰ͖Δ
    • facetsʹొ࿥ͨ͠஋ΛGroup Byʹࢦఆͨ͠Γɺ
    CountʹೖΕΔ͜ͱ΋Ͱ͖Δ
    • ࠨਤͷྫͰ͸ɺOrganization NameຖͷϦΫΤ
    ετ਺΍ฏۉԠ౴࣌ؒͳͲΛදࣔɻίϯςφID
    ຖʹݟͯภΓ͕ແ͍͔ݟͨΓ΋ͯ͠Δ
    • γεςϜܥ͚ͩͰ͸ͳ͘ΞϓϦͷར༻ঢ়گ෼ੳ
    ʹ΋࢖༻ՄೳͳͷͰɺΞϓϦଆͰ޷͖ͳ஋Λग़
    ͤ͹৭ʑͳ؍఺Ͱ෼ੳͰ͖Δ
    • ᮢ஋Λઃ͚ͯΞϥʔτൃ๒Ͱ͖ΔͷͰ1ͭͷγ
    εςϜͰϝτϦΫεɾϩάͷ؂ࢹΛ؅ཧͰ͖Δ
    

    View Slide

39. @rwle1212
    39
    Graph by Logs
    • େ͖͍มߋΛͨ࣌͠͸εςʔλείʔυͷׂ߹
    Λ֬ೝ
    • ઃఆํ๏
    • Group By: @http.status_code
    • Display: Areas
    • Stacked as: % of overall count: *
    

    View Slide

40. @rwle1212
    40
    Pipelines
    • JSON͡Όͳ͍ϩάΛύʔεͨ͠ΓɺDatadog
    ্Ͱʮ͜ͷϑΟʔϧυ͸εςʔλεѻ͍͢Δʯ
    ͱ͍͏Α͏ͳϚοϐϯάΛߦ͏
    • Nginx΍RedisͳͲҰൠతͳϛυϧ΢ΣΞͷϩά
    ͸IntegrationΛೖΕΕ͹উखʹύʔεͯ͘͠Ε
    ΔͷͰࣗ෼Ͱgrokॻ͘ඞཁ͕ͳ͘ศར
    • ԼਤͷΑ͏ʹܭࢉࣜΛೖΕͨΓɺผ໊ʹม͑ͨ
    Γ͢Δ͜ͱ΋Ͱ͖Δ
    

    View Slide

41. @rwle1212
    41
    Indexes
    • ϩάऩूͷআ֎ϑΟϧλ͕ॻ͚ΔɻIndexed
    LogsͰ՝ۚ͞Εͳ͘ͳΔ
    • 10%͚ͩऔಘ͢Δͱ͍͏͜ͱ΋ՄೳɻΫΤϦͰ
    ͔ͳΓߜΓࠐΉ͜ͱ͕Ͱ͖Δ
    • ϩάΛૹ৴͢ΔAgent/ClientଆͰίϯτϩʔϧ
    ͢Δ͜ͱ΋Մೳ͕ͩɺઃఆϑΝΠϧͷ഑෍౳͕
    ໘౗ͳͷͰ͍ͬͨΜશ෦Datadogʹ౤͛ͯɺ
    IndexsͰϩάΛམͱ͢ͱ͍͏࢖͍ํ͕Φεε
    ϝɻʢ҆Ձ͚ͩͲIngested Logsͷ՝ۚ͸ൃੜ͢
    ΔͷͰ஫ҙʣ
    

    View Slide

42. @rwle1212
    42
    Archives
    • Datadogͷϩά͸ܖ໿ظؒʢ7೔ͱ͔15೔ͱ
    ͔ʣ͔͠ݟΕͳ͍
    • աڈϩά΋௕ظؒอଘ͍ͨ͠৔߹͸ArchivesͰ
    S3΍GCSʹஷΊΔ͜ͱ͕Ͱ͖ΔͷͰɺAthena
    ΍BigQueryͰݟΔ͜ͱ΋Ͱ͖Δ
    • Fluentdͱ͔ͰS3ʹૹΔ͜ͱ΋Ͱ͖Δ͚Ͳɺ
    Datadogͷ৔߹͸ͱΓ͋͑ͣऩू͢ΔઃఆϑΝ
    ΠϧΛૹ৴͢Δ͚ͩͰྑ͍ͷͰɺઃఆϑΝΠϧ
    ͷ͹Β·͖΍ߋ৽͕ͳ͍෼ָ
    

    View Slide

43. @rwle1212
    43
    Archives
    • Rehydrate From Archivesͱ͍͏ػೳΛ࢖͑͹
    ArchivesʹೖͬͯΔσʔλΛDatadog Logsʹ໭
    ͢͜ͱ͕Ͱ͖ΔΈ͍ͨʁ͚ͩͲ࢖ͬͨ͜ͱͳ͍
    ͷͰ஫ҙ఺͕Θ͔ͬͯͳ͍ɻ୭͔ڭ͑ͯ
    

    View Slide

44. @rwle1212
    44
    datadog logsʹ͍ͭͯ࢖༻ײ΍ࣄྫ
    ΞϓϦͷϩάΛDatadog LogsʹೖΕ͍ͯΔͷͰɺ΄΅ຖ೔౷ܭతʹݟͯҟৗ͕ແ͞
    ͦ͏͔βοͱݟ͍ͯΔɻ
    ඞཁ͋Β͹ਂງΓ͢ΔͨΊʹ Graph Ͱ৭ʑͳ؍఺Ͱ෼ੳͨ͠Γɺه࿥࢒࣌͢ɺܦ
    ա؍࡯͢Δ͸NotebooksΛ࡞ͬͯܧଓతʹ༷ࢠΛݟͨΓ͍ͯ͠Δɻ
    

    View Slide

45. @rwle1212
    45
    Πϕϯτ৘ใʹσϓϩΠΠϕϯτͳͲɺͲΜͳ΋ͷΛૹͬ
    ͍ͯΔ͔ɻ
    ࿈ܞͤ͞ΔπʔϧΛ૿΍͢ͷ͕ݏͳͷͰ࿈ܞ͍ͤͯ͞·ͤΜ
    

    View Slide

46. @rwle1212
    46
    APMؔ࿈ɺAnomalyϞχλʔͷ༗༻ͳ࢖͍ํ
    APMɺAnomaly͸࢖͍͜ͳ͓ͤͯΓ·ͤΜ
    

    View Slide

47. @rwle1212
    47
    LogsઃఆͷϕετϓϥΫςΟε
    • ͱΓ͋͑ͣ
    • શ෦ͷϩάΛ1Օॴʹू໿͢Δɻѹ౗తޮ཰্͕Δ
    • Pipeline
    • ύʔεͰ͖Δ͚ͲɺͰ͖ΔݶΓJSONͰૹ৴ͯࣗ͠ಈతʹύʔεͯ͠΋Β͏
    • εςʔλεͱ͔͸Ϛοϐϯάͯ͠ɺϏϡʔϫʔͷ੨ɾԫɾ੺৭දࣔͰҰ໨ྎવʹ͢Δ
    • Indexes
    • ίετ࡟ݮ͢ΔͨΊʹݟͳ͍ϩά͸আ֎͢Δ
    • Archives
    • ೦ͷͨΊաڈϩά͸S3/GCSʹอଘ͢Δ
    • S3/GCSଆͰҰఆظؒޙʹࣗಈ࡟আ͢ΔΑ͏ʹϥΠϑαΠΫϧΛೖΕΔ
    

    View Slide

48. 48
    ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide