Upgrade to Pro — share decks privately, control downloads, hide ads and more …

アクセス数5倍の急拡大サービスと共に成長するチーム / Accelerate product and team growth with Datadog

6e910ea53b24d30c3fb5cf6ae0e313f5?s=47 Kaga
April 14, 2021

アクセス数5倍の急拡大サービスと共に成長するチーム / Accelerate product and team growth with Datadog

2021/4/14 Datadog Observability Dayでの登壇資料です。
https://hopin.com/events/datadog-observability-day-spring-edition

6e910ea53b24d30c3fb5cf6ae0e313f5?s=128

Kaga

April 14, 2021
Tweet

Transcript

  1. ΞΫηε਺5ഒͷٸ֦େαʔϏε ͱڞʹ੒௕͢ΔνʔϜ 2021/04/14 Datadog Observability Day Takashi Kaga @ CAMPFIRE

  2. ࣗݾ঺հ ɾՃզ وࢤ (͔͕ ͔ͨ͠)ɹ@TAKA_041 1 ɾגࣜձࣾCAMPFIRE ϓϩμΫτ։ൃ෦ ɹSREνʔϜॴଐ ɾ2020೥10݄ʹCAMPFIREೖࣾ

    ɹझຯ͸DatadogΛ͍͡Δ͜ͱ
  3. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g

    5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
  4. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g

    5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
  5. CAMPFIREͷ঺հ

  6. CAMPFIREͷ঺հ

  7. CAMPFIREͷ঺հ

  8. CAMPFIREͷ঺հ

  9. CAMPFIREͷ։ൃνʔϜ

  10. CAMPFIREͷΠϯϑϥߏ੒

  11. CAMPFIREͷΠϯϑϥߏ੒

  12. CAMPFIREͷΠϯϑϥߏ੒

  13. CAMPFIREͷΠϯϑϥߏ੒

  14. CAMPFIREͷΠϯϑϥߏ੒ ECS on EC2͔Β ECS FargateʹҠߦத

  15. CAMPFIREͷαʔϏε੒௕

  16. CAMPFIREͷαʔϏε੒௕ ϦΫΤετ਺ 4ഒ ϦΫΤετ਺ 5ഒ

  17. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g

    5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
  18. CAMPFIREͷαʔϏε੒௕

  19. CAMPFIREͷαʔϏε੒௕

  20. ౰࣌ͷରԠ ɾΠϯελϯεͷ୆਺ͱεϖοΫͷݟ௚͠ ɾεϩʔΫΤϦͷղফ ɾϞχλϦϯάͷݟ௚͠ ɾϏδωεαΠυͱີͳίϛϡχέʔγϣϯ

  21. ౰࣌ͷରԠ Ұϲ݄ͰΞΫηε͕4ഒʹͳͬͨ࿩ https://note.com/campfire_dev/n/n99fba1c86328

  22. ౰࣌ͷରԠΛৼΓฦͬͯ Πϯϑϥ͚ͩͰ͸ͳ͘ ΞϓϦέʔγϣϯ͔Β΋ վળΛߦ͍͖͍ͬͯͨ

  23. ΞϓϦέʔγϣϯ͔Βͷվળ վળʹ͋ͨͬͯͷݱঢ়੔ཧ ɹɾΠϯϑϥͷϝτϦΫε͸औಘͰ͖͍ͯΔ ɹɾΞϓϦέʔγϣϯͷϩά͸S3ʹอଘͰ͖͍ͯΔ

  24. ΞϓϦέʔγϣϯ͔Βͷվળ վળʹ͋ͨͬͯͷݱঢ়੔ཧ ɹɾΠϯϑϥͷϝτϦΫε͸औಘͰ͖͍ͯΔ ɹɾΞϓϦέʔγϣϯͷϩά͸S3ʹอଘͰ͖͍ͯΔ ɹɾΞϓϦέʔγϣϯͷৄࡉͳಈ͖Λ೺ѲͰ͖͍ͯͳ͍

  25. ΞϓϦέʔγϣϯ͔Βͷվળ ΞϓϦέʔγϣϯͷ ΦϒβʔόϏϦςΟ

  26. ΦϒβʔόϏϦςΟʁ ɾObservability = Մ؍ଌੑ ɹɾγεςϜͷ಺෦ঢ়ଶΛ؍ଌ͢ΔͨΊͷೳྗ

  27. ΦϒβʔόϏϦςΟʁ ɾObservability = Մ؍ଌੑ ɹɾγεςϜͷ಺෦ঢ়ଶΛ؍ଌ͢ΔͨΊͷೳྗ ɹɹɾͦͷγεςϜ͸Կ͕Ͳ͏ಈ͍͍ͯΔ͔

  28. ϞχλϦϯάͱΦϒβʔόϏϦςΟ “ϞχλϦϯά͸νʔϜ͕γεςϜͷঢ়ଶΛ؂ࢹͯ͠೺ѲͰ͖ΔΑ͏ʹ͢Δπʔ ϧ΍ٕज़తͳιϦϡʔγϣϯͰ͢ɻϞχλϦϯάͰ͸ɺࣄલʹఆٛͨ͠ࢦඪ΍ϩ άͷऩूΛߦ͍·͢ɻ ΦϒβʔόϏϦςΟ͸νʔϜ͕γεςϜͷσόοάΛੵۃతʹߦ͑ΔΑ͏ʹ͢Δ πʔϧ΍ٕज़తͳιϦϡʔγϣϯͰ͢ɻΦϒβʔόϏϦςΟͰ͸ɺࣄલʹఆٛ͞ Ε͍ͯͳ͍ϓϩύςΟͱύλʔϯͷௐࠪΛߦ͍·͢ɻ” DevOps ଌఆ: ϞχλϦϯάͱΦϒβʔόϏϦςΟ

    https://cloud.google.com/solutions/devops/devops-measurement-monitoring-and-observability
  29. ϗϫΠτϘοΫε ϞχλϦϯά “࠷΋Ұൠతͳཁૉͱͯ͠͸ɺࢦඪɺϩάɺτϨʔεͷ 3 ͕ͭ͋Γ·͢ɻҰ෦ͷ ϞχλϦϯά γεςϜͰ͸ΠϕϯτΛ௥੻ͯ͠ใࠂ͠·͢ɻΠϕϯτ͸ɺγες ϜͰͷϢʔβʔͷૢ࡞΍γεςϜࣗମͷঢ়ଶͷมԽΛද͠·͢ɻ ʢதུʣ ࢦඪɺϩάɺτϨʔεΛଌఆର৅ͷαʔόʔ·ͨ͸ྡ઀͢ΔΤʔδΣϯτ͔ΒϞ

    χλϦϯά γεςϜʹૹ৴͠ɺγεςϜͷঢ়ଶΛ֬ೝ·ͨ͸ਪଌͰ͖·͢ɻ ” DevOps ଌఆ: ϞχλϦϯάͱΦϒβʔόϏϦςΟ https://cloud.google.com/solutions/devops/devops-measurement-monitoring-and-observability
  30. CAMPFIREͱΦϒβʔόϏϦςΟ CloudWatchʹΑΓ ϝτϦΫεΛऔಘ ֤छϩά͸ S3ʹอଘ ͜͜ͰτϨʔεΛ ୲౰ͤ͞Δ

  31. ΞϓϦέʔγϣϯͷτϨʔε ɾAPM(Application Performance Monitoring)Λ༻͍Δ ɹɾDatadog΍NewRelicɺSplunkͳͲ͕ఏڙ ɹɹɾಋೖ͸αʔϏεܦݧऀ͕ϦʔυͰ͖ͨํ͕ྑ͍ ɹɹɾSREνʔϜʹDatadogར༻ܦݧऀ͕2໊(ࣗ෼ؚΉ)

  32. ΞϓϦέʔγϣϯͷτϨʔε ɾAPM(Application Performance Monitoring)Λ༻͍Δ ɹɾDatadog΍NewRelicɺSplunkͳͲ͕ఏڙ ɹɹɾಋೖ͸αʔϏεܦݧऀ͕ϦʔυͰ͖ͨํ͕ྑ͍ ɹɹɾSREνʔϜʹDatadogར༻ܦݧऀ͕2໊(ࣗ෼ؚΉ)

  33. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g

    5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
  34. DatadogಋೖʹΑΔվળͷτϥΠ վળͷΰʔϧ ɹɾDatadog APMʹΑΓγεςϜͷ಺෦ঢ়ଶΛ؍ଌͰ͖Δ ɹɾ؍ଌ݁Ռ͔ΒΑΓྑ͍ϞχλϦϯά͕ߦ͍͑ͯΔ

  35. DatadogಋೖͷྲྀΕ ɾ΍Γ͍ͨ͜ͱ ɹɾAPMಋೖʹΑΔΞϓϦέʔγϣϯͷτϨʔγϯά

  36. DatadogಋೖͷྲྀΕ ɾ΍Γ͍ͨ͜ͱ ɹɾAPMಋೖʹΑΔΞϓϦέʔγϣϯͷτϨʔγϯά ɾ΍ͬͨ͜ͱ ɹɾDatadogಋೖͷॾʑ(APMɺμογϡϘʔυɺLogs ) ɹɾυΩϡϝϯτͷ੔උ ɹɾϨΫνϟʔձͷ։࠵

  37. Datadogಋೖલ

  38. Datadogಋೖޙ αΠυΧʔͰDatadog AgentίϯςφΛ഑ஔ fluentdίϯςφ͔Β Datadog Logsʹసૹ

  39. DatadogಋೖͷྲྀΕ ɾ΍Γ͍ͨ͜ͱ ɹɾAPMಋೖʹΑΔΞϓϦέʔγϣϯͷτϨʔγϯά ɾ΍ͬͨ͜ͱ ɹɾDatadogಋೖͷॾʑ(APMɺμογϡϘʔυɺLogs ) ɹɾυΩϡϝϯτͷ੔උ ɹɾϨΫνϟʔձͷ։࠵

  40. υΩϡϝϯτ&ϨΫνϟʔ ɾಋೖͯ͠ऴΘΓʹͨ͘͠ͳ͍ ɹɾͦ΋ͦ΋DatadogͰԿ͕Ͱ͖Δͷ͔Λ෍ڭ͢Δ ɹɾಋೖ͢ΔϝϦοτ͕͋ΔࣄΛ෍ڭ͢Δ ɹɾAPMͷ͢͝͞ͱศར͞Λ෍ڭ͢Δ ɹɾՃզ͕࣭໰૭ޱΛ୲౰͢Δ

  41. υΩϡϝϯτ&ϨΫνϟʔ ɾಋೖͯ͠ऴΘΓʹͨ͘͠ͳ͍ ɹɾͦ΋ͦ΋DatadogͰԿ͕Ͱ͖Δͷ͔Λ෍ڭ͢Δ ɹɾಋೖ͢ΔϝϦοτ͕͋ΔࣄΛ෍ڭ͢Δ ɹɾAPMͷ͢͝͞ͱศར͞Λ෍ڭ͢Δ ɹɾՃզ͕࣭໰૭ޱΛ୲౰͢Δ ࣾ಺ͰͷΤόϯδΣϦετ తͳಈ͖Ͱαϙʔτ

  42. DatadogಋೖͷྲྀΕ Datadog APMͷ׆༻Ͱര଎վળͷר https://note.com/campfire_dev/n/ne027713b8a61

  43. Datadog APM “Datadog APM ͱ෼ࢄܕτϨʔγϯά͸ɺWeb αʔϏεɺΩϡʔɺσʔλϕʔε͕ ϦΫΤετɺΤϥʔɺϨΠςϯγʔΛ؂ࢹ͢ΔͨΊͷඪ४ͷύϑΥʔϚϯεμο γϡϘʔυΛ࢖༻ͯ͠ɺΞϓϦέʔγϣϯΛৄࡉʹՄࢹԽ͠·͢ɻ෼ࢄܕτϨʔε ͸ɺϗετɺίϯςφɺϓϩΩγɺαʔόʔϨεػೳશମͰɺϒϥ΢βηογϣϯɺ ϩάɺsynthetic

    νΣοΫɺωοτϫʔΫɺϓϩηεɺΠϯϑϥετϥΫνϟʔͷϝ τϦΫεʹγʔϜϨεʹؔ࿈෇͚ΒΕ·͢ɻ” APMͱ෼ࢄܕτϨʔγϯά https://docs.datadoghq.com/ja/tracing/
  44. Datadog APMͰग़དྷΔ͜ͱ ɾϦΫΤετʙϨεϙϯεͷྲྀΕΛՄࢹԽ ɹɾspanͱ͍͏࿦ཧతͳ࡞ۀ୯ҐͰ·ͱΊΒΕΔ ɹɹɾnginx, rails, redis, mysqlͳͲͷҰ௨Γͷॲཧ ɹɹɾ࣮ߦ͞ΕͨSQLͷΫΤϦ·ͰՄࢹԽ

  45. Datadog APMͰग़དྷΔ͜ͱ

  46. Datadog APMͰग़དྷΔ͜ͱ

  47. Datadog APMͰग़དྷΔ͜ͱ

  48. DatadogಋೖʹΑΔվળͷτϥΠ վળͷΰʔϧ ɹɹDatadog APMʹΑΓγεςϜͷ಺෦ঢ়ଶΛ؍ଌͰ͖Δ ɹɾ؍ଌ݁Ռ͔ΒΑΓྑ͍ϞχλϦϯά͕ߦ͍͑ͯΔ

  49. DatadogಋೖʹΑΔվળͷτϥΠ վળͷΰʔϧ ɹɹDatadog APMʹΑΓγεςϜͷ಺෦ঢ়ଶΛ؍ଌͰ͖Δ ɹɾ؍ଌ݁Ռ͔ΒΑΓྑ͍ϞχλϦϯά͕ߦ͍͑ͯΔ
 ɹɹ(in progress)

  50. ϞχλϦϯάͷઃܭ ɾΑΓޮՌతͳϞχλϦϯάΛߟ͑Δ

  51. ϞχλϦϯάͷઃܭ ɾΑΓޮՌతͳϞχλϦϯάΛߟ͑Δ ɹɾϢʔβʔମݧʹӨڹ͕͋Δ΋ͷ ɹɾϏδωεཁ݅ʹӨڹ͕͋Δ΋ͷ

  52. ϞχλϦϯάͷઃܭ ɾΑΓޮՌతͳϞχλϦϯάΛߟ͑Δ ɹɾϢʔβʔମݧʹӨڹ͕͋Δ΋ͷ ɹɾϏδωεཁ݅ʹӨڹ͕͋Δ΋ͷ ΠϯϑϥدΓͷϞχλϦϯά͔Β ΞϓϦέʔγϣϯدΓͷϞχλϦϯά΁

  53. Datadog͕΋ͨΒͨ͠΋ͷ νʔϜͷ։ൃྗ޲্

  54. νʔϜͷ։ൃྗ޲্ ɾಉ͡ਓ਺Ͱ΋͞Βʹଟ͘ͷࣄ͕Ͱ͖ΔΑ͏ʹ ɹɾૣظͷΤϥʔݕ஌ ɹɾஅଓతʹൃੜ͍ͯ͠ΔΤϥʔͷղফ ɹɾఆྔతͳύϑΥʔϚϯεվળ ɹɾΑΓ౿ΈࠐΜͩΞϥʔτରԠͷٞ࿦

  55. νʔϜͷ։ൃྗ޲্ ɾಉ͡ਓ਺Ͱ΋͞Βʹଟ͘ͷࣄ͕Ͱ͖ΔΑ͏ʹ ɹɾૣظͷΤϥʔݕ஌ ɹɾஅଓతʹൃੜ͍ͯ͠ΔΤϥʔͷղফ ɹɾఆྔతͳύϑΥʔϚϯεվળ ɹɾΑΓ౿ΈࠐΜͩΞϥʔτରԠͷٞ࿦

  56. அଓతΤϥʔͷղফ ɾAPMಋೖޙɺஅଓతʹൃੜ͍ͯ͠Δ5XXΤϥʔ͕ՄࢹԽ ɹɾ։ൃνʔϜͱSREνʔϜͱͰ͜ΕΛௐࠪ ɹɾ౰೔தʹݪҼ൑໌ & σϓϩΠʹͯղফ ɹɾDev🤝Ops

  57. அଓతΤϥʔͷղফ

  58. அଓతΤϥʔͷղফ

  59. Datadog͕΋ͨΒͨ͠΋ͷ Datadog͕ νʔϜͷ੒௕ΛՃ଎ͤ͞Δ

  60. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g

    5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
  61. ܦӦ՝୊ʹ׆༻͢ΔDatadog ɾτοϓϖʔδͷ଎౓վળ͕ܦӦ՝୊ʹ ɹɾ͜Ε·Ͱ΋ܧଓͯ͠ௐࠪΛ࣮ࢪ͖ͯͨ͠

  62. ܦӦ՝୊ʹ׆༻͢ΔDatadog ɾτοϓϖʔδͷ଎౓վળ͕ܦӦ՝୊ʹ ɹɾ͜Ε·Ͱ΋ܧଓͯ͠ௐࠪΛ࣮ࢪ͖ͯͨ͠ ɹɹɾ֎ܗ؂ࢹʹΑΔԠ౴଎౓ͷௐࠪ ɹɹɾίʔυϦʔσΟϯάʹΑΔௐࠪ

  63. ܦӦ՝୊ʹ׆༻͢ΔDatadog Ͳ͏ͤͳΒDatadogΛ ׆༻ͯ͠Έ·ͤΜ͔ʁ

  64. ܦӦ՝୊ʹ׆༻͢ΔDatadog ɾτοϓϖʔδͷ଎౓վળ͕ܦӦ՝୊ʹ ɹɾ͜Ε·Ͱ΋ܧଓͯ͠ௐࠪΛ࣮ࢪ͖ͯͨ͠ ɹɹɾ֎ܗ؂ࢹʹΑΔԠ౴଎౓ͷௐࠪ ɹɹɾίʔυϦʔσΟϯάʹΑΔௐࠪ ɹɹɾDatadog APMʹΑΔΤϯυϙΠϯτ୯ҐͰͷௐࠪ ɹɹɾμογϡϘʔυͰͷܧଓతͳϝτϦΫενΣοΫ

  65. ܦӦ՝୊ʹ׆༻͢ΔDatadog ಛఆͷׂ߹ͷϢʔβʔʹର͠ ଎౓վળͷϦϦʔεΛ࣮ࢪ ૝ఆͨ͠ޮՌ͕֬ೝͰ͖ͨͷͰ ඍௐ੔ͨ͠ޙʹ࠶ϦϦʔε

  66. ܦӦ՝୊ʹ׆༻͢ΔDatadog ɾτοϓϖʔδͷ଎౓վળ͕ܦӦ՝୊ʹ ɹɾ͜Ε·Ͱ΋ܧଓͯ͠ௐࠪΛ࣮ࢪ͖ͯͨ͠ ɹɹɾDatadogʹΑΓৄࡉ͔ͭఆྔతͳݕূ͕Մೳʹ

  67. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g

    5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
  68. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ ɾϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadogͰͷSLI/SLO੔උ

  69. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ ɾϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadogͰͷSLI/SLO੔උ

  70. ϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾDatadog RUM(Real User Monitoring)ͷಋೖ

  71. ϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾDatadog RUM(Real User Monitoring)ͷಋೖ ɹɾΑΓϢʔβʔʹ͍ۙҐஔ͔ΒͷϞχλϦϯά ɹɹɾϢʔβʔମݧʹӨڹ͕͋Δ΋ͷ

  72. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ ɾϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadogͰͷSLI/SLO੔උ

  73. ίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadog Continuous Profilerͷಋೖ

  74. ίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadog Continuous Profilerͷಋೖ ɹɾίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɹɹɾ։ൃऀ͕ΑΓߴ඼࣭ͳίʔυΛॻ͚ΔΑ͏ʹ ɹɹɾطଘͷίʔυͷϦϑΝΫλϦϯάΛࢧԉ

  75. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ ɾϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadogͰͷSLI/SLO੔උ

  76. DatadogͰͷSLI/SLO੔උ ɾDatadogͰͷSLI/SLO؅ཧ

  77. DatadogͰͷSLI/SLO੔උ ɾDatadogͰͷSLI/SLO؅ཧ ɹɾαʔϏεͷ඼࣭ΛఆΊɺҡ࣋͢Δ ɹɹɾϨΠςϯγͳͲͷύϑΥʔϚϯεόδΣοτ ɹɹɾ5XXΤϥʔͳͲͷΤϥʔόδΣοτ

  78. ·ͱΊ

  79. ·ͱΊ ɾ1೥ؒͰΞΫηε਺5ഒʹαʔϏε͕੒௕ ɾͦΕʹ൐͍ϞχλϦϯάͷվળ͕ٸ຿ʹ ɾDatadog APMͷಋೖͰΦϒβʔόϏϦςΟΛ࣮ݱ ɾDatadog͕༷ʑͳ՝୊ղܾΛޙԡ͠ ɾࠓޙ΋ܧଓతͳαʔϏεվળʹDatadogΛ׆༻

  80. ʲએ఻ʳ

  81. ҰਓͰ΋ଟ͘ҰԁͰ΋ଟ͘ɺ ૝͍ͱ͓͕ۚΊ͙ΔੈքΛͭ͘Δɻ