何でも屋になっている SRE 的なチームから責務を分離するまでの道のり〜新設チームでオンコール体制を構築するまで〜 / SRE Lounge #15

Slide 1

Slide 1 text

Slide 2

Slide 2 text

ࣗݾ঺հ Takeshi Arabiki • X (چ Twitter): @a_bicky • Blog: ͋Βͼ͖೔ه • ॴଐ: Rerpo גࣜձࣾʢ2017 ೥ 8 ݄ʙʣ • SREɺσʔλΤϯδχΞϦϯάɺΞϓϦέʔγϣ ϯ։ൃɺΞʔΩςΫτͬΆ͍͜ͱΛ΍ΔνʔϜͰ ϚωʔδϟʔΛͭͭ͠ɺΤϯδχΞͱͯ͠͸ίʔ υϨϏϡʔ΍ো֐ରԠɾ҆ఆԽʹؔ͢Δ͜ͱΛϝ Πϯʹɺͨ·ʹػೳ։ൃ΋΍Δਓʢ΋ͬͱίʔυ ॻ͖͍ͨʣ

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

© 2023 Repro Inc. │ CONFIDENTIAL ৘ใऩू ෼ੳɾηάϝϯτ ҰਓͻͱΓʹ࠷దͳࢪࡦΛ࣮ࢪ ଟ༷ͳσʔλΛ෼ੳ ϢʔβʔΛηάϝϯτ ϙοϓΞοϓ ਧ͖ग़͠ ϝʔϧ ΞϓϦ಺઀٬ ΞϓϦ಺઀٬ ΞϓϦ಺઀٬ ίϯςϯπຒΊࠐΈ ϓογϡ௨஌ Ϩίϝϯυ Web ళฮ ΞϓϦ ϦΞϧλΠϜͷ One to One ίϛϡχέʔγϣϯ SDK ͔Βͷେྔͷ ϦΫΤετΛࡹ͘ API େྔͷσʔλΛՃ޻͢Δ ΞϓϦέʔγϣϯ ४ϦΞϧλΠϜʹର৅ ϢʔβʔΛߋ৽͢Δ ΞϓϦέʔγϣϯ

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

© 2023 Repro Inc. │ CONFIDENTIAL Ұ෦ͷΞϓϦέʔγϣϯͷ։ൃɾͦͷଞશൠͷӡ༻ Amazon Corretto (Java) BigQuery Kafka Streams Rundeck Amazon EC2 Amazon ECS Amazon Aurora (MySQL Compatible) Amazon ElastiCache (Redis) Amazon S3 SDK ΍ Rails ͷϑϩϯτΤϯυͷ։ൃҎ֎ ʹؔͯ͠͸΄΅શͯʹਂؔ͘༩

Slide 15

Slide 15 text

Slide 16

Slide 16 text

© 2023 Repro Inc. │ CONFIDENTIAL 16 Ͳ͏ͯ͜͠͏ͳͬͨ ● Rails ΞϓϦέʔγϣϯͷ֤ػೳ୲౰͕ᐆດ - ಛఆͷػೳͷ։ൃ͕׬ྃ͢Δ౓ʹνʔϜΛεΫϥοϓΞϯυϏϧυ - ৽ػೳ։ൃͰ͋·ΓखΛೖΕͳ͍ೃછΈͷͳ͍ΞϓϦέʔγϣϯͷଘࡏ ● εέδϡʔϧΛ༏ઌͯ͠Sys-InfraΛத৺ʹߦͬͨΞϓϦέʔγϣϯ։ൃ ● Chief Architectʢݩ CTOʣ͕ಋೖ͢Δج൫ͷड͚ೖΕઌ͕͍͍ͩͨSys-Infra - ೝ஌ෛՙͷݶք͔Βػೳ։ൃνʔϜ΋ड͚ೖΕઌʹͳΓͭͭ͋ͬͨ ● ͦ΋ͦ΋Sys-Infraͷ੹຿͕ᐆດ - Կ౓͔੹຿Λ੔ཧ͠Α͏ͱ͢Δ΋ड͚ೖΕઌ͕ͳ͘಴࠳

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

© 2023 Repro Inc. │ CONFIDENTIAL 20 Repro Core ৽ઃʹΑͬͯղܾ͍ͨ͠՝୊ ● ReproΛࢧ͑Δॏཁͳج൫ͷӡ༻͕ଐਓԽ͍ͯ͠Δ্ʹઐ೦ͮ͠Β͍ঢ়گ - ސ٬͕௚઀ར༻͠ͳ͍Օॴͷӡ༻ෛՙ͕ Sys-Infra ʹҰۃूத ● Chief Architectͷ஌ࣝͷ఻ঝઌ͕ෆࡏ ● ػೳ։ൃ͢ΔνʔϜ͕஌Βͳ͍ͱ͍͚ͳ͍͜ͱ͕ଟ্͍ʹखް͍αϙʔτ΋ͳ͍

Slide 21

Slide 21 text

© 2023 Repro Inc. │ CONFIDENTIAL 21 Repro Core ৽ઃʹΑͬͯղܾ͍ͨ͠՝୊ ● ReproΛࢧ͑Δॏཁͳج൫ͷӡ༻͕ଐਓԽ͍ͯ͠Δ্ʹઐ೦ͮ͠Β͍ঢ়گ - Repro Core ͕୲͏ ● Chief Architectͷ஌ࣝͷ఻ঝઌ͕ෆࡏ - Repro Core ͕୲͏ ● ػೳ։ൃ͢ΔνʔϜ͕஌Βͳ͍ͱ͍͚ͳ͍͜ͱ͕ଟ্͍ʹखް͍αϙʔτ΋ͳ͍ - Repro Core ͱ Sys-Infra ͕୲͏

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

© 2023 Repro Inc. │ CONFIDENTIAL ৘ใऩू ෼ੳɾηάϝϯτ ҰਓͻͱΓʹ࠷దͳࢪࡦΛ࣮ࢪ ଟ༷ͳσʔλΛ෼ੳ ϢʔβʔΛηάϝϯτ ϙοϓΞοϓ ਧ͖ग़͠ ϝʔϧ ΞϓϦ಺઀٬ ΞϓϦ಺઀٬ ΞϓϦ಺઀٬ ίϯςϯπຒΊࠐΈ ϓογϡ௨஌ Ϩίϝϯυ Web ళฮ ΞϓϦ Repro Coreͷίϯϙʔωϯτͷྫ େྔͷσʔλΛՃ޻͢Δ ΞϓϦέʔγϣϯ ४ϦΞϧλΠϜʹର৅ ϢʔβʔΛߋ৽͢Δ ΞϓϦέʔγϣϯ

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

© 2023 Repro Inc. │ CONFIDENTIAL 28 ౰࣌ͷΦϯίʔϧମ੍ ● ౰࣌ͷ PagerDuty αʔϏε͸ओʹ 4 छྨͰ Feature Team ͱ Sys-Infra Ͱ෼୲ - ϓογϡ௨஌ؔ࿈αʔϏε - Sys-Infra ؔ࿈αʔϏε - Kafka BrokerɾKafka Streams ΞϓϦέʔγϣϯؔ࿈αʔϏε - Rails ΞϓϦέʔγϣϯαʔϏε ● ্هͷαʔϏε͸खಈͰ؅ཧ ● ֤αʔϏεৗʹ Primary ͱ Secondary ͷ 2 ໊͕଴ػ

Slide 29

Slide 29 text

© 2023 Repro Inc. │ CONFIDENTIAL 29 ౰࣌ͷΦϯίʔϧମ੍ ● ౰࣌ͷ PagerDuty αʔϏε͸ओʹ 4 छྨͰ Feature Team ͱ Sys-Infra Ͱ෼୲ - ϓογϡ௨஌ؔ࿈αʔϏε - Sys-Infra ؔ࿈αʔϏε - Kafka BrokerɾKafka Streams ΞϓϦέʔγϣϯؔ࿈αʔϏε - Rails ΞϓϦέʔγϣϯαʔϏε ● ্هͷαʔϏε͸खಈͰ؅ཧ ● ֤αʔϏεৗʹ Primary ͱ Secondary ͷ 2 ໊͕଴ػ ← ࠓޙ͸Repro Core͕Ұ෦Λ୲౰ ← ࠓޙ͸Repro Core͕Ұ෦Λ୲౰ ← ࠓޙ͸Repro Core͕Ұ෦Λ୲౰ ← ࠓޙ͸Repro Core͕Ұ෦Λ୲౰

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

© 2023 Repro Inc. │ CONFIDENTIAL 32 PagerDuty ͷ༻ޠ ● αʔϏε - ઃఆ͞ΕͨΤεΧϨʔγϣϯϙϦγʔʹैͬͯΦϯίʔϧ୲౰ऀʹ௨஌Λߦ͏୯Ґ ● ΤεΧϨʔγϣϯϙϦγʔ - αʔϏεͰΠϯγσϯτ͕࡞੒͞Εͨࡍʹ   Ͳͷεέδϡʔϧʹैͬͯ௨஌͢Δ͔Λܾఆ͢Δ ● εέδϡʔϧ - ࣌ؒଳʹΑͬͯΦϯίʔϧ୲౰ऀΛมߋ͢Δ࢓૊Έ - ௨ৗ͸1νʔϜ1εέδϡʔϧͳͷͰνʔϜͰಡସ͑Մ ΤεΧϨʔγϣϯϙϦγʔ εέδϡʔϧ

Slide 33

Slide 33 text

© 2023 Repro Inc. │ CONFIDENTIAL 33 PagerDuty ͷαʔϏεͱ͸ You can call this something a microservice, a piece of functionality, a feature, a slice of a monolith, a component, a shared piece of infrastructure, or an internal tool. For the purposes of this guide, these somethings will all be labeled as a type of "service." Although the monolith shares the same code base, each logical source of functionality you identify can be considered its own service. That logical service can be represented as a different service for your documentation, runbooks, and wikis, along with your on-call ownership in PagerDuty. Introduction - PagerDuty Full-Service Ownership Documentation Defining a Service - PagerDuty Full-Service Ownership Documentation

Slide 34

Slide 34 text

© 2023 Repro Inc. │ CONFIDENTIAL 34 PagerDuty ͷαʔϏεͱ͸ You can call this something a microservice, a piece of functionality, a feature, a slice of a monolith, a component, a shared piece of infrastructure, or an internal tool. For the purposes of this guide, these somethings will all be labeled as a type of "service." Although the monolith shares the same code base, each logical source of functionality you identify can be considered its own service. That logical service can be represented as a different service for your documentation, runbooks, and wikis, along with your on-call ownership in PagerDuty. Introduction - PagerDuty Full-Service Ownership Documentation Defining a Service - PagerDuty Full-Service Ownership Documentation ϚΠΫϩαʔϏε΍ίϯϙʔωϯτͱ͔ΛαʔϏεͱݺͿΑ ಉ͡ίʔυΛڞ༗͢ΔϞϊϦγοΫαʔϏεͰ΋ҟͳΔػೳ Λఏڙ͍ͯ͠ΔͳΒผͷαʔϏεͱΈͳͤΔΑ

Slide 35

Slide 35 text

© 2023 Repro Inc. │ CONFIDENTIAL 35 ౰࣌ͷΦϯίʔϧମ੍ʢ࠶ܝʣ ● ౰࣌ͷ PagerDuty αʔϏε͸ओʹ 4 छྨͰ Feature Team ͱ Sys-Infra Ͱ෼୲ - ϓογϡ௨஌ؔ࿈αʔϏε - Sys-Infra ؔ࿈αʔϏε - Kafka BrokerɾKafka Streams ΞϓϦέʔγϣϯؔ࿈αʔϏε - Rails ΞϓϦέʔγϣϯαʔϏε ● ্هͷαʔϏε͸खಈͰ؅ཧ ● ֤αʔϏεৗʹ Primary ͱ Secondary ͷ 2 ໊͕଴ػ

Slide 36

Slide 36 text

© 2023 Repro Inc. │ CONFIDENTIAL 36 PagerDuty αʔϏεͷ෼ׂ ● ౰࣌ͷ PagerDuty αʔϏε͸ओʹ 4 छྨͰ Feature Team ͱ Sys-Infra Ͱ෼୲ - ϓογϡ௨஌ؔ࿈αʔϏε → 5 ͭʹ෼ׂ - Sys-Infra ؔ࿈αʔϏε → 3 ͭʹ෼ׂ - Kafka BrokerɾKafka Streams ΞϓϦέʔγϣϯؔ࿈αʔϏε → 2 ͭʹ෼ׂ - Rails ΞϓϦέʔγϣϯαʔϏε → 4 ͭʹ෼ׂ

Slide 37

Slide 37 text

Slide 38

Slide 38 text

© 2023 Repro Inc. │ CONFIDENTIAL 38 PagerDuty ͷ༻ޠʢ࠶ܝʣ ● αʔϏε - ઃఆ͞ΕͨΤεΧϨʔγϣϯϙϦγʔʹैͬͯΦϯίʔϧ୲౰ऀʹ௨஌Λߦ͏୯Ґ ● ΤεΧϨʔγϣϯϙϦγʔ - αʔϏεͰΠϯγσϯτ͕࡞੒͞Εͨࡍʹ   Ͳͷεέδϡʔϧʹैͬͯ௨஌͢Δ͔Λܾఆ͢Δ ● εέδϡʔϧ - ࣌ؒଳʹΑͬͯΦϯίʔϧ୲౰ऀΛมߋ͢Δ࢓૊Έ - ௨ৗ͸1νʔϜ1εέδϡʔϧͳͷͰνʔϜͰಡସ͑Մ ΤεΧϨʔγϣϯϙϦγʔ εέδϡʔϧ

Slide 39

Slide 39 text

© 2023 Repro Inc. │ CONFIDENTIAL ● Repro Core - Repro Core ͚ͩͰΦϯίʔϧʹೖΔ   αʔϏε༻ ● Repro Core Trial with FT - Feature Team ͱҰॹʹΦϯίʔϧʹೖΔ   αʔϏε༻ ● Repro Core Trial with Sys Infra - Sys-Infra ͱҰॹʹΦϯίʔϧʹೖΔ   αʔϏε༻ ● etc. 39 PagerDuty ΤεΧϨʔγϣϯϙϦγʔͷํ਑ Repro Core Trial with Sys Infra ͷྫ

Slide 40

Slide 40 text

© 2023 Repro Inc. │ CONFIDENTIAL 40 PagerDuty εέδϡʔϧͷํ਑ ● Repro Core Trial Primary/Secondary - Repro Core ͱଞͷνʔϜͷΦϯίʔϧ୲౰͕Ұॹʹ଴ػ͢Δεέδϡʔϧ - ฏ೔೔த͸ෆ׳Εͳϝϯόʔ͕ৗʹ଴ػͯ͠ܦݧΛੵΉ - ฏ೔೔தҎ֎͸׳Ε͍ͯΔϝϯόʔ͕଴ػ͢Δ͔ɺ୲౰ෆࡏͷঢ়ଶ ● Repro Core Primary/Secondary - Repro Core ͚͕ͩ଴ػ͢ΔϙϦγʔ - ฏ೔೔த͸ෆ׳Εͳϝϯόʔ͕ৗʹ଴ػͯ͠ܦݧΛੵΉ - ฏ೔೔தҎ֎͸ϩʔςʔγϣϯʹैͬͯ଴ػ - Repro Core Trial Primary/Secondary ͔Β࢝Ίͯͪ͜Βʹ੾Γସ͍͑ͯ͘

Slide 41

Slide 41 text

© 2023 Repro Inc. │ CONFIDENTIAL ● ෆ׳ΕͳαʔϏε͸γϟυʔΠϯά͔Β࢝ΊΔ - γϟυʔΠϯά = ରԠ͍ͯ͠Δਓͷ࡞ۀΛࣗ෼Ͱ΋ͳͧΔ͜ͱ ● ͋Δఔ౓׳Ε͍ͯΔαʔϏε͸ࣗ෼ҰਓͰΦϯίʔϧʹೖ͍ͬͯΔͭ΋ΓͰରԠ - खʹෛ͑ͳ͍΋ͷʹؔͯ͠͸ผνʔϜͷ౰൪ͷਓͷॿ͚ΛआΓΔ ● Φϯίʔϧظؒʹൃੜͨ͠Ξϥʔτʹશͯʹؔͯ࣍͠ͷ 2 ఺Λԡ͑͞Δ - ΞϥʔτΛ์ஔͨ͠৔߹ʹαʔϏεʹͲͷΑ͏ͳӨڹΛ༩͑ΔՄೳੑ͕͋Δ͔ - ΞϥʔτΛղফ͢ΔͨΊʹ͸ͲͷΑ͏ͳ࡞ۀΛߦΘͳ͚Ε͹ͳΒͳ͍͔ ● ࣍ͷΑ͏ͳ͜ͱ΋Ͱ͖Δͱ๬·͍͕͠ඞਢͰ͸ͳ͍ - Ξϥʔτ͕໐ΔݪҼʹ͍ͭͯͷٕज़తͳཧղɾΞϥʔτͷࠜຊղফ 41 ΦϯίʔϧҠߦظؒதͷ৺ߏ͑

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

© 2023 Repro Inc. │ CONFIDENTIAL ● ࠓͷ Repro Core ͸Ҏલͷ Sys-Infra ฒΈʹೝ஌ෛՙ͕େ͖͍ ● ೝ஌ෛՙͷݶքʹ߹ΘͤͯνʔϜΛ෼ׂ͍͕ͨ͠෼ׂͰ͖Δ΄Ͳͷن໛͡Όͳ͍ ● Φϯίʔϧ΋ߟ͑Δͱ 1 νʔϜ࠷௿ 4 ਓ͸֬อ͍ͨ͠ - ೖ໳؂ࢹʹΑΕ͹ɺΦϯίʔϧͰηΧϯμϦΛ༻ҙ͢ΔͳΒ࠷௿8ਓ͸ඞཁ ● Repro Core͸ࣾ಺ͷϝϯόʔͰ݁੒͞ΕͨͷͰ࠾༻ϑϩʔ͕੔͍ͬͯͳ͍ - ͜Ε͔Β੔උ͍͔ͯ͠ͳ͚Ε͹… 44 ਓ͕଍Γͳ͍ʂʂ

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Slide 47

Slide 47 text

© 2023 Repro Inc. │ CONFIDENTIAL 47 ·ͱΊ ● ໿1೥લɺReproͰ͸SREతͳνʔϜ͕ԿͰ΋԰ʹͳ͍ͬͯͨ ● ৽νʔϜઃཱ΍૊৫ମ੍มߋʹ൐ͬͯ੹຿Λ੔ཧͨ͠ - ৽νʔϜઃཱ͸੹຿Λ੔ཧ͢Δνϟϯεʂ ● ৽ઃνʔϜͰطଘνʔϜ͔Β੹຿Λണ͕ͯ͠Φϯίʔϧମ੍Λߏங͢Δ޻෉ - PagerDuty ͷαʔϏεཻ౓ͷݟ௚͠ - Terraform Ͱ؅ཧ͢Δ͜ͱͰมߋΛՃ͑΍ͨ͘͢͠ - ΦϯίʔϧʹೖΔ࣌ؒଳͷ޻෉ - ΦϯίʔϧҠߦظؒதͷ৺ߏ͑Λ໌จԽ

Slide 48

Slide 48 text

Slide 49

Slide 49 text

© 2023 Repro Inc. │ CONFIDENTIAL 49 طଘίϯϙʔωϯτ΍ඞཁͳ஌ࣝͷΩϟονΞοϓ ● طଘίϯϙʔωϯτͷϦϑΝΫλϦϯάɾόʔδϣϯΞοϓ ● ಡॻձͷ࣮ࢪ - σʔλࢦ޲ΞϓϦέʔγϣϯσβΠϯ - ιϑτ΢ΣΞΞʔΩςΫνϟɾϋʔυύʔπ ● ୲౰ίϯϙʔωϯτ୯ҐͰεΩϧϚοϓΛ࡞੒ - جૅతͳεΩϧ͸ίϯϙʔωϯτͷεΩϧʹؚ·ΕΔ ● ษڧձͷ࣮ࢪ - εΩϧϚοϓͰॏཁ͔ͭଐਓੑͷߴ͍΋ͷΛத৺ʹ࣮ࢪ ● Ұ෦ͷίϯϙʔωϯτͰ͸ staging ؀ڥͰٖࣅతͳো֐Λൃੜͤͯ͞ରԠ܇࿅Λ࣮ࢪ

Slide 50

Slide 50 text

© 2023 Repro Inc. │ CONFIDENTIAL 50 GRPI ϞσϧΛҙࣝͨ͠νʔϜϏϧσΟϯά ● Goal: νʔϜͱͯ͠ͷํ޲ੑʹ͍ͭͯશһ͕ཧղ͍ͯ͠Δ - → νʔϜͷ੹຿ɾ໨ඪ౳ʹ͍ͭͯσΟεΧογϣϯ ● Role: ͓ޓ͍ʹର͢Δظ଴஋ʢ໾ׂʣΛཧղ͍ͯ͠Δ - → ࣗ෼ͷ໾ׂɾ૬खʹظ଴͢Δ໾ׂͷ͢Γ߹Θͤ ● Process: ීஈͲ͏ߦಈ͢΂͖͔ɾͲ͏໨ඪΛୡ੒͢Δ͔શһ͕ཧղ͍ͯ͠Δ - → ੹຿ɾ໨ඪ͔Β۩ମతͳλεΫʹམͱ͢ɾWorking Agreement ͷઃఆ ● Interaction: ͓ޓ͍ͷಛੑʢੑ֨ɾՁ஋؍౳ʣΛཧղ͍ͯ͠Δ - → ετϨϯάεϑΝΠϯμʔ౳Λར༻ͨ͠ϫʔΫγϣοϓͷ࣮ࢪ

Slide 51

Slide 51 text

© 2023 Repro Inc. │ CONFIDENTIAL 51 Repro Coreͷϛογϣϯɾ੹຿ ● ϛογϣϯ - Repro ͷػೳΛఏڙ͢Δ্Ͱத֩ͱͳΔج൫Λఏڙ͠ɺFeature Team ͕ސ٬Ձ஋ ͷఏڙʹूதͰ͖Δ૊৫Λ࡞Δ ● ੹຿ - SDK͔ΒͷσʔλΛଞͷαʔϏεͰར༻Ͱ͖ΔΑ͏ʹ֤छσʔλετΞʹอଘ͢Δ - Feature Team ͕ػೳ։ൃ͢Δࡍʹར༻͢Δڞ௨ج൫ɾϥΠϒϥϦͷ։ൃ - ্هڞ௨ج൫ɾϥΠϒϥϦΛ࢖ͬͨ։ൃͷࢧԉ - Feature Team ͷઃܭ૬ஊ

Slide 52

Slide 52 text

Slide 53

Slide 53 text

© 2023 Repro Inc. │ CONFIDENTIAL ● Datadog - Πϯϑϥ౳ͷϞχλϦϯάαʔϏε ● Rollbar - ΞϓϦέʔγϣϯΞϥʔτΛ؅ཧ͢ΔαʔϏε ● Rundeck - δϣϒεέδϡʔϥ ● Papertrail - ϩά؅ཧαʔϏε ● CloudWatch Logs - AWS ͷϞχλϦϯάαʔϏε 53 PagerDuty ͱ࿈ܞ͍ͯ͠ΔओͳαʔϏε

Slide 54

Slide 54 text

Slide 55

Slide 55 text

© 2023 Repro Inc. │ CONFIDENTIAL 55 DatadogϞχλʔͷPagerDuty࿈ܞઌͷมߋʢม਺ฤʣ locals { clusters = flatten(values(var.services).*.clusters) } resource "datadog_monitor" "cpu" { ... query = "avg(last_5m):cpu{cluster NOT IN (${join(",", local.clusters)})} ..." message = <<-EOF {{#is_alert}} @pagerduty-Sys-Infra {{/is_alert}} {{#is_recovery}} @pagerduty-resolve {{/is_recovery}} EOF } resource "datadog_monitor" "cpu_others" { for_each = var.services ... query = "avg(last_5m):cpu{cluster IN (${join(",", each.value.clusters)})) ..." message = <<-EOF {{#is_alert}} ${each.value.on_alert} {{/is_alert}} {{#is_recovery}} ${each.value.on_recovery} {{/is_recovery}} EOF }

Slide 56

Slide 56 text

© 2023 Repro Inc. │ CONFIDENTIAL 56 DatadogϞχλʔͷPagerDuty࿈ܞઌͷมߋʢ෼ذฤʣ resource "datadog_monitor" "consumer_lag" { ... query = "min(last_10m):max:kafka.consumer_lag{*} by {consumer_group} > 1000" message = <<-EOF {{#is_alert}} {{#is_match "consumer_group.name" "xxxxx" }} @pagerduty-Push-Service {{else}} @pagerduty-Repro-Core-Service-1 {{/is_match}} {{/is_alert}} EOF }

Slide 57

Slide 57 text

Slide 58

Slide 58 text

© 2023 Repro Inc. │ CONFIDENTIAL ● ReproͰ͸ΞϓϦέʔγϣϯΞϥʔτͷ֤छ࿈ܞʹ Rollbar Λ࢖༻ ● ReproͷRailsΞϓϦέʔγϣϯ͸ϞϊϨϙͰRollbarͷϓϩδΣΫτ͸ڞ௨ - ECSαʔϏε͸Կݸʹ΋෼͔Ε͍ͯΔ ● RollbarͰϓϩδΣΫτΛ෼͚ΔͱϓϩδΣΫτΛԣஅͯ͠ΞϥʔτΛ֬ೝͮ͠Β͍ - ෳ਺ͷϓϩδΣΫτΛԣஅͯ͠ΞϥʔτΛ֬ೝͰ͖Δ͕ ● Rollbarͷ௨஌ઌͷഉଞ੍ޚ͸େม - Τϥʔϝοηʔδʹ A ؚ͕·ΕͨΒ PagerDuty ͷ Service A ʹ incident Λ࡞੒ - Τϥʔϝοηʔδʹ B ؚ͕·ΕͨΒ PagerDuty ͷ Service B ʹ incident Λ࡞੒ - → Τϥʔϝοηʔδʹ A ͱ B ؚ͕·Ε͍ͯͨΒͲͪΒʹ΋ incident ͕࡞੒͞ΕΔ 58 ΞϓϦέʔγϣϯΞϥʔτͷPagerDuty࿈ܞઌมߋͷ՝୊

Slide 59

Slide 59 text

Slide 60

Slide 60 text