一人から始めたSREチーム3年の歩み - 求められるスキルの変化とチームのあり方 - / The three-year journey of the SRE team, which started all by myself

Slide 1

Slide 1 text

Ұਓ͔Β࢝Ίͨ SREνʔϜ3೥ؒͷาΈ - ٻΊΒΕΔεΩϧͷมԽͱνʔϜͷ͋Γํ - VTRyo Money Forward, Inc - Platform and Reliability Engineeringຊ෦ SRE Kaigi 2025

Slide 2

Slide 2 text

SES, ࣗࣾ։ൃ, ελʔτΞοϓͱ਺ࣾ ܦͯ2021೥ʹϚωʔϑΥϫʔυೖࣾ ଞελʔτΞοϓʹ΋ࢀըத ࠷ۙ͸EbitengineͰήʔϜ੍࡞த VTRyo -> VTuberͱ͸ͳΜͷؔ܎΋͋Γ·ͤΜ Money Forward TECH DAY’24 Developers Boost 2023 ࣗ෼͚ͩͷ୭΋૝૾Ͱ͖ͳ͍ΩϟϦΞͷҭͯํ ϕετεϐʔΧʔ৆ SRE NEXT 2022 Ұਓ͔Β࢝ΊΔϓϩμΫτSRE ੿ஶ ʮITΤϯδχΞͷͨΊͷ ۮવ͔Β࢝ΊΔ ΩϟϦΞϓϥϯʯ

Slide 3

Slide 3 text

ࡢ೥Ϋϥϑτ🍺ͷ৹ࠪһࢿ֨Λऔಘ α΢φ݈߁࢜, ೤೾ࢣࢿ֨ࡁΈ झຯ͕ଟ͍ͷͰ࠙਌ձͰ΋ͥͻ ϏʔϧΛҿΈʹυΠπɾνΣίɾϕϧΪʔ ΧϨʔΛ৯΂ʹεϦϥϯΧ ϥʔϝϯೋ࿠Λ৯΂ʹશࠃ Ҏલ͸ٕज़ಉਓࢽ΋੍࡞ͯ͠·ͨ͠ ଞʹ΋͋Δ͚Ͳॻ͖͖Εͳ͍ͷͰུ

Slide 4

Slide 4 text

͓஌Βͤ SRE Kaigi 2025͸͔ͤͬ͘ͷΦϑϥΠϯ։࠵ͳͷͰ… εϥΠυ͸ਤΛத৺ʹ ݴ༿Ͱઆ໌ͨ͠΋ͷ͸ޙ೔ϒϩάʹ Xʹͯ #srekaigi #room_b Ͱͷ࣮گײ૝͸େ׻ܴͰ͢

Slide 5

Slide 5 text

ࠓ೔࣋ͪؼΕΔ͜ͱ ૊৫ઃܭʹؔΘ͍ͬͯΔํ اۀ΍νʔϜͷϑΣʔζʹΑͬͯSREͷ͋Γํ͕ Ͳ͏มΘΔͷ͔ ͲͷϑΣʔζʹͲΜͳεΩϧΛ࣋ͬͨSRE͕͍Δͱ Α͍ͷ͔ɻνʔϜઃܭͷࢀߟʹ

Slide 6

Slide 6 text

ࠓ೔࣋ͪؼΕΔ͜ͱ SREݸਓͷํ ϑΣʔζʹΑͬͯٻΊΒΕΔεΩϧηοτ͕ Ͳ͏มΘΔ͔ ૲૑ظɺ֦େظɺ҆ఆظ͕͋ͬͨͱͯ͠ɺ ࣗ෼͕Ͳͷن໛ͷSREͱͯ͠ద೚ͳͷ͔

Slide 7

Slide 7 text

ઌʹࣗຫ͍͍ͯ͠Ͱ͔͢ɺ͠·͢Ͷ ࣗࣾͷ൒ظදজࣜʹͯνʔϜ͕ϓϩδΣΫτ৆Λ֫ಘ͠·ͨ͠ʂ

Slide 8

Slide 8 text

໨࣍ ࣌ܥྻͰݟΔࢲͨͪSREͷऔΓ૊Έ 3೥ͰมΘͬͨ͜ͱɺมΘΒͳ͍͜ͱ SREݸਓͱͯ͠ߟ͍͑ͯͨ͜ͱ ·ͱΊ

Slide 9

Slide 9 text

ϚωʔϑΥʔϫʔυͷSREʹ͍ͭͯ

Slide 10

Slide 10 text

࣌ܥྻͰݟΔࢲͨͪSREͷऔΓ૊Έ ॱΛ௥ͬͯ঺հ͍͖ͯ͠·͢ Ұਓ໨ͷSREظ νʔϜ֦େظ ҆ఆظ

Slide 11

Slide 11 text

Ұਓ໨ͷSREظ ϩʔυϚοϓ ࠷ॳʹܭը͞Εͨ͜ͱ͕ มߋʹͳΔՄೳੑ͕͋Δ Embedded SRE ઙ͘ͱ΋޿͍ܦݧ ࣮ߦྗ ΧΦεঢ়ଶΛ੔͑ΒΕΔ ମ੍ εΩϧηοτ

Slide 12

Slide 12 text

Ұਓ໨ͷSREظ

Slide 13

Slide 13 text

Ұਓ໨ͷSREظ Ҿ༻: SRE NEXT 2022 Ұਓ͔Β࢝ΊΔϓϩμΫτSRE

Slide 14

Slide 14 text

Ұਓ໨ͷSREظ Ҿ༻: SRE NEXT 2022 Ұਓ͔Β࢝ΊΔϓϩμΫτSRE 1೥ؒͰ100ຊఔ౓ υΩϡϝϯτΛॻ͖ ଟ͘ͷਓʹಡΜͰ΋Β͑Δ Α͏ʹ͍ͯͨ͠

Slide 15

Slide 15 text

Ұਓ໨ͷSREظ Ҿ༻: SRE NEXT 2022 Ұਓ͔Β࢝ΊΔϓϩμΫτSRE

Slide 16

Slide 16 text

Ұਓ໨ͷSREظ Ҿ༻: SRE NEXT 2022 Ұਓ͔Β࢝ΊΔϓϩμΫτSRE

Slide 17

Slide 17 text

Ұਓ໨ͷSREظ ࠾༻޿ใͰར༻Ͱ͖Δʂ ࣗ෼ͨͪ͸Կऀͳͷ͔ ظ଴஋ΛࣔͤΔ ࠓޙ΋݁ہ ΊͪΌͪ͘Ό࢖͏͜ͱʹͳΔ Ҿ༻: SRE NEXT 2022 Ұਓ͔Β࢝ΊΔϓϩμΫτSRE ૣΊʹ࡞͓ͬͯ͘ͱྑ͍ ϛογϣϯ&Ϗδϣϯ

Slide 18

Slide 18 text

Ұਓ໨ͷSREظ Ҿ༻: SRE NEXT 2022 Ұਓ͔Β࢝ΊΔϓϩμΫτSRE

Slide 19

Slide 19 text

Ұਓ໨ͷSREظ औΓ૊ΈαϚϦ SREͱ͸Ͳ͏͍͏໾ׂͳͷ͔ɺԿΛ͢Δͷ͔෍ڭ׆ಈ ݱ৔͕ۤ͠ΜͰ͍Δ෦෼Λੵۃతʹर্͍͛ͯ·ͣ͸ վળ͢Δ ؂ࢹվળɺΠϯϑϥ໘ͷαϙʔτʹ୅ද͞ΕΔ ༏ઌ౓ͷ౰ͨΓΛ͚ͭͳ͕ΒϩʔυϚοϓ૊Έ্͛ ϛογϣϯɾϏδϣϯͷ࡞੒ ࠾༻׆ಈͷ։࢝

Slide 20

Slide 20 text

νʔϜ֦େظ ϩʔυϚοϓ ϩʔυϚοϓͱϓϩμΫτ ͔ΒͷґཔʹରԠ͢Δ όϥϯε͕͏·͘औΕͳ͍ Enabling SRE ઙ͘ͱ΋޿͍ܦݧ ࣮ߦྗ λεΫ؅ཧ ˑνʔϜϏϧσΟϯά ˑଞऀධՁ ମ੍ εΩϧηοτ

Slide 21

Slide 21 text

νʔϜ֦େظ ໢ཏ཰Λ্͛ΔͨΊʹͦΕͧΕ ଞͷϝϯόʔͱର৅͕ඃΒͳ͍Α͏ ͳମ੍Λ૊ΜͰ͍ͨ

Slide 22

Slide 22 text

νʔϜ֦େظ ͨͩ໢ཏతʹEnabling׆ಈ͸ਐḿ A͞Μ͕ٳΉͱ୲౰ϓϩμΫτͷ ਐḿ͕0ʹͳΔ શମͷঢ়ଶఈ্͛ͱͯ͠͸ औͬͯ΋Α͍τϨʔυΦϑͩͱࢥ͏

Slide 23

Slide 23 text

ͳͥνʔϜʹ͸ϛογϣϯ͕ඞཁ͔ʁͦͷॏཁੑ Ξϓϩʔνํ๏΍ղܾͷํ਑͸ɺϛογϣϯʹΑͬͯ มΘΔ ҙࢥܾఆͷ࣠ʹϛογϣϯͷଘࡏ͕໾ཱͭ νʔϜ֦େظ ՝୊Λ͢΂ͯSRE͕ר͖औΔ ͜ͱ΋Ͱ͖Δ SREҎ֎͕ӡ༻Մೳͳঢ়ଶΛ ੒Ռͱͯ͠࢓্͛Δ͜ͱ΋Ͱ͖Δ ࣗ෼ͨͪͷ࣮ݱ͍ͨ͠ੈք ʹΑͬͯ੒Ռ෺͕มΘΔ

Slide 24

Slide 24 text

౰࣌ͷϓϩμΫτ͝ͱͷ՝୊ A: େن໛σʔλϕʔεͷετϨʔδݶք఺ B: Amazon EKS΁ͷϦϓϨΠε͕͞Ε͍ͯͳ͍ C: ·ͩSRE׆ಈͰ͖͍ͯͳ͍νʔϜ͕͋Δ νʔϜ֦େظ ͦΕͧΕશ͘ҟͳΔ՝୊͕ଘࡏ ϝϯόʔͷօ͞Μ͸ඇৗʹ ؤுͬͯ͘Ε·ͨ͠

Slide 25

Slide 25 text

ҰํͰ՝୊΋

Slide 26

Slide 26 text

ͦΕͧΕͷݸੑ͕ࡍཱͪ࢝Ί͍͕ͯͨ… ϝϯόʔͷಘҙ෼໺ͰΞαΠϯ͢Δ͜ͱʹͳΔ ͦΕͧΕͷઐ໳ੑΛνʔϜͱͯ͠࠷େԽ͢Δ༨༟͸ͳ ͔ͬͨ ༨༟Λ࣋ͬͯऔΓ૊Ήʹ͸ɺ΍΍ਓ͕଍Γͳ͍ঢ়ଶ νʔϜ֦େظ

Slide 27

Slide 27 text

νʔϜ֦େظ औΓ૊ΈαϚϦ ҰͭͷϓϩμΫτ͚ͩͰͳ͘ɺෳ਺୲౰͢ΔΑ͏ʹ EmbeddedͰ͸ݶքͱͳΓEnablingମ੍ʹมߋ ϓϩμΫτݸผͷ՝୊ʹରԠ SRE׆ಈʹඞཁͳجૅ౔୆ߏஙʢSLO, จԽৢ੒ͳͲʣ ΋ܧଓ νʔϜϏϧσΟϯάʹؔ࿈͢ΔऔΓ૊Έ͕։࢝͞ΕΔ

Slide 28

Slide 28 text

νʔϜ҆ఆظ ϩʔυϚοϓ ϩʔυϚοϓ͕ܭը௨Γʹ ਐΈɺґཔλεΫ΋࣮֬ʹ ղܾͰ͖͍ͯΔঢ়ଶ Enabling SRE ߴ͍ઐ໳ੑ νʔϜྗΛ࠷େԽ ௕ظࢹ఺Ͱͷ໰୊ղܾ ଟ͘ͷؔ܎ऀͱௐ੔ ମ੍ εΩϧηοτ

Slide 29

Slide 29 text

νʔϜ҆ఆظ νʔϜͷ૯߹ྗͰରԠ ϓϩμΫτݸผͷ՝୊ʹ΋ରԠ Ͱ͖Δ

Slide 30

Slide 30 text

΍Δ΂͖͜ͱʹऔΓ૊Έͳ͕ΒɺνϟϨϯδ΋͢Δ ܦݧ͕ઙ͍ྖҬʹνϟϨϯδ͢Δ ܦݧ͕͋Δϝϯόʔ͕ϨϏϡʔ΍αϙʔτʹճΔ ϓϩμΫτνʔϜ͔ΒͷґཔʢHelp wantedλεΫʣΛ ࢖ͬͯ͋ΒΏΔྖҬͷλεΫΛܦݧ͢Δ ߴ౓ͳઐ໳ੑ͕ඞཁͳ΋ͷ͸ɺͦͷਓ͕Ϧʔυ͢Δ νʔϜ҆ఆظ

Slide 31

Slide 31 text

ΦϯϘʔσΟϯάϑϩʔͷ޲্ νʔϜ҆ఆظ ※͜ͷ͋ͱ֦େ͠·͢ άϩʔόϧԽͨ͠ͷΛ͖͔͚ͬʹ ໨Ͱݟ΍͍͢ܗʹ ԿΑΓɺ͜ͷํָ͕͠ΜͰ΋Β͑ ΔͷͰ͸ͱࢥͬͯ࡞੒

Slide 32

Slide 32 text

νʔϜ҆ఆظ

Slide 33

Slide 33 text

܁Γฦ͠Λճආ͢ΔͨΊͷπʔϧ܈Λ։ൃ DBΤϯδϯΞοϓάϨʔυݕূΛ࠷଎Ͱճͨ͢Ί CLI: PlatinumʢPercona Toolkit pt-upgradeΛwrapʣ Amazon RDS Blue/Green DeploymentsΛ҆શ͔ͭ࠷଎ Ͱ࣮ߦ͢ΔͨΊ CLI: Turquoiseʢ੨/྘͔Β༝དྷɻAWS CLIΛwrapʣ νʔϜ҆ఆظ ϝϯόʔશһ͕։ൃΛ ָ͠Μͩ

Slide 34

Slide 34 text

কདྷൃੜ͢Δ໰୊΍ɺ৽͍͠औΓ૊ΈͷτϥΠ ංେԽ͠ଓ͚Δେن໛σʔλϕʔεɻࠓޙͷରԠ KafkaʢAmazon MSKʣͷӡ༻؂ࢹ PostmortemͷϩʔϧϓϨΠݚम ଞ νʔϜ҆ఆظ ϝϯόʔ͕෼୲ͯ͠ ਐΊ͍ͯͨ

Slide 35

Slide 35 text

ҰํͰ՝୊΋

Slide 36

Slide 36 text

νʔϜ҆ఆظ SRE͸SREͷϩʔυϚοϓΛ ಺෦ͰਐΊΒΕΔ ϓϩμΫτνʔϜ͸ ૬ஊࣄ߲ΛSREʹ໰͍߹ΘͤΔ

Slide 37

Slide 37 text

νʔϜ҆ఆظ ϦϦʔε৘ใΛ͢΂ͯ೺ѲͰ͖ͳ ͘ͳͬͨ ϓϩμΫτͷݒ೦ࣄ߲Λහײʹ ΩϟονͰ͖ͳ͘ͳͬͨ ͩΜͩΜ ࡞ۀΛड͚෇͚͍ͯΔ͚ͩײ

Slide 38

Slide 38 text

ؔ܎ੑΛҡ࣋͢ΔͨΊʹ Production meetingΛఆظ։࠵ ϓϩμΫτͷ੹೚ऀ΍։ൃ੹೚ऀΛট଴ͨ͠૬ޓ৘ ใަ׵ͷ৔ SRE͔Β͸ҰఆظؒͷϝτϦΫεϨϙʔτ ϓϩμΫτ͔Β͸ϦϦʔε৘ใ΍ݒ೦ࣄ߲ڞ༗ νʔϜ҆ఆظ

Slide 39

Slide 39 text

ݴ༿Ͱ఻͑Δ͚ͩͰ͸จԽ͸࢒Βͳ͍ ಄Ͱ͸Θ͔ͬͯ͸͍͕ͨɺจԽ͸࢓૊Έ΋ؚΊͯ࡞Β ͳ͍ͱਓͷೖΕସΘΓͰফ͍͑ͯ͘Մೳੑ͕ߴ͍ Կ΋ߟ͑ͣʹґཔʹରԠ͍ͯ͠Δͱɺ࠷ॳʹةዧ͍ͯ͠ ͨ࡞ۀһײ͕͡ΜΘΓͱਫ਼ਆΛḝΈͩ͢ νʔϜ҆ఆظ

Slide 40

Slide 40 text

νʔϜ҆ఆظ औΓ૊ΈαϚϦ EOLͷΞοϓσʔτରԠɻશ෦Ͱ5ͭͷDBΤϯδϯΞο ϓσʔτݕূΛπʔϧ։ൃͰ୹ॖ ֤ϓϩμΫτͷSLOͷ࠶ఆٛ, PostmortemϩʔϧϓϨΠ ݚम, Production meetingͳͲͷSRE׆ಈ ԣஅతʹඞཁͳऔΓ૊ΈʢKafka, Envoy, Datadog logs ϑΥʔϚοτͷ౷Ұʣ ๯಄ͷϓϩδΣΫτ৆͸͜ͷ࿩

Slide 41

Slide 41 text

3೥ͰมΘͬͨ͜ͱɺมΘΒͳ͍͜ͱ ϩʔυϚοϓ ࠷ऴ౸ୡ஍఺͸ มΘΒͳ͍ɻม͑ͳ͍ SREͷମ੍͸มΘΔ ਓͷೖΕସΘΓ΋͋Δ νʔϜͷϑΣʔζ ૊৫ͷϑΣʔζ ϓϩμΫτͷϑΣʔζ ͳͲͰҟͳΔ ମ੍ εΩϧηοτ

Slide 42

Slide 42 text

࠷ऴ౸ୡ఺͸ಉ͡ɺମ੍͸ਐԽɺ׆༂εΩϧͷมԽ ૊৫ͷٸ੒௕ɾฤ੒มߋʹɺҰ౓͸ϛογϣϯΛม͑ Δ΂͖͔໎ͬͨɻ৺͔Β৴͡ΒΕΔ৴೦͸ม͑ͳ͍ ૊৫ͷܗʹ߹ΘͤɺSREͷରԠํ๏΋ਐԽ͢Ε͹͍͍ ϑΣʔζͰඞཁͳਓࡐ͕ҟͳΔͷ͸ඞવ 3೥ͰมΘͬͨ͜ͱɺมΘΒͳ͍͜ͱ

Slide 43

Slide 43 text

SREݸਓͱͯ͠ߟ͍͑ͯͨ͜ͱ

Slide 44

Slide 44 text

ࣗ෼͸ͲͷϑΣʔζʹ߹͏ͷͰ͠ΐ͏͔ ΋ͪΖΜͲͷϑΣʔζʹ΋దԠͰ͖ͨํ͕ྑ͍͕… Ͳ͏ߩݙͰ͖Δ͔Θ͔Βͣ໎ࢠʹͳΔ͜ͱ΋͋Δ ࠷΋ً͘ॠ͕ؒͲ͜ͳͷ͔͸ߟ͓͍͑ͯͯଛ͸ͳ͍ SREݸਓͱͯ͠ߟ͍͑ͯͨ͜ͱ

Slide 45

Slide 45 text

SREݸਓͱͯ͠ߟ͍͑ͯͨ͜ͱ ͜ΕΒ ͢΂ͯͷೳྗ͕͋Δਓ͸ك

Slide 46

Slide 46 text

SREݸਓͱͯ͠ߟ͍͑ͯͨ͜ͱ ঃʑʹɺͲ͏ߩݙ͢΂͖͔Θ͔Βͳ͘ͳͬͨ ߴ͍ٕज़ྗΛ࣋ͬͨਓͷ ෣୆͕੔ͬͨͱࢥͬͨ

Slide 47

Slide 47 text

SREݸਓͱͯ͠ߟ͍͑ͯͨ͜ͱ ൴Β͸૬ޓతʹྑ͍ӨڹΛ ༩͑ΒΕΔͷͰ͸ͳ͍͔

Slide 48

Slide 48 text

·ͱΊ

Slide 49

Slide 49 text

·ͱΊ ૊৫΍ϓϩμΫτͷϑΣʔζ͕มΘΕ͹SRE΋มΘΔ ର৅ʹ߹ΘͤͯॊೈʹมԽ͢Δ ͨͩ͠ϛογϣϯ͸ม͑ͳͯ͘΋ྑ͍ ࣗ෼ً͕͚ΔϑΣʔζ͕͋Δ ্ཱͪ͛ʁ֦େʁ҆ఆʁ΋ͬͱࡉ͔͍୯ҐͰ΋̋ ޓ͍ͷً͖͸૬৐ޮՌΛൃੜͤ͞Δ͜ͱ΋Ͱ͖Δ

Slide 50

Slide 50 text

No content