ポストモーテム運用を支える文化と技術 / Culture and Technology Supporting Postmortem Operations

https://findy.connpass.com/event/273197/

Transcript

1. ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽͱٕज़ Takeshi Kondo / @chaspy 2023/02/07 ΠϯγσϯτʹͲ͏ରԠ͖͔ͯͨ͠ʁΈΜͳͰֶͿϙετϞʔςϜ Lunch LT
2. Who am I chaspy chaspy_ Engineering Manager Site Reliability and Web Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me
3. લఏɿϓϩμΫτ঺հ - ελσΟαϓϦ
4. ࠓ೔࿩͢͜ͱ ʮϙετϞʔςϜӡ༻ʯͷલఏͱͳΔจԽͱٕज़
5. ࠓ೔࿩͞ͳ͍͜ͱ ʮϙετϞʔςϜӡ༻ʯͦΕࣗମͷ޻෉
6. Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ·ͱΊ
7. Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ·ͱΊ
8. ϙετϞʔςϜӡ༻ͷݱঢ় • ো֐ൃੜޙʮϙετϞʔςϜॻ͖·͠ΐ͏ʯͷ੠ • ؔ܎ऀͰू·ͬͯڞ༗ • ΞΫγϣϯ͸֤νʔϜͷΠγϡʔͱͯ͠ੵ·ΕΔ
9. ΧδϡΞϧʹϙετϞʔςϜ͕ߦΘΕΔ༷ࢠ ܰඍͳ΋ͷͰ΋ʮֶͼͷνϟϯεʯͱଊ͑Δ ໨త͕ਁಁ͍ͯ͠Δ །Ұͷ޻෉ͱͯ͠ Slack ΧελϜ ϨεϙϯεͰ issue template ͕ग़ ͯ͘Δͷ͸ॻͨ͘ΊͷϋʔυϧΛ Լ͍͛ͯΔ…?
10. ੲॻ͍ͨهࣄ͕ࠓͰ΋Ҿ༻͞Ε͍ͯΔ ࠓճ Findy ͞Μʹ੠͔͚ͯ΋Βͬ ͨͷ΋͜ͷهࣄΛݟͯ΋Β͔ͬͨ ΒͰͨ͠🙏 2019೥… ʮো֐ରԠͱϙετϞʔςϜ ελσΟαϓϦʯͰݕࡧʂ
11. Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ·ͱΊ
12. ϙετϞʔςϜӡ༻ͷྺ࢙ • Issue Template ͷ First Commit ͸2019೥5݄ • ͦΕ͔ΒςϯϓϨʔτͷߋ৽͸΄ͱΜͲͳ͍
13. ϙετϞʔςϜӡ༻ͷྺ࢙ • SRE ຊ͔ΒςϯϓϨʔτྲྀ༻ • Issue Template ͷ First Commit ͸2019೥5݄
14. ϙετϞʔςϜӡ༻ͷྺ࢙ • TTD/TTR Λ௥ه
15. Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ·ͱΊ
16. ϙετϞʔςϜΛࢧ͑ΔจԽ • ୭͔1ਓͷ͍ͤʹͳΒͳ͍Α͏ʹ͢Δ • Design Doc • Production Readiness Checklist • ૉૣ͘ɺΈΜͳͰରԠ͢Δ • ো֐ରԠϑϩʔ • ো֐͔ΒֶͿ • ϙετϞʔςϜڞ༗ձ • ϙετϞʔςϜಡॻձ ඪ४Խ͢Δ ໨తҙࣝͷৢ੒
17. Design Doc / Production Readiness Checklist • ʮ͏͔ͬΓʯΛඪ४Խ͢Δ • ෳ਺ਓͰϨϏϡʔ͢Δ͜ͱͰʮݸਓͷ͍ͤʯʹͮ͠Β͘͢Δ • ϨϏϡʔͳ͠୯ಠΦϖϨʔγϣϯͰϛεΔͱͲ͏ͯ͠΋ݪҼ͕ݸਓʹ ޲͍ͯ͠·͏Ͱ͠ΐ͏ 
 ʮProduction Readiness ελσΟαϓϦʯͰݕࡧʂ
18. ো֐ରԠϑϩʔ • ো֐ରԠϑϩʔɾো֐Ϩϕϧ͕ఆٛ͞Ε͍ͯΔ • Slack work fl ow ͰใࠂͰ͖Δ • ো֐͔΋ʁͰ΋ใࠂ͢Δ͜ͱΛਪ঑͍ͯ͠Δ
19. ো֐ରԠϑϩʔ ઌ೔ͷ CircleCI ͷ݅ͷใࠂྫ ੹೚ऀʹࣗಈͰϝϯγϣϯ͕ඈͿ
20. ϙετϞʔςϜಡॻձ • SRE νʔϜͰ͸ΦϯϘʔσΟϯάͰϙετϞʔςϜಡॻձΛ ࣮ࢪ • શ෦͸ಡΊͳ͍ʢ૿͑ΔʣͷͰʮ͓͢͢ΊʯϙετϞʔςϜ ΛϥϕϧͰ؅ཧ • ֶͼ͕ଟ͍΋ͷ • ݱࡏͷߏ੒ཧղʹͭͳ͕Δ΋ͷ • ো֐ൃੜ࣌ͷಈ͖ͱͯ͠ࢀߟʹͳΔ΋ͷ
21. ͓͢͢ΊϙετϞʔςϜ8બ
22. ϙετϞʔςϜΛࢧ͑ΔจԽ·ͱΊ • ϋʔυϧΛԼ͛Δࡉ͔ͳ࢓૊Έ • Issue Template, Slack custom response • ඪ४Խ • Production Readiness Checklist, ো֐ରԠϑϩʔɺϨϕϧఆٛ • ʮֶͼͷͨΊʯͱ͍͏໨తҙࣝͷৢ੒ • ࠷ॳ͸ݴ͍ଓ͚Δɾॻ͖ଓ͚Δ͔͠ͳ͍ؾ͕͠·͢ • աڈ Slack ݕࡧͯ͠ΈΔͱো֐ʹରͯ͠ʮॻ͍ͯ΋Β͑·͔͢ʁʯͱΑ͓͘ئ͍͍ͯͨ͠ • ॻ͍ͨ݅਺΋ chaspy ͕Ұ൪ଟͦ͏… • ϒϩάΛॻ͘ͷ΋ޮՌ͋ͬͨͱࢥ͍·͢
23. ϙετϞʔςϜΛࢧ͑Δٕज़ • ॏཁͳো֐͸ࣄલʹ๷͛ΔΑ͏ʹͳ͍ͬͯ·͔͢ʁ • ద੾ʹϦεΫΛऔΔ͜ͱ͕Ͱ͖͍ͯ·͔͢ʁ • ʮ೦ͷҝ֬ೝʯ͕؆୯ʹͰ͖ΔΑ͏ʹͳ͍ͬͯ·͔͢ʁ
24. ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ෛՙςετ • Canary Release • E2E Test Automation • σʔλϕʔεϦετΞ
25. ෛՙςετ Production Readiness Checklist Ͱ Performance Risk Λಛఆͯ͠΋Β͍ɺ ඞཁͰ͋Ε͹ Loadtest ΛҊ಺ Load Test ࣮ࢪ಺༰ͷ Template Requirements Λهࡌͯ͠ SRE ͱ։ൃ νʔϜͰ໨ઢΛ߹ΘͤΔ
26. ෛՙςετ • Gatling ͷίʔυΛॻ͍ͯςετ͕࣮ࢪͰ͖Δ؀ڥ • ςετ݁Ռ͕ PR ʹషΒΕΔ • ෛՙςετ͕ߴ଎ʹࢼߦࡨޡͰ͖Δ Ϩϙʔτੜ੒
27. ෛՙςετ • ؀ڥ४උ͸؆୯ͱ͸ݴΘͳ͍͕ɺϋʔυϧ͸Լ͕͍ͬͯΔ • Databaseʢຊ൪͔ΒϦετΞ͢Δɻޙड़ʣ • Application (Pull Request Λ࡞Ε͹Ͱ͖Δʣ • EKS Node Group • Test code
28. Canary Release • Argo Rollouts Λ׆༻ • Rails Upgrade ͳͲɺػೳมߋ͸ͳ͍͕ɺϦεΫͷߴ͍มߋʹ࢖͏ φΠεTryͰ͢ΑͶ 1% ͔ΒϦϦʔε͠ɺΤϥʔ͕ग़ͨΒ͙͢ ໭͢͜ͱͰඃ֐Λ࠷খݶʹͰ͖·ͨ͠
29. E2E Test Automation • ϒϩάΛݟ͍ͯͩ͘͞ʂ • ݕࡧʮελσΟαϓϦ E2Eʯ • ݕग़͢Δෆ۩߹΋ͦΕͳΓʹ͋Γɺຊ൪ো֐Λ๷͍Ͱ͍Δ
30. σʔλϕʔεϦετΞ • ͪ͜Β΋ৄࡉ͸ϒϩάΛ͝ཡ͍ͩ͘͞ʂ • ݕࡧʮελσΟαϓϦ σʔλϕʔεϦετΞʯ
31. ·ͱΊ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽͱٕज़Λ঺հ͠·ͨ͠ • ϓϩηεɾจԽ໘͸ඪ४Խͱ໨తҙࣝͷৢ੒͕ॏཁ • ٕज़໘͸ൃੜޙͷ࠶ൃ๷ࢭͷੵΈॏͶ • จԽͱٕज़ɺ྆ํ͕૬ޓʹ࿈ܞ͢Δ • ੵΈॏͶΔ͜ͱͰʮಉ͡ো֐ʯ͸ى͖ͮΒ͘ͳΔ • ʮ৽͍͠ো֐ʯ͸ֶͼͷνϟϯεʹͳΔ
32. ࠓ೔࿩͞ͳ͔ͬͨ͜ͱʢεϐʔΧʔτʔΫͰ࿩ͤͨΒخ͍͠ʣ • ো֐ͷධՁɺϨϕϧ෇͚ • MTTR / MTTD ͷܭଌ • ࣄޙͷλεΫΛ͍͔ʹ։ൃΛ͠ͳ͕Β࣮ࢪ͢Δ͔ • ো֐ͱ SLI/SLO
33. Thank you! chaspy chaspy_ Engineering Manager Site Reliability and Web Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me