Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ポストモーテム運用を支える文化と技術 / Culture and Technology Sup...

ポストモーテム運用を支える文化と技術 / Culture and Technology Supporting Postmortem Operations

Takeshi Kondo

February 09, 2023
Tweet

More Decks by Takeshi Kondo

Other Decks in Technology

Transcript

  1. Who am I chaspy chaspy_ Engineering Manager Site Reliability and

    Web Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me
  2. ϙετϞʔςϜΛࢧ͑ΔจԽ • ୭͔1ਓͷ͍ͤʹͳΒͳ͍Α͏ʹ͢Δ • Design Doc • Production Readiness Checklist

    • ૉૣ͘ɺΈΜͳͰରԠ͢Δ • ো֐ରԠϑϩʔ • ো֐͔ΒֶͿ • ϙετϞʔςϜڞ༗ձ • ϙετϞʔςϜಡॻձ ඪ४Խ͢Δ ໨తҙࣝͷৢ੒
  3. Design Doc / Production Readiness Checklist • ʮ͏͔ͬΓʯΛඪ४Խ͢Δ • ෳ਺ਓͰϨϏϡʔ͢Δ͜ͱͰʮݸਓͷ͍ͤʯʹͮ͠Β͘͢Δ

    • ϨϏϡʔͳ͠୯ಠΦϖϨʔγϣϯͰϛεΔͱͲ͏ͯ͠΋ݪҼ͕ݸਓʹ ޲͍ͯ͠·͏Ͱ͠ΐ͏ 
 ʮProduction Readiness ελσΟαϓϦʯͰݕࡧʂ
  4. ϙετϞʔςϜΛࢧ͑ΔจԽ·ͱΊ • ϋʔυϧΛԼ͛Δࡉ͔ͳ࢓૊Έ • Issue Template, Slack custom response •

    ඪ४Խ • Production Readiness Checklist, ো֐ରԠϑϩʔɺϨϕϧఆٛ • ʮֶͼͷͨΊʯͱ͍͏໨తҙࣝͷৢ੒ • ࠷ॳ͸ݴ͍ଓ͚Δɾॻ͖ଓ͚Δ͔͠ͳ͍ؾ͕͠·͢ • աڈ Slack ݕࡧͯ͠ΈΔͱো֐ʹରͯ͠ʮॻ͍ͯ΋Β͑·͔͢ʁʯͱΑ͓͘ئ͍͍ͯͨ͠ • ॻ͍ͨ݅਺΋ chaspy ͕Ұ൪ଟͦ͏… • ϒϩάΛॻ͘ͷ΋ޮՌ͋ͬͨͱࢥ͍·͢
  5. ෛՙςετ Production Readiness Checklist Ͱ Performance Risk Λಛఆͯ͠΋Β͍ɺ ඞཁͰ͋Ε͹ Loadtest

    ΛҊ಺ Load Test ࣮ࢪ಺༰ͷ Template Requirements Λهࡌͯ͠ SRE ͱ։ൃ νʔϜͰ໨ઢΛ߹ΘͤΔ
  6. Canary Release • Argo Rollouts Λ׆༻ • Rails Upgrade ͳͲɺػೳมߋ͸ͳ͍͕ɺϦεΫͷߴ͍มߋʹ࢖͏

    φΠεTryͰ͢ΑͶ 1% ͔ΒϦϦʔε͠ɺΤϥʔ͕ग़ͨΒ͙͢ ໭͢͜ͱͰඃ֐Λ࠷খݶʹͰ͖·ͨ͠
  7. Thank you! chaspy chaspy_ Engineering Manager Site Reliability and Web

    Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me