ポストモーテム運用を支える文化と技術 / Culture and Technology Supporting Postmortem Operations

by Takeshi Kondo

Slide 1

Slide 1 text

ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽͱٕज़ Takeshi Kondo / @chaspy 2023/02/07 ΠϯγσϯτʹͲ͏ରԠ͖͔ͯͨ͠ʁΈΜͳͰֶͿϙετϞʔςϜ Lunch LT

Slide 2

Slide 2 text

Who am I chaspy chaspy_ Engineering Manager Site Reliability and Web Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me

Slide 3

Slide 3 text

લఏɿϓϩμΫτ঺հ - ελσΟαϓϦ

Slide 4

Slide 4 text

ࠓ೔࿩͢͜ͱ ʮϙετϞʔςϜӡ༻ʯͷલఏͱͳΔจԽͱٕज़

Slide 5

Slide 5 text

ࠓ೔࿩͞ͳ͍͜ͱ ʮϙετϞʔςϜӡ༻ʯͦΕࣗମͷ޻෉

Slide 6

Slide 6 text

Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ·ͱΊ

Slide 7

Slide 7 text

Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ·ͱΊ

Slide 8

Slide 8 text

ϙετϞʔςϜӡ༻ͷݱঢ় • ো֐ൃੜޙʮϙετϞʔςϜॻ͖·͠ΐ͏ʯͷ੠ • ؔ܎ऀͰू·ͬͯڞ༗ • ΞΫγϣϯ͸֤νʔϜͷΠγϡʔͱͯ͠ੵ·ΕΔ

Slide 9

Slide 9 text

ΧδϡΞϧʹϙετϞʔςϜ͕ߦΘΕΔ༷ࢠ ܰඍͳ΋ͷͰ΋ʮֶͼͷνϟϯεʯͱଊ͑Δ ໨త͕ਁಁ͍ͯ͠Δ །Ұͷ޻෉ͱͯ͠ Slack ΧελϜ ϨεϙϯεͰ issue template ͕ग़ ͯ͘Δͷ͸ॻͨ͘ΊͷϋʔυϧΛ Լ͍͛ͯΔ…?

Slide 10

Slide 10 text

ੲॻ͍ͨهࣄ͕ࠓͰ΋Ҿ༻͞Ε͍ͯΔ ࠓճ Findy ͞Μʹ੠͔͚ͯ΋Βͬ ͨͷ΋͜ͷهࣄΛݟͯ΋Β͔ͬͨ ΒͰͨ͠🙏 2019೥… ʮো֐ରԠͱϙετϞʔςϜ ελσΟαϓϦʯͰݕࡧʂ

Slide 11

Slide 11 text

Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ·ͱΊ

Slide 12

Slide 12 text

ϙετϞʔςϜӡ༻ͷྺ࢙ • Issue Template ͷ First Commit ͸2019೥5݄ • ͦΕ͔ΒςϯϓϨʔτͷߋ৽͸΄ͱΜͲͳ͍

Slide 13

Slide 13 text

ϙετϞʔςϜӡ༻ͷྺ࢙ • SRE ຊ͔ΒςϯϓϨʔτྲྀ༻ • Issue Template ͷ First Commit ͸2019೥5݄

Slide 14

Slide 14 text

ϙετϞʔςϜӡ༻ͷྺ࢙ • TTD/TTR Λ௥ه

Slide 15

Slide 15 text

Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ·ͱΊ

Slide 16

Slide 16 text

ϙετϞʔςϜΛࢧ͑ΔจԽ • ୭͔1ਓͷ͍ͤʹͳΒͳ͍Α͏ʹ͢Δ • Design Doc • Production Readiness Checklist • ૉૣ͘ɺΈΜͳͰରԠ͢Δ • ো֐ରԠϑϩʔ • ো֐͔ΒֶͿ • ϙετϞʔςϜڞ༗ձ • ϙετϞʔςϜಡॻձ ඪ४Խ͢Δ ໨తҙࣝͷৢ੒

Slide 17

Slide 17 text

Design Doc / Production Readiness Checklist • ʮ͏͔ͬΓʯΛඪ४Խ͢Δ • ෳ਺ਓͰϨϏϡʔ͢Δ͜ͱͰʮݸਓͷ͍ͤʯʹͮ͠Β͘͢Δ • ϨϏϡʔͳ͠୯ಠΦϖϨʔγϣϯͰϛεΔͱͲ͏ͯ͠΋ݪҼ͕ݸਓʹ ޲͍ͯ͠·͏Ͱ͠ΐ͏   ʮProduction Readiness ελσΟαϓϦʯͰݕࡧʂ

Slide 18

Slide 18 text

ো֐ରԠϑϩʔ • ো֐ରԠϑϩʔɾো֐Ϩϕϧ͕ఆٛ͞Ε͍ͯΔ • Slack work fl ow ͰใࠂͰ͖Δ • ো֐͔΋ʁͰ΋ใࠂ͢Δ͜ͱΛਪ঑͍ͯ͠Δ

Slide 19

Slide 19 text

ো֐ରԠϑϩʔ ઌ೔ͷ CircleCI ͷ݅ͷใࠂྫ ੹೚ऀʹࣗಈͰϝϯγϣϯ͕ඈͿ

Slide 20

Slide 20 text

ϙετϞʔςϜಡॻձ • SRE νʔϜͰ͸ΦϯϘʔσΟϯάͰϙετϞʔςϜಡॻձΛ ࣮ࢪ • શ෦͸ಡΊͳ͍ʢ૿͑ΔʣͷͰʮ͓͢͢ΊʯϙετϞʔςϜ ΛϥϕϧͰ؅ཧ • ֶͼ͕ଟ͍΋ͷ • ݱࡏͷߏ੒ཧղʹͭͳ͕Δ΋ͷ • ো֐ൃੜ࣌ͷಈ͖ͱͯ͠ࢀߟʹͳΔ΋ͷ

Slide 21

Slide 21 text

͓͢͢ΊϙετϞʔςϜ8બ

Slide 22

Slide 22 text

ϙετϞʔςϜΛࢧ͑ΔจԽ·ͱΊ • ϋʔυϧΛԼ͛Δࡉ͔ͳ࢓૊Έ • Issue Template, Slack custom response • ඪ४Խ • Production Readiness Checklist, ো֐ରԠϑϩʔɺϨϕϧఆٛ • ʮֶͼͷͨΊʯͱ͍͏໨తҙࣝͷৢ੒ • ࠷ॳ͸ݴ͍ଓ͚Δɾॻ͖ଓ͚Δ͔͠ͳ͍ؾ͕͠·͢ • աڈ Slack ݕࡧͯ͠ΈΔͱো֐ʹରͯ͠ʮॻ͍ͯ΋Β͑·͔͢ʁʯͱΑ͓͘ئ͍͍ͯͨ͠ • ॻ͍ͨ݅਺΋ chaspy ͕Ұ൪ଟͦ͏… • ϒϩάΛॻ͘ͷ΋ޮՌ͋ͬͨͱࢥ͍·͢

Slide 23

Slide 23 text

ϙετϞʔςϜΛࢧ͑Δٕज़ • ॏཁͳো֐͸ࣄલʹ๷͛ΔΑ͏ʹͳ͍ͬͯ·͔͢ʁ • ద੾ʹϦεΫΛऔΔ͜ͱ͕Ͱ͖͍ͯ·͔͢ʁ • ʮ೦ͷҝ֬ೝʯ͕؆୯ʹͰ͖ΔΑ͏ʹͳ͍ͬͯ·͔͢ʁ

Slide 24

Slide 24 text

ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ෛՙςετ • Canary Release • E2E Test Automation • σʔλϕʔεϦετΞ

Slide 25

Slide 25 text

ෛՙςετ Production Readiness Checklist Ͱ Performance Risk Λಛఆͯ͠΋Β͍ɺ ඞཁͰ͋Ε͹ Loadtest ΛҊ಺ Load Test ࣮ࢪ಺༰ͷ Template Requirements Λهࡌͯ͠ SRE ͱ։ൃ νʔϜͰ໨ઢΛ߹ΘͤΔ

Slide 26

Slide 26 text

ෛՙςετ • Gatling ͷίʔυΛॻ͍ͯςετ͕࣮ࢪͰ͖Δ؀ڥ • ςετ݁Ռ͕ PR ʹషΒΕΔ • ෛՙςετ͕ߴ଎ʹࢼߦࡨޡͰ͖Δ Ϩϙʔτੜ੒

Slide 27

Slide 27 text

ෛՙςετ • ؀ڥ४උ͸؆୯ͱ͸ݴΘͳ͍͕ɺϋʔυϧ͸Լ͕͍ͬͯΔ • Databaseʢຊ൪͔ΒϦετΞ͢Δɻޙड़ʣ • Application (Pull Request Λ࡞Ε͹Ͱ͖Δʣ • EKS Node Group • Test code

Slide 28

Slide 28 text

Canary Release • Argo Rollouts Λ׆༻ • Rails Upgrade ͳͲɺػೳมߋ͸ͳ͍͕ɺϦεΫͷߴ͍มߋʹ࢖͏ φΠεTryͰ͢ΑͶ 1% ͔ΒϦϦʔε͠ɺΤϥʔ͕ग़ͨΒ͙͢ ໭͢͜ͱͰඃ֐Λ࠷খݶʹͰ͖·ͨ͠

Slide 29

Slide 29 text

E2E Test Automation • ϒϩάΛݟ͍ͯͩ͘͞ʂ • ݕࡧʮελσΟαϓϦ E2Eʯ • ݕग़͢Δෆ۩߹΋ͦΕͳΓʹ͋Γɺຊ൪ো֐Λ๷͍Ͱ͍Δ

Slide 30

Slide 30 text

σʔλϕʔεϦετΞ • ͪ͜Β΋ৄࡉ͸ϒϩάΛ͝ཡ͍ͩ͘͞ʂ • ݕࡧʮελσΟαϓϦ σʔλϕʔεϦετΞʯ

Slide 31

Slide 31 text

·ͱΊ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽͱٕज़Λ঺հ͠·ͨ͠ • ϓϩηεɾจԽ໘͸ඪ४Խͱ໨తҙࣝͷৢ੒͕ॏཁ • ٕज़໘͸ൃੜޙͷ࠶ൃ๷ࢭͷੵΈॏͶ • จԽͱٕज़ɺ྆ํ͕૬ޓʹ࿈ܞ͢Δ • ੵΈॏͶΔ͜ͱͰʮಉ͡ো֐ʯ͸ى͖ͮΒ͘ͳΔ • ʮ৽͍͠ো֐ʯ͸ֶͼͷνϟϯεʹͳΔ

Slide 32

Slide 32 text

ࠓ೔࿩͞ͳ͔ͬͨ͜ͱʢεϐʔΧʔτʔΫͰ࿩ͤͨΒخ͍͠ʣ • ো֐ͷධՁɺϨϕϧ෇͚ • MTTR / MTTD ͷܭଌ • ࣄޙͷλεΫΛ͍͔ʹ։ൃΛ͠ͳ͕Β࣮ࢪ͢Δ͔ • ো֐ͱ SLI/SLO

Slide 33

Slide 33 text

Thank you! chaspy chaspy_ Engineering Manager Site Reliability and Web Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me