On-call Engineering - Speaker Deck

On-call Engineering

by Tatsuhiko Kubo

Slide 1

Slide 1 text

Tatsuhiko Kubo@cubicdaiya SRE Tech Talks#2 2017/01/30 On-call Engineering

Slide 2

Slide 2 text

@cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

SRE mission @ Mercari •Operation •γεςϜͷ໰୊఺Λൃݟɾղܾ͠ɺɹɹɹɹɹ αʔϏεͷ৴པੑΛ޲্ͤ͞Δ •Software Engineering •αʔϏεΛεέʔϧͤ͞ΔͨΊͷπʔϧɺ ϛυϧ΢ΣΞɺγεςϜج൫ͷ։ൃɾӡ༻

Slide 5

Slide 5 text

SREͷۀ຿಺༰ @ Mercari • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ޲্ • On-call౰൪ • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ • ηΩϡϦςΟͷ୲อ

Slide 6

Slide 6 text

Mercari Engineering BlogͰ΋Ұ෦ެ։த • PascalʙPuree + ngx_lua + Fluentd + BigQueryͰͭ͘ΔϝϧΧϦͷϩά෼ੳج൫ʙ • http://tech.mercari.com/entry/2015/09/09/163007 • DockerͱMakeΛར༻ͨ͠RPMύοέʔδͷϏϧυ؀ڥ • http://tech.mercari.com/entry/2016/08/15/163219 • ϋΠύϑΥʔϚϯεGaurunʙϝϧΧϦͷେن໛ϓογϡ഑৴Λࢧ͑Δϛυϧ΢ΣΞʙ • http://tech.mercari.com/entry/2016/11/08/170343 • ؂ࢹ͚ͩ͡Όͳ͍ʂσϓϩΠʹMackerelΛ࢖͏࿩ • http://tech.mercari.com/entry/2016/11/14/120000 refs: http://tech.mercari.com/

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

On-call(Φϯίʔϧ)

Slide 10

Slide 10 text

On-call • ۓٸ࣌(ྫɿγεςϜো֐)ʹඋ͑ͯి࿩౳Ͱ ࿈བྷ͕औΕΔঢ়ଶΛอͭ͜ͱ

Slide 11

Slide 11 text

WebαʔϏεͱOn-call • WebαʔϏε͸24/7ͰՔಇ • ۓٸ࣌ʹඋ͑ͯ࠷௿1~2ਓ͸଴ػ͓ͯ͘͠ ඞཁ͕͋Δ

Slide 12

Slide 12 text

On-call౰൪ @ Mercari • SREνʔϜͰϩʔςʔγϣϯʢ̍िؒຖʹަ୅ʣ • ೔༵೔0:00͔Β౔༵೔23:59 • ΞϥʔτͷҰ࣍ड͚औΓͱۓٸରԠ • ٳ೔Ͱ΋15-20෼Ҏ಺ʹରԠͰ͖Δ͜ͱ͕๬·͍͠ • ฏ೔த͸ଞSRE͕ग़ࣾ͢Δ·Ͱࣗ୐଴ػ(SREશһ͕ిं௨ۈதͰ͋Δ ͜ͱΛආ͚Δ) • ౰൪த͸ߦಈʹ੍ݶ͕͔͔Δ

Slide 13

Slide 13 text

େܕ࿈ٳத͸มଇϩʔς

Slide 14

Slide 14 text

On-call౰൪ @ Mercari • On-call౰൪͸৭ʑͱෛ୲͕͔͔Δ • ਫ਼ਆతͳϓϨογϟʔɺಉډਓ΁ͷෛ୲ • ਂ໷ʹΞϥʔτɺւͷ޲͜͏͔Βి࿩ • ౰൪೔਺ʹԠͨ͡ख౰΍ৼٳΛࢧڅ

Slide 15

Slide 15 text

On-callʹؔΘΔॾ໰୊ͷྫ • ϩʔςʔγϣϯ΍౰൪ͷۈଵɺৼସٳՋͷ؅ཧ͸ʁ • खಈͰ΍ΔͷΊΜͲ͍ • On-call౰൪ͷి࿩ͬͯԿ൪͚ͩͬʁ • ౰൪ʹ࿈བྷ͢Δଆͷෛ୲Λܰݮ͢Δඞཁ͕͋Δ • On-call౰൪͕ి࿩ʹग़ͳ͔ͬͨΒʁ • ΤεΧϨʔγϣϯ͕ඞཁ Software EngineeringͰղܾ͠Α͏ʂ

Slide 16

Slide 16 text

On-call Engineering

Slide 17

Slide 17 text

bot hello

Slide 18

Slide 18 text

bot hello • ۈଵଧࠁ༻ͷίϚϯυ in Slack • On-call౰൪SREͷࣗ୐଴ػελʔτͷ߹ਤ • ଞSRE͕ग़ࣾͨ͠Β౰൪SRE΋ग़ࣾ͢Δ • bot bye΋͋Δ

Slide 19

Slide 19 text

bot touban call

Slide 20

Slide 20 text

bot touban call • On-call౰൪ʹి࿩Λ͔͚ΔίϚϯυ in Slack • Powered by PagerDuty • allΛ଍͢ͱSREશһʹి࿩ • Slack͕μ΢ϯͯ͠Δ৔߹ʹඋ͑ͯผ్ۓٸ࿈ བྷઌͷϦετΛwikiʹ·ͱΊͯ͋Δ

Slide 21

Slide 21 text

bot touban help

Slide 22

Slide 22 text

On-call౰൪ϩʔςͷ ؅ཧͱΤεΧϨʔγϣϯ • PagerDutyͷػೳΛ׆༻ • On Call Schedules • Escalation Policy • https://www.pagerduty.com/

Slide 23

Slide 23 text

PagerDuty API & Google Apps Script • On-call౰൪ͷ༧ఆ΍࣮੷ΛGoogleͷεϓ Ϩουγʔτ΍ΧϨϯμʔʹಉظ • ఆظతʹूܭͯ͠څ༩΍ٳՋ೔਺ʹ൓ө

Slide 24

Slide 24 text

·ͱΊ • ۓٸ࣌ʹඋ͑ͯOn-callମ੍Λ੔͑Α͏ • On-callۀ຿͸ෛ୲͕େ͖͍ • ੍౓΍͘͠ΈͰΧόʔ • On-call΋Software Engineeringʂ

Slide 25

Slide 25 text

We are hiring! https://www.mercari.com/jp/jobs/