On-call Engineering

Transcript

1. Tatsuhiko Kubo@cubicdaiya SRE Tech Talks#2 2017/01/30 On-call Engineering

2. @cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.

3. None

4. SRE mission @ Mercari •Operation •γεςϜͷ໰୊఺Λൃݟɾղܾ͠ɺɹɹɹɹɹ αʔϏεͷ৴པੑΛ޲্ͤ͞Δ •Software Engineering •αʔϏεΛεέʔϧͤ͞ΔͨΊͷπʔϧɺ

   ϛυϧ΢ΣΞɺγεςϜج൫ͷ։ൃɾӡ༻

5. SREͷۀ຿಺༰ @ Mercari • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ޲্ • On-call౰൪ • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ

   • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ • ηΩϡϦςΟͷ୲อ

6. Mercari Engineering BlogͰ΋Ұ෦ެ։த • PascalʙPuree + ngx_lua + Fluentd +

   BigQueryͰͭ͘ΔϝϧΧϦͷϩά෼ੳج൫ʙ • http://tech.mercari.com/entry/2015/09/09/163007 • DockerͱMakeΛར༻ͨ͠RPMύοέʔδͷϏϧυ؀ڥ • http://tech.mercari.com/entry/2016/08/15/163219 • ϋΠύϑΥʔϚϯεGaurunʙϝϧΧϦͷେن໛ϓογϡ഑৴Λࢧ͑Δϛυϧ΢ΣΞʙ • http://tech.mercari.com/entry/2016/11/08/170343 • ؂ࢹ͚ͩ͡Όͳ͍ʂσϓϩΠʹMackerelΛ࢖͏࿩ • http://tech.mercari.com/entry/2016/11/14/120000 refs: http://tech.mercari.com/

7. SREͷۀ຿಺༰ @ Mercari • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ޲্ • On-call౰൪ • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ

   • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ • ηΩϡϦςΟͷ୲อ

8. SREͷۀ຿಺༰ @ Mercari • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ޲্ • On-call౰൪ • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ

   • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ • ηΩϡϦςΟͷ୲อ

9. On-call(Φϯίʔϧ)

10. On-call • ۓٸ࣌(ྫɿγεςϜো֐)ʹඋ͑ͯి࿩౳Ͱ ࿈བྷ͕औΕΔঢ়ଶΛอͭ͜ͱ

11. WebαʔϏεͱOn-call • WebαʔϏε͸24/7ͰՔಇ • ۓٸ࣌ʹඋ͑ͯ࠷௿1~2ਓ͸଴ػ͓ͯ͘͠ ඞཁ͕͋Δ

12. On-call౰൪ @ Mercari • SREνʔϜͰϩʔςʔγϣϯʢ̍िؒຖʹަ୅ʣ • ೔༵೔0:00͔Β౔༵೔23:59 • ΞϥʔτͷҰ࣍ड͚औΓͱۓٸରԠ •

    ٳ೔Ͱ΋15-20෼Ҏ಺ʹରԠͰ͖Δ͜ͱ͕๬·͍͠ • ฏ೔த͸ଞSRE͕ग़ࣾ͢Δ·Ͱࣗ୐଴ػ(SREશһ͕ిं௨ۈதͰ͋Δ ͜ͱΛආ͚Δ) • ౰൪த͸ߦಈʹ੍ݶ͕͔͔Δ

13. େܕ࿈ٳத͸มଇϩʔς

14. On-call౰൪ @ Mercari • On-call౰൪͸৭ʑͱෛ୲͕͔͔Δ • ਫ਼ਆతͳϓϨογϟʔɺಉډਓ΁ͷෛ୲ • ਂ໷ʹΞϥʔτɺւͷ޲͜͏͔Βి࿩ •

    ౰൪೔਺ʹԠͨ͡ख౰΍ৼٳΛࢧڅ

15. On-callʹؔΘΔॾ໰୊ͷྫ • ϩʔςʔγϣϯ΍౰൪ͷۈଵɺৼସٳՋͷ؅ཧ͸ʁ • खಈͰ΍ΔͷΊΜͲ͍ • On-call౰൪ͷి࿩ͬͯԿ൪͚ͩͬʁ • ౰൪ʹ࿈བྷ͢Δଆͷෛ୲Λܰݮ͢Δඞཁ͕͋Δ •

    On-call౰൪͕ి࿩ʹग़ͳ͔ͬͨΒʁ • ΤεΧϨʔγϣϯ͕ඞཁ Software EngineeringͰղܾ͠Α͏ʂ

16. On-call Engineering

17. bot hello

18. bot hello • ۈଵଧࠁ༻ͷίϚϯυ in Slack • On-call౰൪SREͷࣗ୐଴ػελʔτͷ߹ਤ • ଞSRE͕ग़ࣾͨ͠Β౰൪SRE΋ग़ࣾ͢Δ

    • bot bye΋͋Δ

19. bot touban call

20. bot touban call • On-call౰൪ʹి࿩Λ͔͚ΔίϚϯυ in Slack • Powered by

    PagerDuty • allΛ଍͢ͱSREશһʹి࿩ • Slack͕μ΢ϯͯ͠Δ৔߹ʹඋ͑ͯผ్ۓٸ࿈ བྷઌͷϦετΛwikiʹ·ͱΊͯ͋Δ

21. bot touban help

22. On-call౰൪ϩʔςͷ ؅ཧͱΤεΧϨʔγϣϯ • PagerDutyͷػೳΛ׆༻ • On Call Schedules • Escalation

    Policy • https://www.pagerduty.com/

23. PagerDuty API & Google Apps Script • On-call౰൪ͷ༧ఆ΍࣮੷ΛGoogleͷεϓ Ϩουγʔτ΍ΧϨϯμʔʹಉظ •

    ఆظతʹूܭͯ͠څ༩΍ٳՋ೔਺ʹ൓ө

24. ·ͱΊ • ۓٸ࣌ʹඋ͑ͯOn-callମ੍Λ੔͑Α͏ • On-callۀ຿͸ෛ୲͕େ͖͍ • ੍౓΍͘͠ΈͰΧόʔ • On-call΋Software Engineeringʂ

25. We are hiring! https://www.mercari.com/jp/jobs/