On-call Engineering

Transcript

1. Tatsuhiko Kubo@cubicdaiya
   SRE Tech Talks#2 2017/01/30
   On-call Engineering
   

   View Slide

2. @cubicdaiya / Tatsuhiko Kubo
   Principal Engineer, SRE @ Mercari, Inc.
   

   View Slide

3. View Slide

4. SRE mission @ Mercari
   •Operation
   •γεςϜͷ໰୊఺Λൃݟɾղܾ͠ɺɹɹɹɹɹ
   αʔϏεͷ৴པੑΛ޲্ͤ͞Δ
   •Software Engineering
   •αʔϏεΛεέʔϧͤ͞ΔͨΊͷπʔϧɺ
   ϛυϧ΢ΣΞɺγεςϜج൫ͷ։ൃɾӡ༻
   

   View Slide

5. SREͷۀ຿಺༰ @ Mercari
   • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ
   ύϑΥʔϚϯεͷҡ࣋ɾ޲্
   • On-call౰൪
   • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ
   • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ
   • ηΩϡϦςΟͷ୲อ
   

   View Slide

6. Mercari Engineering BlogͰ΋Ұ෦ެ։த
   • PascalʙPuree + ngx_lua + Fluentd + BigQueryͰͭ͘ΔϝϧΧϦͷϩά෼ੳج൫ʙ
   • http://tech.mercari.com/entry/2015/09/09/163007
   • DockerͱMakeΛར༻ͨ͠RPMύοέʔδͷϏϧυ؀ڥ
   • http://tech.mercari.com/entry/2016/08/15/163219
   • ϋΠύϑΥʔϚϯεGaurunʙϝϧΧϦͷେن໛ϓογϡ഑৴Λࢧ͑Δϛυϧ΢ΣΞʙ
   • http://tech.mercari.com/entry/2016/11/08/170343
   • ؂ࢹ͚ͩ͡Όͳ͍ʂσϓϩΠʹMackerelΛ࢖͏࿩
   • http://tech.mercari.com/entry/2016/11/14/120000
   refs: http://tech.mercari.com/
   

   View Slide

7. SREͷۀ຿಺༰ @ Mercari
   • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ
   ύϑΥʔϚϯεͷҡ࣋ɾ޲্
   • On-call౰൪
   • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ
   • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ
   • ηΩϡϦςΟͷ୲อ
   

   View Slide

8. SREͷۀ຿಺༰ @ Mercari
   • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ
   ύϑΥʔϚϯεͷҡ࣋ɾ޲্
   • On-call౰൪
   • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ
   • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ
   • ηΩϡϦςΟͷ୲อ
   

   View Slide

9. On-call(Φϯίʔϧ)
   

   View Slide

10. On-call
    • ۓٸ࣌(ྫɿγεςϜো֐)ʹඋ͑ͯి࿩౳Ͱ
    ࿈བྷ͕औΕΔঢ়ଶΛอͭ͜ͱ
    

    View Slide

11. WebαʔϏεͱOn-call
    • WebαʔϏε͸24/7ͰՔಇ
    • ۓٸ࣌ʹඋ͑ͯ࠷௿1~2ਓ͸଴ػ͓ͯ͘͠
    ඞཁ͕͋Δ
    

    View Slide

12. On-call౰൪ @ Mercari
    • SREνʔϜͰϩʔςʔγϣϯʢ̍िؒຖʹަ୅ʣ
    • ೔༵೔0:00͔Β౔༵೔23:59
    • ΞϥʔτͷҰ࣍ड͚औΓͱۓٸରԠ
    • ٳ೔Ͱ΋15-20෼Ҏ಺ʹରԠͰ͖Δ͜ͱ͕๬·͍͠
    • ฏ೔த͸ଞSRE͕ग़ࣾ͢Δ·Ͱࣗ୐଴ػ(SREશһ͕ిं௨ۈதͰ͋Δ
    ͜ͱΛආ͚Δ)
    • ౰൪த͸ߦಈʹ੍ݶ͕͔͔Δ
    

    View Slide

13. େܕ࿈ٳத͸มଇϩʔς
    

    View Slide

14. On-call౰൪ @ Mercari
    • On-call౰൪͸৭ʑͱෛ୲͕͔͔Δ
    • ਫ਼ਆతͳϓϨογϟʔɺಉډਓ΁ͷෛ୲
    • ਂ໷ʹΞϥʔτɺւͷ޲͜͏͔Βి࿩
    • ౰൪೔਺ʹԠͨ͡ख౰΍ৼٳΛࢧڅ
    

    View Slide

15. On-callʹؔΘΔॾ໰୊ͷྫ
    • ϩʔςʔγϣϯ΍౰൪ͷۈଵɺৼସٳՋͷ؅ཧ͸ʁ
    • खಈͰ΍ΔͷΊΜͲ͍
    • On-call౰൪ͷి࿩ͬͯԿ൪͚ͩͬʁ
    • ౰൪ʹ࿈བྷ͢Δଆͷෛ୲Λܰݮ͢Δඞཁ͕͋Δ
    • On-call౰൪͕ి࿩ʹग़ͳ͔ͬͨΒʁ
    • ΤεΧϨʔγϣϯ͕ඞཁ
    Software EngineeringͰղܾ͠Α͏ʂ
    

    View Slide

16. On-call Engineering
    

    View Slide

17. bot hello
    

    View Slide

18. bot hello
    • ۈଵଧࠁ༻ͷίϚϯυ in Slack
    • On-call౰൪SREͷࣗ୐଴ػελʔτͷ߹ਤ
    • ଞSRE͕ग़ࣾͨ͠Β౰൪SRE΋ग़ࣾ͢Δ
    • bot bye΋͋Δ
    

    View Slide

19. bot touban call
    

    View Slide

20. bot touban call
    • On-call౰൪ʹి࿩Λ͔͚ΔίϚϯυ in Slack
    • Powered by PagerDuty
    • allΛ଍͢ͱSREશһʹి࿩
    • Slack͕μ΢ϯͯ͠Δ৔߹ʹඋ͑ͯผ్ۓٸ࿈
    བྷઌͷϦετΛwikiʹ·ͱΊͯ͋Δ
    

    View Slide

21. bot touban help
    

    View Slide

22. On-call౰൪ϩʔςͷ
    ؅ཧͱΤεΧϨʔγϣϯ
    • PagerDutyͷػೳΛ׆༻
    • On Call Schedules
    • Escalation Policy
    • https://www.pagerduty.com/
    

    View Slide

23. PagerDuty API &
    Google Apps Script
    • On-call౰൪ͷ༧ఆ΍࣮੷ΛGoogleͷεϓ
    Ϩουγʔτ΍ΧϨϯμʔʹಉظ
    • ఆظతʹूܭͯ͠څ༩΍ٳՋ೔਺ʹ൓ө
    

    View Slide

24. ·ͱΊ
    • ۓٸ࣌ʹඋ͑ͯOn-callମ੍Λ੔͑Α͏
    • On-callۀ຿͸ෛ୲͕େ͖͍
    • ੍౓΍͘͠ΈͰΧόʔ
    • On-call΋Software Engineeringʂ
    

    View Slide

25. We are hiring!
    https://www.mercari.com/jp/jobs/
    

    View Slide