Upgrade to Pro — share decks privately, control downloads, hide ads and more …

On-call Engineering

On-call Engineering

Tatsuhiko Kubo

January 30, 2017
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Tatsuhiko Kubo@cubicdaiya
    SRE Tech Talks#2 2017/01/30
    On-call Engineering

    View full-size slide

  2. @cubicdaiya / Tatsuhiko Kubo
    Principal Engineer, SRE @ Mercari, Inc.

    View full-size slide

  3. SRE mission @ Mercari
    •Operation
    •γεςϜͷ໰୊఺Λൃݟɾղܾ͠ɺɹɹɹɹɹ
    αʔϏεͷ৴པੑΛ޲্ͤ͞Δ
    •Software Engineering
    •αʔϏεΛεέʔϧͤ͞ΔͨΊͷπʔϧɺ
    ϛυϧ΢ΣΞɺγεςϜج൫ͷ։ൃɾӡ༻

    View full-size slide

  4. SREͷۀ຿಺༰ @ Mercari
    • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ
    ύϑΥʔϚϯεͷҡ࣋ɾ޲্
    • On-call౰൪
    • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ
    • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ
    • ηΩϡϦςΟͷ୲อ

    View full-size slide

  5. Mercari Engineering BlogͰ΋Ұ෦ެ։த
    • PascalʙPuree + ngx_lua + Fluentd + BigQueryͰͭ͘ΔϝϧΧϦͷϩά෼ੳج൫ʙ
    • http://tech.mercari.com/entry/2015/09/09/163007
    • DockerͱMakeΛར༻ͨ͠RPMύοέʔδͷϏϧυ؀ڥ
    • http://tech.mercari.com/entry/2016/08/15/163219
    • ϋΠύϑΥʔϚϯεGaurunʙϝϧΧϦͷେن໛ϓογϡ഑৴Λࢧ͑Δϛυϧ΢ΣΞʙ
    • http://tech.mercari.com/entry/2016/11/08/170343
    • ؂ࢹ͚ͩ͡Όͳ͍ʂσϓϩΠʹMackerelΛ࢖͏࿩
    • http://tech.mercari.com/entry/2016/11/14/120000
    refs: http://tech.mercari.com/

    View full-size slide

  6. SREͷۀ຿಺༰ @ Mercari
    • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ
    ύϑΥʔϚϯεͷҡ࣋ɾ޲্
    • On-call౰൪
    • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ
    • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ
    • ηΩϡϦςΟͷ୲อ

    View full-size slide

  7. SREͷۀ຿಺༰ @ Mercari
    • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ
    ύϑΥʔϚϯεͷҡ࣋ɾ޲্
    • On-call౰൪
    • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ
    • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ
    • ηΩϡϦςΟͷ୲อ

    View full-size slide

  8. On-call(Φϯίʔϧ)

    View full-size slide

  9. On-call
    • ۓٸ࣌(ྫɿγεςϜো֐)ʹඋ͑ͯి࿩౳Ͱ
    ࿈བྷ͕औΕΔঢ়ଶΛอͭ͜ͱ

    View full-size slide

  10. WebαʔϏεͱOn-call
    • WebαʔϏε͸24/7ͰՔಇ
    • ۓٸ࣌ʹඋ͑ͯ࠷௿1~2ਓ͸଴ػ͓ͯ͘͠
    ඞཁ͕͋Δ

    View full-size slide

  11. On-call౰൪ @ Mercari
    • SREνʔϜͰϩʔςʔγϣϯʢ̍िؒຖʹަ୅ʣ
    • ೔༵೔0:00͔Β౔༵೔23:59
    • ΞϥʔτͷҰ࣍ड͚औΓͱۓٸରԠ
    • ٳ೔Ͱ΋15-20෼Ҏ಺ʹରԠͰ͖Δ͜ͱ͕๬·͍͠
    • ฏ೔த͸ଞSRE͕ग़ࣾ͢Δ·Ͱࣗ୐଴ػ(SREશһ͕ిं௨ۈதͰ͋Δ
    ͜ͱΛආ͚Δ)
    • ౰൪த͸ߦಈʹ੍ݶ͕͔͔Δ

    View full-size slide

  12. େܕ࿈ٳத͸มଇϩʔς

    View full-size slide

  13. On-call౰൪ @ Mercari
    • On-call౰൪͸৭ʑͱෛ୲͕͔͔Δ
    • ਫ਼ਆతͳϓϨογϟʔɺಉډਓ΁ͷෛ୲
    • ਂ໷ʹΞϥʔτɺւͷ޲͜͏͔Βి࿩
    • ౰൪೔਺ʹԠͨ͡ख౰΍ৼٳΛࢧڅ

    View full-size slide

  14. On-callʹؔΘΔॾ໰୊ͷྫ
    • ϩʔςʔγϣϯ΍౰൪ͷۈଵɺৼସٳՋͷ؅ཧ͸ʁ
    • खಈͰ΍ΔͷΊΜͲ͍
    • On-call౰൪ͷి࿩ͬͯԿ൪͚ͩͬʁ
    • ౰൪ʹ࿈བྷ͢Δଆͷෛ୲Λܰݮ͢Δඞཁ͕͋Δ
    • On-call౰൪͕ి࿩ʹग़ͳ͔ͬͨΒʁ
    • ΤεΧϨʔγϣϯ͕ඞཁ
    Software EngineeringͰղܾ͠Α͏ʂ

    View full-size slide

  15. On-call Engineering

    View full-size slide

  16. bot hello
    • ۈଵଧࠁ༻ͷίϚϯυ in Slack
    • On-call౰൪SREͷࣗ୐଴ػελʔτͷ߹ਤ
    • ଞSRE͕ग़ࣾͨ͠Β౰൪SRE΋ग़ࣾ͢Δ
    • bot bye΋͋Δ

    View full-size slide

  17. bot touban call

    View full-size slide

  18. bot touban call
    • On-call౰൪ʹి࿩Λ͔͚ΔίϚϯυ in Slack
    • Powered by PagerDuty
    • allΛ଍͢ͱSREશһʹి࿩
    • Slack͕μ΢ϯͯ͠Δ৔߹ʹඋ͑ͯผ్ۓٸ࿈
    བྷઌͷϦετΛwikiʹ·ͱΊͯ͋Δ

    View full-size slide

  19. bot touban help

    View full-size slide

  20. On-call౰൪ϩʔςͷ
    ؅ཧͱΤεΧϨʔγϣϯ
    • PagerDutyͷػೳΛ׆༻
    • On Call Schedules
    • Escalation Policy
    • https://www.pagerduty.com/

    View full-size slide

  21. PagerDuty API &
    Google Apps Script
    • On-call౰൪ͷ༧ఆ΍࣮੷ΛGoogleͷεϓ
    Ϩουγʔτ΍ΧϨϯμʔʹಉظ
    • ఆظతʹूܭͯ͠څ༩΍ٳՋ೔਺ʹ൓ө

    View full-size slide

  22. ·ͱΊ
    • ۓٸ࣌ʹඋ͑ͯOn-callମ੍Λ੔͑Α͏
    • On-callۀ຿͸ෛ୲͕େ͖͍
    • ੍౓΍͘͠ΈͰΧόʔ
    • On-call΋Software Engineeringʂ

    View full-size slide

  23. We are hiring!
    https://www.mercari.com/jp/jobs/

    View full-size slide