On-call Engineering

On-call Engineering

5d74d743eabd2bf7d4d2f68b9d3c727d?s=128

Tatsuhiko Kubo

January 30, 2017
Tweet

Transcript

  1. Tatsuhiko Kubo@cubicdaiya SRE Tech Talks#2 2017/01/30 On-call Engineering

  2. @cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.

  3. None
  4. SRE mission @ Mercari •Operation •γεςϜͷ໰୊఺Λൃݟɾղܾ͠ɺɹɹɹɹɹ αʔϏεͷ৴པੑΛ޲্ͤ͞Δ •Software Engineering •αʔϏεΛεέʔϧͤ͞ΔͨΊͷπʔϧɺ

    ϛυϧ΢ΣΞɺγεςϜج൫ͷ։ൃɾӡ༻
  5. SREͷۀ຿಺༰ @ Mercari • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ޲্ • On-call౰൪ • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ

    • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ • ηΩϡϦςΟͷ୲อ
  6. Mercari Engineering BlogͰ΋Ұ෦ެ։த • PascalʙPuree + ngx_lua + Fluentd +

    BigQueryͰͭ͘ΔϝϧΧϦͷϩά෼ੳج൫ʙ • http://tech.mercari.com/entry/2015/09/09/163007 • DockerͱMakeΛར༻ͨ͠RPMύοέʔδͷϏϧυ؀ڥ • http://tech.mercari.com/entry/2016/08/15/163219 • ϋΠύϑΥʔϚϯεGaurunʙϝϧΧϦͷେن໛ϓογϡ഑৴Λࢧ͑Δϛυϧ΢ΣΞʙ • http://tech.mercari.com/entry/2016/11/08/170343 • ؂ࢹ͚ͩ͡Όͳ͍ʂσϓϩΠʹMackerelΛ࢖͏࿩ • http://tech.mercari.com/entry/2016/11/14/120000 refs: http://tech.mercari.com/
  7. SREͷۀ຿಺༰ @ Mercari • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ޲্ • On-call౰൪ • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ

    • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ • ηΩϡϦςΟͷ୲อ
  8. SREͷۀ຿಺༰ @ Mercari • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ޲্ • On-call౰൪ • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ

    • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ • ηΩϡϦςΟͷ୲อ
  9. On-call(Φϯίʔϧ)

  10. On-call • ۓٸ࣌(ྫɿγεςϜো֐)ʹඋ͑ͯి࿩౳Ͱ ࿈བྷ͕औΕΔঢ়ଶΛอͭ͜ͱ

  11. WebαʔϏεͱOn-call • WebαʔϏε͸24/7ͰՔಇ • ۓٸ࣌ʹඋ͑ͯ࠷௿1~2ਓ͸଴ػ͓ͯ͘͠ ඞཁ͕͋Δ

  12. On-call౰൪ @ Mercari • SREνʔϜͰϩʔςʔγϣϯʢ̍िؒຖʹަ୅ʣ • ೔༵೔0:00͔Β౔༵೔23:59 • ΞϥʔτͷҰ࣍ड͚औΓͱۓٸରԠ •

    ٳ೔Ͱ΋15-20෼Ҏ಺ʹରԠͰ͖Δ͜ͱ͕๬·͍͠ • ฏ೔த͸ଞSRE͕ग़ࣾ͢Δ·Ͱࣗ୐଴ػ(SREશһ͕ిं௨ۈதͰ͋Δ ͜ͱΛආ͚Δ) • ౰൪த͸ߦಈʹ੍ݶ͕͔͔Δ
  13. େܕ࿈ٳத͸มଇϩʔς

  14. On-call౰൪ @ Mercari • On-call౰൪͸৭ʑͱෛ୲͕͔͔Δ • ਫ਼ਆతͳϓϨογϟʔɺಉډਓ΁ͷෛ୲ • ਂ໷ʹΞϥʔτɺւͷ޲͜͏͔Βి࿩ •

    ౰൪೔਺ʹԠͨ͡ख౰΍ৼٳΛࢧڅ
  15. On-callʹؔΘΔॾ໰୊ͷྫ • ϩʔςʔγϣϯ΍౰൪ͷۈଵɺৼସٳՋͷ؅ཧ͸ʁ • खಈͰ΍ΔͷΊΜͲ͍ • On-call౰൪ͷి࿩ͬͯԿ൪͚ͩͬʁ • ౰൪ʹ࿈བྷ͢Δଆͷෛ୲Λܰݮ͢Δඞཁ͕͋Δ •

    On-call౰൪͕ి࿩ʹग़ͳ͔ͬͨΒʁ • ΤεΧϨʔγϣϯ͕ඞཁ Software EngineeringͰղܾ͠Α͏ʂ
  16. On-call Engineering

  17. bot hello

  18. bot hello • ۈଵଧࠁ༻ͷίϚϯυ in Slack • On-call౰൪SREͷࣗ୐଴ػελʔτͷ߹ਤ • ଞSRE͕ग़ࣾͨ͠Β౰൪SRE΋ग़ࣾ͢Δ

    • bot bye΋͋Δ
  19. bot touban call

  20. bot touban call • On-call౰൪ʹి࿩Λ͔͚ΔίϚϯυ in Slack • Powered by

    PagerDuty • allΛ଍͢ͱSREશһʹి࿩ • Slack͕μ΢ϯͯ͠Δ৔߹ʹඋ͑ͯผ్ۓٸ࿈ བྷઌͷϦετΛwikiʹ·ͱΊͯ͋Δ
  21. bot touban help

  22. On-call౰൪ϩʔςͷ ؅ཧͱΤεΧϨʔγϣϯ • PagerDutyͷػೳΛ׆༻ • On Call Schedules • Escalation

    Policy • https://www.pagerduty.com/
  23. PagerDuty API & Google Apps Script • On-call౰൪ͷ༧ఆ΍࣮੷ΛGoogleͷεϓ Ϩουγʔτ΍ΧϨϯμʔʹಉظ •

    ఆظతʹूܭͯ͠څ༩΍ٳՋ೔਺ʹ൓ө
  24. ·ͱΊ • ۓٸ࣌ʹඋ͑ͯOn-callମ੍Λ੔͑Α͏ • On-callۀ຿͸ෛ୲͕େ͖͍ • ੍౓΍͘͠ΈͰΧόʔ • On-call΋Software Engineeringʂ

  25. We are hiring! https://www.mercari.com/jp/jobs/