Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
On-call Engineering
Tatsuhiko Kubo
January 30, 2017
Technology
8
5.7k
On-call Engineering
Tatsuhiko Kubo
January 30, 2017
Tweet
Share
More Decks by Tatsuhiko Kubo
See All by Tatsuhiko Kubo
Handling a tremendous amount of images with Fastly / Yamagoya Traverse 2020
cubicdaiya
2
1k
System Integration with Fastly
cubicdaiya
0
410
実例で学ぶ画像最適化集 with ImageFlux / ImageFlux meetup#2
cubicdaiya
4
17k
Software Engineer, Infrastructure
cubicdaiya
4
2.7k
High Performance Count Up!
cubicdaiya
0
200
ImageFluxを利用した画像配信の最適化 / ImageFlux meetup 201801
cubicdaiya
0
2.3k
Building high performance push notification server in Go
cubicdaiya
5
2.7k
メルカリのデータ分析基盤 / mercari data analysis infrastructure
cubicdaiya
11
11k
Load balancer management with Consul
cubicdaiya
12
4.7k
Other Decks in Technology
See All in Technology
テスト自動化を最速で軌道に乗せるために
nozomiito
0
150
第22回 MLOps 勉強会:みてねのMLOps事情
tonouchi510
1
1.1k
聊聊 Cgo 的二三事
david74chou
0
350
ふりかえりの技術 / retrospectives
soudai
3
190
Step-by-Step MLOps and Microsoft Products
shisyu_gaku
3
640
Sysdig Secure/Falcoの活用術! ~Kubernetes基盤の脅威モデリングとランタイムセキュリティの強化~
owlinux1000
0
320
DevRel組織についての考察
taijihagino
PRO
0
160
AWS Step Functions を用いた非同期学習処理の例
hacarus
0
110
ロボットの実行すらメンドクサイ!?
kou12092
0
230
サイバー攻撃を想定したクラウドネイティブセキュリティガイドラインとCNAPP及びSecurity Observabilityの未来
sakon310
4
480
ここが好きだよAWS管理ポリシー_devio2022/i_am_iam_lover
yukihirochiba
0
3.3k
質の良い”カイゼン”の為の質の良い「振り返り」
shirayanagiryuji
0
140
Featured
See All Featured
Designing for humans not robots
tammielis
242
24k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
107
16k
What's in a price? How to price your products and services
michaelherold
229
9.4k
The MySQL Ecosystem @ GitHub 2015
samlambert
239
11k
A better future with KSS
kneath
226
16k
Designing with Data
zakiwarfel
91
4k
Producing Creativity
orderedlist
PRO
334
37k
Code Reviewing Like a Champion
maltzj
506
37k
Mobile First: as difficult as doing things right
swwweet
213
7.6k
Visualization
eitanlees
125
12k
Done Done
chrislema
174
14k
Bash Introduction
62gerente
598
210k
Transcript
Tatsuhiko Kubo@cubicdaiya SRE Tech Talks#2 2017/01/30 On-call Engineering
@cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.
None
SRE mission @ Mercari •Operation •γεςϜͷΛൃݟɾղܾ͠ɺɹɹɹɹɹ αʔϏεͷ৴པੑΛ্ͤ͞Δ •Software Engineering •αʔϏεΛεέʔϧͤ͞ΔͨΊͷπʔϧɺ
ϛυϧΣΞɺγεςϜج൫ͷ։ൃɾӡ༻
SREͷۀ༰ @ Mercari • ֤छAPIɺϛυϧΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ্ • On-call൪ • αʔόϓϩϏδϣχϯάͷ֤छࣗಈԽ
• ։ൃɺσϓϩΠɺϩάੳͷͨΊͷج൫උ • ηΩϡϦςΟͷ୲อ
Mercari Engineering BlogͰҰ෦ެ։த • PascalʙPuree + ngx_lua + Fluentd +
BigQueryͰͭ͘ΔϝϧΧϦͷϩάੳج൫ʙ • http://tech.mercari.com/entry/2015/09/09/163007 • DockerͱMakeΛར༻ͨ͠RPMύοέʔδͷϏϧυڥ • http://tech.mercari.com/entry/2016/08/15/163219 • ϋΠύϑΥʔϚϯεGaurunʙϝϧΧϦͷେنϓογϡ৴Λࢧ͑ΔϛυϧΣΞʙ • http://tech.mercari.com/entry/2016/11/08/170343 • ࢹ͚ͩ͡Όͳ͍ʂσϓϩΠʹMackerelΛ͏ • http://tech.mercari.com/entry/2016/11/14/120000 refs: http://tech.mercari.com/
SREͷۀ༰ @ Mercari • ֤छAPIɺϛυϧΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ্ • On-call൪ • αʔόϓϩϏδϣχϯάͷ֤छࣗಈԽ
• ։ൃɺσϓϩΠɺϩάੳͷͨΊͷج൫උ • ηΩϡϦςΟͷ୲อ
SREͷۀ༰ @ Mercari • ֤छAPIɺϛυϧΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ্ • On-call൪ • αʔόϓϩϏδϣχϯάͷ֤छࣗಈԽ
• ։ൃɺσϓϩΠɺϩάੳͷͨΊͷج൫උ • ηΩϡϦςΟͷ୲อ
On-call(Φϯίʔϧ)
On-call • ۓٸ࣌(ྫɿγεςϜো)ʹඋ͑ͯిͰ ࿈བྷ͕औΕΔঢ়ଶΛอͭ͜ͱ
WebαʔϏεͱOn-call • WebαʔϏε24/7ͰՔಇ • ۓٸ࣌ʹඋ͑ͯ࠷1~2ਓػ͓ͯ͘͠ ඞཁ͕͋Δ
On-call൪ @ Mercari • SREνʔϜͰϩʔςʔγϣϯʢ̍िؒຖʹަʣ • ༵0:00͔Β༵23:59 • ΞϥʔτͷҰ࣍ड͚औΓͱۓٸରԠ •
ٳͰ15-20ҎʹରԠͰ͖Δ͜ͱ͕·͍͠ • ฏதଞSRE͕ग़ࣾ͢Δ·Ͱࣗػ(SREશһ͕ిं௨ۈதͰ͋Δ ͜ͱΛආ͚Δ) • ൪தߦಈʹ੍ݶ͕͔͔Δ
େܕ࿈ٳதมଇϩʔς
On-call൪ @ Mercari • On-call൪৭ʑͱෛ୲͕͔͔Δ • ਫ਼ਆతͳϓϨογϟʔɺಉډਓͷෛ୲ • ਂʹΞϥʔτɺւͷ͜͏͔Βి •
൪ʹԠͨ͡खৼٳΛࢧڅ
On-callʹؔΘΔॾͷྫ • ϩʔςʔγϣϯ൪ͷۈଵɺৼସٳՋͷཧʁ • खಈͰΔͷΊΜͲ͍ • On-call൪ͷిͬͯԿ൪͚ͩͬʁ • ൪ʹ࿈བྷ͢Δଆͷෛ୲Λܰݮ͢Δඞཁ͕͋Δ •
On-call൪͕ిʹग़ͳ͔ͬͨΒʁ • ΤεΧϨʔγϣϯ͕ඞཁ Software EngineeringͰղܾ͠Α͏ʂ
On-call Engineering
bot hello
bot hello • ۈଵଧࠁ༻ͷίϚϯυ in Slack • On-call൪SREͷࣗػελʔτͷ߹ਤ • ଞSRE͕ग़ࣾͨ͠Β൪SREग़ࣾ͢Δ
• bot bye͋Δ
bot touban call
bot touban call • On-call൪ʹిΛ͔͚ΔίϚϯυ in Slack • Powered by
PagerDuty • allΛ͢ͱSREશһʹి • Slack͕μϯͯ͠Δ߹ʹඋ͑ͯผ్ۓٸ࿈ བྷઌͷϦετΛwikiʹ·ͱΊͯ͋Δ
bot touban help
On-call൪ϩʔςͷ ཧͱΤεΧϨʔγϣϯ • PagerDutyͷػೳΛ׆༻ • On Call Schedules • Escalation
Policy • https://www.pagerduty.com/
PagerDuty API & Google Apps Script • On-call൪ͷ༧ఆ࣮ΛGoogleͷεϓ ϨουγʔτΧϨϯμʔʹಉظ •
ఆظతʹूܭͯ͠څ༩ٳՋʹө
·ͱΊ • ۓٸ࣌ʹඋ͑ͯOn-callମ੍Λ͑Α͏ • On-callۀෛ୲͕େ͖͍ • ੍͘͠ΈͰΧόʔ • On-callSoftware Engineeringʂ
We are hiring! https://www.mercari.com/jp/jobs/