Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
On-call Engineering
Tatsuhiko Kubo
January 30, 2017
Technology
8
5.9k
On-call Engineering
Tatsuhiko Kubo
January 30, 2017
Tweet
Share
More Decks by Tatsuhiko Kubo
See All by Tatsuhiko Kubo
Handling a tremendous amount of images with Fastly / Yamagoya Traverse 2020
cubicdaiya
2
1.1k
System Integration with Fastly
cubicdaiya
0
440
実例で学ぶ画像最適化集 with ImageFlux / ImageFlux meetup#2
cubicdaiya
4
17k
Software Engineer, Infrastructure
cubicdaiya
4
2.8k
High Performance Count Up!
cubicdaiya
0
210
ImageFluxを利用した画像配信の最適化 / ImageFlux meetup 201801
cubicdaiya
0
2.4k
Building high performance push notification server in Go
cubicdaiya
5
2.8k
メルカリのデータ分析基盤 / mercari data analysis infrastructure
cubicdaiya
11
11k
Load balancer management with Consul
cubicdaiya
12
4.8k
Other Decks in Technology
See All in Technology
Google Cloud Workflows: API automation, patterns and best practices
glaforge
0
100
Pentesting Password Reset Functionality
anugrahsr
0
490
Bill One 開発エンジニア 紹介資料
sansantech
PRO
0
120
Oracle Transaction Manager for Microservices Free 22.3 製品概要
oracle4engineer
PRO
5
110
オンプレk8sとEKSの並行運用の実際
ch1aki
0
290
スクラム導入して変わったチーム、組織のありかた
yumechi
0
190
開発者と協働できるメトリクスダッシュボードを作ろう!/SRE Lounge 2023
lmi
3
500
re:Invent2022 前後の Amazon EventBridge のアップデートを踏まえつつ、情シスの仕事をより楽しくしたい話。 / EventBridge for Information Systems Department
_kensh
2
750
AWS re:Invent 2022で発表された新機能を試してみた ~Cloud OperationとSecurity~ / New Cloud Operation and Security Features Announced at AWS reInvent 2022
yuj1osm
1
210
Raspberry Pi Camera 3 介紹
piepie_tw
PRO
0
150
API連携に伴う規制と対応 / Regulations and responses to API linkage
moneyforward
0
160
OpenShiftのリリースノートを整理してみた
loftkun
2
400
Featured
See All Featured
10 Git Anti Patterns You Should be Aware of
lemiorhan
643
54k
Product Roadmaps are Hard
iamctodd
38
7.7k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
500
130k
Facilitating Awesome Meetings
lara
33
4.6k
Build The Right Thing And Hit Your Dates
maggiecrowley
22
1.4k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
2
400
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1.1M
WebSockets: Embracing the real-time Web
robhawkes
58
6k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
13
1.1k
A Modern Web Designer's Workflow
chriscoyier
689
180k
A Tale of Four Properties
chriscoyier
149
21k
What the flash - Photography Introduction
edds
64
10k
Transcript
Tatsuhiko
[email protected]
SRE Tech Talks#2 2017/01/30 On-call Engineering
@cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.
None
SRE mission @ Mercari •Operation •γεςϜͷΛൃݟɾղܾ͠ɺɹɹɹɹɹ αʔϏεͷ৴པੑΛ্ͤ͞Δ •Software Engineering •αʔϏεΛεέʔϧͤ͞ΔͨΊͷπʔϧɺ
ϛυϧΣΞɺγεςϜج൫ͷ։ൃɾӡ༻
SREͷۀ༰ @ Mercari • ֤छAPIɺϛυϧΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ্ • On-call൪ • αʔόϓϩϏδϣχϯάͷ֤छࣗಈԽ
• ։ൃɺσϓϩΠɺϩάੳͷͨΊͷج൫උ • ηΩϡϦςΟͷ୲อ
Mercari Engineering BlogͰҰ෦ެ։த • PascalʙPuree + ngx_lua + Fluentd +
BigQueryͰͭ͘ΔϝϧΧϦͷϩάੳج൫ʙ • http://tech.mercari.com/entry/2015/09/09/163007 • DockerͱMakeΛར༻ͨ͠RPMύοέʔδͷϏϧυڥ • http://tech.mercari.com/entry/2016/08/15/163219 • ϋΠύϑΥʔϚϯεGaurunʙϝϧΧϦͷେنϓογϡ৴Λࢧ͑ΔϛυϧΣΞʙ • http://tech.mercari.com/entry/2016/11/08/170343 • ࢹ͚ͩ͡Όͳ͍ʂσϓϩΠʹMackerelΛ͏ • http://tech.mercari.com/entry/2016/11/14/120000 refs: http://tech.mercari.com/
SREͷۀ༰ @ Mercari • ֤छAPIɺϛυϧΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ্ • On-call൪ • αʔόϓϩϏδϣχϯάͷ֤छࣗಈԽ
• ։ൃɺσϓϩΠɺϩάੳͷͨΊͷج൫උ • ηΩϡϦςΟͷ୲อ
SREͷۀ༰ @ Mercari • ֤छAPIɺϛυϧΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ্ • On-call൪ • αʔόϓϩϏδϣχϯάͷ֤छࣗಈԽ
• ։ൃɺσϓϩΠɺϩάੳͷͨΊͷج൫උ • ηΩϡϦςΟͷ୲อ
On-call(Φϯίʔϧ)
On-call • ۓٸ࣌(ྫɿγεςϜো)ʹඋ͑ͯిͰ ࿈བྷ͕औΕΔঢ়ଶΛอͭ͜ͱ
WebαʔϏεͱOn-call • WebαʔϏε24/7ͰՔಇ • ۓٸ࣌ʹඋ͑ͯ࠷1~2ਓػ͓ͯ͘͠ ඞཁ͕͋Δ
On-call൪ @ Mercari • SREνʔϜͰϩʔςʔγϣϯʢ̍िؒຖʹަʣ • ༵0:00͔Β༵23:59 • ΞϥʔτͷҰ࣍ड͚औΓͱۓٸରԠ •
ٳͰ15-20ҎʹରԠͰ͖Δ͜ͱ͕·͍͠ • ฏதଞSRE͕ग़ࣾ͢Δ·Ͱࣗػ(SREશһ͕ిं௨ۈதͰ͋Δ ͜ͱΛආ͚Δ) • ൪தߦಈʹ੍ݶ͕͔͔Δ
େܕ࿈ٳதมଇϩʔς
On-call൪ @ Mercari • On-call൪৭ʑͱෛ୲͕͔͔Δ • ਫ਼ਆతͳϓϨογϟʔɺಉډਓͷෛ୲ • ਂʹΞϥʔτɺւͷ͜͏͔Βి •
൪ʹԠͨ͡खৼٳΛࢧڅ
On-callʹؔΘΔॾͷྫ • ϩʔςʔγϣϯ൪ͷۈଵɺৼସٳՋͷཧʁ • खಈͰΔͷΊΜͲ͍ • On-call൪ͷిͬͯԿ൪͚ͩͬʁ • ൪ʹ࿈བྷ͢Δଆͷෛ୲Λܰݮ͢Δඞཁ͕͋Δ •
On-call൪͕ిʹग़ͳ͔ͬͨΒʁ • ΤεΧϨʔγϣϯ͕ඞཁ Software EngineeringͰղܾ͠Α͏ʂ
On-call Engineering
bot hello
bot hello • ۈଵଧࠁ༻ͷίϚϯυ in Slack • On-call൪SREͷࣗػελʔτͷ߹ਤ • ଞSRE͕ग़ࣾͨ͠Β൪SREग़ࣾ͢Δ
• bot bye͋Δ
bot touban call
bot touban call • On-call൪ʹిΛ͔͚ΔίϚϯυ in Slack • Powered by
PagerDuty • allΛ͢ͱSREશһʹి • Slack͕μϯͯ͠Δ߹ʹඋ͑ͯผ్ۓٸ࿈ བྷઌͷϦετΛwikiʹ·ͱΊͯ͋Δ
bot touban help
On-call൪ϩʔςͷ ཧͱΤεΧϨʔγϣϯ • PagerDutyͷػೳΛ׆༻ • On Call Schedules • Escalation
Policy • https://www.pagerduty.com/
PagerDuty API & Google Apps Script • On-call൪ͷ༧ఆ࣮ΛGoogleͷεϓ ϨουγʔτΧϨϯμʔʹಉظ •
ఆظతʹूܭͯ͠څ༩ٳՋʹө
·ͱΊ • ۓٸ࣌ʹඋ͑ͯOn-callମ੍Λ͑Α͏ • On-callۀෛ୲͕େ͖͍ • ੍͘͠ΈͰΧόʔ • On-callSoftware Engineeringʂ
We are hiring! https://www.mercari.com/jp/jobs/