Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Failure Friday: Start Injecting Failure Today!
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
Doug Barth
September 12, 2014
Technology
150
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Failure Friday: Start Injecting Failure Today!
DevOpsDays Toronto 2014
Video:
http://vimeo.com/107528697
Doug Barth
September 12, 2014
Transcript
32 slides
9/15/14 @dougbarth DEVOPSDAYS TORONTO 2014 Failure Friday!
9/15/14 FAILURE FRIDAY! Dev Ops
9/15/14 FAILURE FRIDAY! DevOps Engineer
9/15/14 “DO NOT FEAR FAILURE” BY TOMASZ STASIUK
9/15/14 FAILURE FRIDAY! How is babby PagerDuty formed?
9/15/14 FAILURE FRIDAY!
9/15/14 Designed for reliability FAILURE FRIDAY! Downstream providers fail 3
phone providers 3 email providers 6 SMS providers PagerDuty providers fail 2 cloud providers 3 data centers
9/15/14 Hung up on details FAILURE FRIDAY! Bugs in exceptional
code paths Systems not recovering as quickly as expected What is normal when things are abnormal?
9/15/14 FAILURE FRIDAY!
9/15/14 Simian Army FAILURE FRIDAY! Chaos Monkey Latency Monkey Chaos
Gorilla Chaos Kong “WP7WALLPAPER_EVIL_MONKEY_09” BY SKYLER817
9/15/14 Keep it simple FAILURE FRIDAY! “KISS BAND MEMBER CUPCAKES”
BY CLEVER CUPCAKES
9/15/14 Process FAILURE FRIDAY! “HOW TO DRAW AN OWL” BY
CHESTER
9/15/14 Get buy in FAILURE FRIDAY! “ANGRY BOSS” BY KAUSHAL
KARKHANIS
9/15/14 Schedule FAILURE FRIDAY! 1 hour recurring meeting Developers &
Operations List of attacks and identify victim Finish as much as possible
9/15/14 Before starting FAILURE FRIDAY! Disable cron jobs & CM
system Announce the start Open up relevant dashboards Leave alarms enabled
9/15/14 Attacks FAILURE FRIDAY! Test a single host and then
DC 5 minutes Return to a working state Stop if things break
9/15/14 Keep a log FAILURE FRIDAY! Keep track of actions
taken Times are super important Also track discoveries and TODOs Share dashboards/metrics Chat rooms make this easy
9/15/14 Graphs are awesome FAILURE FRIDAY!
9/15/14 Finishing up FAILURE FRIDAY! Sound the all clear Enable
crons & CM Move TODOs to issue tracker
9/15/14 Attack Strategies FAILURE FRIDAY! “UNICORN ATTACK!” BY SAM HOWZIT
9/15/14 FAILURE FRIDAY! SERVICE STOP CASSANDRA
9/15/14 FAILURE FRIDAY! SHUTDOWN -R NOW
9/15/14 FAILURE FRIDAY! IPTABLES -I INPUT 1 -P TCP --DPORT
9160 -J DROP IPTABLES -I INPUT 1 -P TCP --DPORT 7000 -J DROP ! IPTABLES -I OUTPUT 1 -P TCP --SPORT 9160 -J DROP IPTABLES -I OUTPUT 1 -P TCP --SPORT 7000 -J DROP
9/15/14 FAILURE FRIDAY! TC QDISC ADD DEV ETH0 ROOT NETEM
DELAY 500MS 100MS LOSS 5%
9/15/14 “RESULTS READER BOARD” BY ROSA SAY
9/15/14 Issues fixed FAILURE FRIDAY! Aggressive restarts by monit Large
files on ext3 volumes Failing to restart due to bad /etc/fstab file High latency from network isolated cache Low capacity with a lost DC Missing alerts/metrics
9/15/14 Cultural impact FAILURE FRIDAY! Knowledge sharing Highlights untestable systems
Keeps failure handling on everyone’s mind
9/15/14 Future plans “ROBOT SWORDSMAN FIGHT.” BY PATRICK GAGE KELLEY
9/15/14 Break more things FAILURE FRIDAY! Start testing whole DC
outages Break multiple services at once Distribute failure testing to teams Automate
9/15/14 Break more things FAILURE FRIDAY! Start testing whole DC
outages Break multiple services at once Distribute failure testing to teams Automate
9/15/14 Summary FAILURE FRIDAY! Failures will happen Proactively test failure
handling now Choose something easy: app server, cache Automate later
9/15/14 pagerduty.com/jobs Thank you.
More Decks by Doug Barth
See All by Doug Barth
Zero Trust Networks: In Theory and In Practice
dougbarth
1
570
Culture from Chaos
dougbarth
0
140
IPsec mesh network: Perfect for the cloud?
dougbarth
1
1.7k
Failure Friday: Start Injecting Failure Today! (DevOpsDays Austin 2015)
dougbarth
0
810
Making PagerDuty More Reliable Using XtraDB Cluster
dougbarth
0
240
Ensuring Success During Disaster - SRECon 2015
dougbarth
0
310
Minimum Viable Ops
dougbarth
0
140
Policy as Code
dougbarth
1
340
Living without scheduled maintenance
dougbarth
2
1.2k
Other Decks in Technology
See All in Technology
HTTP通信に使おうとすると実はめんどくさいfile_get_contents
maguroalternative
0
260
AIエージェントのアーキテクチャ設計
nsatoh
0
530
Platform Engineering をベースとした Agentic Software Factory の実践
110y
3
960
共通基盤の実装に Kotlin - Quarkus を選定した理由と採用したアーキテクチャ
nealle
1
160
Web Performance Is a Journey [PerfNow 2026]
tammyeverts
2
140
ノードが消える前提で設計する!Karpenterで金融システムを止めないためのノウハウ大全
daitak
1
240
[NW-JAWS DAY]「うちのAWS、インターネット出られないんで」― AWS外の自社サービスをPrivateLinkでシュッと公開する方法
ykimi
3
920
信頼性を犠牲にせず、PR マージからデプロイまで人の介入を完全にゼロにするまでの道のり
knwoop
4
2.1k
プラガブルで宣言的なソフトウェアファクトリーを支えるTypeScript
timakin
0
160
OCI Artifact による自動運転タクシーに向けた OTA 基盤の構築
biwashi
1
300
メルカリの小規模チーム「Pod」で、 エンジニアがデータアナリストなしで、 AIを使ってデータ分析の自走を目指す取り組み
jackojacko_
1
350
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
17
120k
Featured
See All Featured
Agile that works and the tools we love
rasmusluckow
331
22k
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
uxyall
2
540
So, you think you're a good person
axbom
PRO
2
2.3k
Exploring anti-patterns in Rails
aemeredith
4
580
The Hidden Cost of Media on the Web [PixelPalooza 2025]
tammyeverts
2
580
[Rails World 2026] Durable orchestration on Rails: from continuation to workflow
palkan
4
1.3k
Java REST API Framework Comparison - PWX 2021
mraible
34
9.8k
Believing is Seeing
oripsolob
1
290
What's in a price? How to price your products and services
michaelherold
247
13k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
9
1.7k
The World Runs on Bad Software
bkeepers
PRO
72
12k
WENDY [Excerpt]
tessaabrams
15
41k