Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
0
58

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.
Avatar for Eric Sigler

Eric Sigler

November 09, 2017
Tweet

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
370
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
110
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
120
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
170
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
220
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
120
3AM, a survey.
Avatar for Eric Sigler esigler
0
230
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
160
Engineering for Engineers
Avatar for Eric Sigler esigler
0
89

Other Decks in Technology

See All in Technology
Definition of Done
Avatar for Yasunobu Kawaguchi kawaguti
PRO
6
480
Agentic Workflowという選択肢を考える
Avatar for takuya kikuchi tkikuchi1002
1
480
25分で解説する「最小権限の原則」を実現するための AWS「ポリシー」大全 / 20250625-aws-summit-aws-policy
Avatar for opelab opelab
9
1.1k
Кто отправит outbox? Валентин Удальцов, автор канала Пых
Avatar for Lamoda Tech lamodatech
0
330
JSX - 歴史を振り返り、⾯⽩がって、エモくなろう
Avatar for 晴れ井戸 pal4de
4
1.1k
Claude Code Actionを使ったコード品質改善の取り組み
Avatar for Katsunori Kanda potix2
PRO
6
2.1k
2年でここまで成長！AWSで育てたAI Slack botの軌跡
Avatar for iwamot iwamot
PRO
4
650
Абьюзим random_bytes(). Фёдор Кулаков, разработчик Lamoda Tech
Avatar for Lamoda Tech lamodatech
0
330
Amazon Bedrockで実現する 新たな学習体験
Avatar for Kazuki Maeda kzkmaeda
1
510
Prox Industries株式会社 会社紹介資料
Avatar for Prox proxindustries
0
260
生成AIでwebアプリケーションを作ってみた
Avatar for tajimon tajimon
2
140
～宇宙最速～ 2025年AWS Summit レポート
Avatar for sleepingsato satodesu
1
1.8k

Featured

See All Featured
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
41
7.3k
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1031
460k
Designing for Performance
Avatar for Lara Hogan lara
609
69k
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
39
1.9k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
138
34k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
36
2.8k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
94
14k
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
270
20k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
51
8.4k
Navigating Team Friction
Avatar for Lara Hogan lara
187
15k
Bootstrapping a Software Product
Avatar for Garrett Dimon garrettdimon
PRO
307
110k
Side Projects
Avatar for Sacha Greif sachag
455
42k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler