Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
0
62

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017
Tweet

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
390
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
120
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
130
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
190
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
230
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
130
3AM, a survey.
Avatar for Eric Sigler esigler
0
240
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
180
Engineering for Engineers
Avatar for Eric Sigler esigler
0
110

Other Decks in Technology

See All in Technology
わが10年の叡智をぶつけたカオスなクラウドインフラが、なくなるということ。
Avatar for Hisashi SOGA sogaoh
PRO
1
190
フルカイテン株式会社 エンジニア向け採用資料
Avatar for FULL KAITEN fullkaiten
0
10k
AWS re:Invent 2025 を振り返る
Avatar for kazzpapa3 kazzpapa3
2
110
AI との良い付き合い方を僕らは誰も知らない (WSS 2026 静岡版)
Avatar for Asei Sugiyama asei
1
210
20251225_たのしい出張報告＆IgniteRecap！
Avatar for ponponmikan ponponmikankan
0
110
モノタロウ x クリエーションラインで実現する チームトポロジーにおける プラットフォームチーム・ ストリームアラインドチームの 効果的なコラボレーション
Avatar for Creationline creationline
0
340
形式手法特論：コンパイラの「正しさ」は証明できるか？ #burikaigi / BuriKaigi 2026
Avatar for y_taka_23 ytaka23
15
3.7k
人工知能のための哲学塾 ニューロフィロソフィ篇 第零夜 「ニューロフィロソフィとは何か？」
Avatar for miyayou miyayou
0
330
Node vs Deno vs Bun 〜推しランタイムを見つけよう〜
Avatar for すずとも kamekyame
1
180
ハッカソンから社内プロダクトへ AIエージェント ko☆shi 開発で学んだ4つの重要要素
Avatar for Tech Leverages leveragestech
0
540
Master Dataグループ紹介資料
Avatar for Sansan, Inc. sansan33
PRO
1
4.2k
製造業から学んだ「本質を守り現場に合わせるアジャイル実践」
Avatar for yaskamito kamitokusari
0
220

Featured

See All Featured
Game over? The fight for quality and originality in the time of robots
Avatar for Wayne Barker wayneb77
1
74
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
49
3.3k
Navigating Team Friction
Avatar for Lara Hogan lara
191
16k
The SEO Collaboration Effect
Avatar for Kristina Bergwall kristinabergwall1
0
320
AI Search:
Where Are We & What Can We Do About It?
Avatar for Aleyda Solis aleyda
0
6.8k
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
615
210k
Are puppies a ranking factor?
Avatar for Jono Alderson jonoalderson
0
2.6k
The Hidden Cost of Media on the Web [PixelPalooza 2025]
Avatar for Tammy Everts tammyeverts
2
130
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
254
22k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
65
8.3k
First, design no harm
Avatar for Per Axbom axbom
PRO
1
1.1k
Claude Code のすすめ
Avatar for schroneko schroneko
67
210k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler