Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
0
62

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017
Tweet

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
400
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
120
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
130
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
190
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
240
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
130
3AM, a survey.
Avatar for Eric Sigler esigler
0
250
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
180
Engineering for Engineers
Avatar for Eric Sigler esigler
0
110

Other Decks in Technology

See All in Technology
日本の85%が使う公共SaaSは、どう育ったのか
Avatar for たけだ taketakekaho
1
240
会社紹介資料 / Sansan Company Profile
Avatar for Sansan, Inc. sansan33
PRO
15
400k
生成AIと余白 〜開発スピードが向上した今、何に向き合う？〜
Avatar for KAKEHASHI kakehashi
PRO
0
160
ECS障害を例に学ぶ、インシデント対応に備えた AIエージェントの育て方 / How to develop AI agents for incident response with ECS outage
Avatar for iselegant iselegant
4
400
ClickHouseはどのように大規模データを活用したAIエージェントを全社展開しているのか
Avatar for Miki Matsumoto mikimatsumoto
0
270
フルカイテン株式会社 エンジニア向け採用資料
Avatar for FULL KAITEN fullkaiten
0
10k
外部キー制約の知っておいて欲しいこと - RDBMSを正しく使うために必要なこと / FOREIGN KEY Night
Avatar for soudai sone soudai
PRO
12
5.6k
予期せぬコストの急増を障害のように扱う――「コスト版ポストモーテム」の導入とその後の改善
Avatar for Masahiro Yoshizawa muziyoshiz
1
2.1k
SRE Enabling戦記 - 急成長する組織にSREを浸透させる戦いの歴史
Avatar for Masaki Ishigaki markie1009
0
170
Embedded SREの終わりを設計する　「なんとなく」から計画的な自立支援へ
Avatar for SansanTech sansantech
PRO
3
2.6k
pool.ntp.orgに ⾃宅サーバーで 参加してみたら...
Avatar for Fuminori -Tany- Tanizaki(谷崎文義) tanyorg
0
1.1k
30万人の同時アクセスに耐えたい！新サービスの盤石なリリースを支える負荷試験 / SRE Kaigi 2026
Avatar for GENDA genda
4
1.4k

Featured

See All Featured
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k
Leveraging LLMs for student feedback in introductory data science courses - posit::conf(2025)
Avatar for Mine Cetinkaya-Rundel minecr
0
160
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
Avatar for konakalab konakalab
0
350
Ten Tips & Tricks for a 🌱 transition
Avatar for Manu Carrasco Molina stuffmc
0
71
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
698
190k
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
1
300
Max Prin - Stacking Signals: How International SEO Comes Together (And Falls Apart)
Avatar for Tech SEO Connect techseoconnect
PRO
0
89
The SEO Collaboration Effect
Avatar for Kristina Bergwall kristinabergwall1
0
350
How to make the Groovebox
Avatar for あそなす asonas
2
1.9k
Design in an AI World
Avatar for tapps tapps
0
150
Leveraging Curiosity to Care for An Aging Population
Avatar for Cassini Nazir cassininazir
1
170
The Curse of the Amulet
Avatar for Matthew Lei leimatthew05
1
8.7k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler