Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
0
58

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017
Tweet

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
370
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
110
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
120
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
180
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
230
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
120
3AM, a survey.
Avatar for Eric Sigler esigler
0
230
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
160
Engineering for Engineers
Avatar for Eric Sigler esigler
0
91

Other Decks in Technology

See All in Technology
【Grafana Meetup Japan #6】Grafanaをリバプロ配下で動かすときにやること ~ Grafana Liveってなんだ ~
Avatar for よしたけ yoshitake945
0
220
トヨタ生産方式(TPS)入門
Avatar for Recruit recruitengineers
PRO
6
1.4k
『FailNet～やらかし共有SNS～』エレベーターピッチ
Avatar for yoko yokomachi
1
200
AWSで推進するデータマネジメント
Avatar for かわなご kawanago
0
850
見てわかるテスト駆動開発
Avatar for Recruit recruitengineers
PRO
6
2.4k
実践アプリケーション設計 ①データモデルとドメインモデル
Avatar for Recruit recruitengineers
PRO
5
1.4k
Snowflakeの生成AI機能を活用したデータ分析アプリの作成 〜Cortex AnalystとCortex Searchの活用とStreamlitアプリでの利用〜
Avatar for Nayuta S. nayuts
0
150
攻撃と防御で実践するプロダクトセキュリティ演習～導入パート～
Avatar for Recruit recruitengineers
PRO
3
1.8k
実践AIガバナンス
Avatar for Asei Sugiyama asei
3
300
異業種出身エンジニアが気づいた、転向して十数年経っても変わらない自分の武器とは
Avatar for macneko-ayu macnekoayu
0
260
20250903_1つのAWSアカウントに複数システムがある環境におけるアクセス制御をABACで実現.pdf
Avatar for yhana yhana
2
270
データアナリストからアナリティクスエンジニアになった話
Avatar for hatsu hiyokko_data
0
280

Featured

See All Featured
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.2k
How GitHub (no longer) Works
Avatar for Zach Holman holman
315
140k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
37
2.8k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
26
3k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
236
140k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
69
4.8k
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
9
570
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
246
12k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
3
610
Testing 201, or: Great Expectations
Avatar for Joseph Mastey jmmastey
45
7.6k
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Docker and Python
Avatar for Tania Allard trallard
45
3.5k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler