Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Four years of breaking things in production, on...

Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
90
0

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
420
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
140
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
170
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
240
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
300
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
160
3AM, a survey.
Avatar for Eric Sigler esigler
0
300
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
210
Engineering for Engineers
Avatar for Eric Sigler esigler
0
130

Other Decks in Technology

See All in Technology
MCPでAIが自分で調べに行く
Avatar for CData Software Japan cdataj
0
130
AWS Quick 入門 | 目指せ Quick マスター 01
Avatar for Riku INADA inariku
PRO
2
270
コードレビューをなくして見えたもの
Avatar for henteko henteko
0
140
BALLAS Technology Deck
Avatar for 株式会社BALLAS ballas_inc
0
380
The Author Pays
Avatar for James Thomas hiccupps
0
150
[JAWS Festa 2026]一番いい仕事は、請求書に現れない / jawsfesta2026
Avatar for sh_fk2 sh_fk2
2
430
慶應義塾大学 AI実践05 論文執筆の仕組み化
Avatar for Semantic Machine Intelligence Lab., Keio Univ. keio_smilab
PRO
7
3.8k
田んぼの見回りをIoTで減らしたい!水位監視から電池駆動カメラまでの試行錯誤
Avatar for Katsuyoshi Ito itosoft
1
100
AI Agent 時代のデータ基盤： ClickHouse で実現する 超高速分析とコスト削減の両立
Avatar for jozono jozono
0
220
公共分野での生成AI活用
Avatar for Toru_Kubota toru_kubota
1
310
cdkd (CDK Direct)
Avatar for k.goto gotok365
4
280
FinOpsを導入してマルチプロダクトのコスト最適化を実現する
Avatar for KNOWLEDGE WORK / 株式会社ナレッジワーク kworkdev
PRO
0
140

Featured

See All Featured
BBQ
Avatar for Matthew Crist matthewcrist
90
10k
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
50
15k
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
274
21k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3.4k
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
516
110k
Mozcon NYC 2025: Stop Losing SEO Traffic
Avatar for Sam Torres samtorres
1
600
Darren the Foodie - Storyboard
Avatar for Kevinho.art khoart
PRO
4
4.1k
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.6k
Introduction to Domain-Driven Design and Collaborative software design
Avatar for Kenny Baas-Schwegler baasie
1
1.1k
How to Talk to Developers About Accessibility
Avatar for Jason CranfordTeague jct
2
620
The Spectacular Lies of Maps
Avatar for Per Axbom axbom
PRO
1
1.1k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
12
1.4k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler