Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
0
62

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017
Tweet

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
390
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
120
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
130
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
190
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
230
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
130
3AM, a survey.
Avatar for Eric Sigler esigler
0
240
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
180
Engineering for Engineers
Avatar for Eric Sigler esigler
0
110

Other Decks in Technology

See All in Technology
松尾研LLM講座2025 応用編Day3「軽量化」 講義資料
Avatar for Aratako aratako
11
4.4k
AWSに革命を起こすかもしれない新サービス・アップデートについてのお話
Avatar for Yuuki Yamashita yama3133
0
520
Identity Management for Agentic AI 解説
Avatar for Naohiro Fujie fujie
0
490
Entity Framework Core におけるIN句クエリ最適化について
Avatar for tkym htkym
0
130
AIBuildersDay_track_A_iidaxs
Avatar for iidaxs iidaxs
4
1.4k
AWSの新機能をフル活用した「re:Inventエージェント」開発秘話
Avatar for みのるん minorun365
2
480
フィッシュボウルのやり方 / How to do a fishbowl
Avatar for Pauli pauli
2
400
業務の煩悩を祓うAI活用術108選 / AI 108 Usages
Avatar for 株式会社スマートバンク smartbank
9
15k
普段使ってるClaude Skillsの紹介(by Notebooklm)
Avatar for Higuchi kokoro zerebom
8
2.3k
AWSインフルエンサーへの道 / load of AWS Influencer
Avatar for WHIsaiyo whisaiyo
0
230
7,000万ユーザーの信頼を守る「TimeTree」のオブザーバビリティ実践 ( Datadog Live Tokyo )
Avatar for bell033 bell033
1
100
Knowledge Work の AI Backend
Avatar for KNOWLEDGE WORK / 株式会社ナレッジワーク kworkdev
PRO
0
290

Featured

See All Featured
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
304
21k
Building AI with AI
Avatar for Ines Montani inesmontani
PRO
1
580
State of Search Keynote: SEO is Dead Long Live SEO
Avatar for Ryan Jones ryanjones
0
73
AI Search:
Where Are We & What Can We Do About It?
Avatar for Aleyda Solis aleyda
0
6.7k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
671
58k
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
Avatar for soudai sone soudai
PRO
61
45k
The World Runs on Bad Software
Avatar for Brandon Keepers bkeepers
PRO
72
12k
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
410
Leadership Guide Workshop - DevTernity 2021
Avatar for David Neal reverentgeek
1
170
Jamie Indigo - Trashchat’s Guide to Black Boxes: Technical SEO Tactics for LLMs
Avatar for Tech SEO Connect techseoconnect
PRO
0
32
Joys of Absence: A Defence of Solitary Play
Avatar for Sebastian Deterding codingconduct
1
260
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
37
6.2k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler