Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Eric Sigler

November 09, 2017

More Decks by Eric Sigler

See All by Eric Sigler

Instrumenting The Rest Of The Company: Hunting For Metrics

0

410

A Brief Introduction To DevOps

0

130

Humans are terrible compilers: A User's Guide

0

150

Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.

0

220

"Is there any strong objection?"

0

270

Fear, Uncertainty, and Continuous Deployment

1

150

0

280

Strategies For Being On Call & Keeping Your Sanity At The Same Time

0

200

Engineering for Engineers

0

110

Other Decks in Technology

See All in Technology

徹底討論！ECS vs EKS！

3

1.7k

週末にループ・エンジニアリングの理解を深めるためのスライド

0

310

フィジカル版Github Onshapeの紹介

0

320

【2026年版】ベクトル検索とEmbedding最前線

24

7.6k

AI時代に求められる技術力フロンティア・クリエイティビティ / Technical Excellence in the AI Era: Frontier Creativity

0

110

飲食店もAIで。レジ締めやハンディシステムをつくってる話 / Using AI for restaurant management

0

180

2026年6月23日 Syncable Tech + Start Python Club にて

0

150

AIネイティブな開発のサプライチェーンリスク対策〜激動の開発現場でリスクに立ち向かう〜【ZennFes】

PRO

2

160

いまさら聞けない「仕様駆動開発入門」〜AI活用時代の開発プロセスを考える〜

findy_eventslides

2

200

LayerX コーポレートエンジニアリング室におけるサプライチェーンセキュリティへの取り組み / Supply Chain Security at LayerX Corporate Engineering

3

840

AI Agentをシステムに組み込む前にゆるく向き合ってみる

0

150

[チョークトーク資料]AWS DevOps Agent を使いこなす / AWS Dev Ops Agent Chalk Talk AWS Summit Japan 2026

4

770

Featured

See All Featured

PRO

1

260

Helping Users Find Their Own Way: Creating Modern Search Experiences

31

3.2k

Creating an realtime collaboration tool: Agile Flush - .NET Oxford

35

2.5k

AI in Enterprises - Java and Open Source to the Rescue

0

1.3k

Building Flexible Design Systems

yeseniaperezcruz

330

40k

The Art of Programming - Codeland 2020

57

14k

How to Talk to Developers About Accessibility

2

250

Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum

0

620

Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO

PRO

0

200

Sharpening the Axe: The Primacy of Toolmaking

46

2.9k

[SF Ruby Conf 2025] Rails X

2

1.1k

Discover your Explorer Soul

2

1.1k

Transcript

Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of
breaking things in production, on purpose.
@esigler Obligatory disclaimer: This is what works for us. Take
away ideas, not dogmas.
@esigler
@esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016
2017
@esigler 2013 2014 2015 2016 2017
None
@esigler 2014: Expanding Scope 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2015: Automation 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Also 2016: Putting It All Together 2013 2014 2015
2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections
resulting in a public postmortem: 3
@esigler Simulated full AZ failures: 4 Simulated full Region failures:
3 Simulated partial Disaster Recovery: 2
@esigler Tickets created from Failure Friday: over 225 Distinct services
that had faults injected: 49
@esigler
@esigler Optimized for learning first, tooling second Built the toolchain
to enable other teams Distributed chaos engineering knowledge
@esigler