Four years of breaking things in production, on...

November 09, 2017

88

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Eric Sigler

November 09, 2017

More Decks by Eric Sigler

See All by Eric Sigler

Instrumenting The Rest Of The Company: Hunting For Metrics

0

420

A Brief Introduction To DevOps

0

140

Humans are terrible compilers: A User's Guide

0

160

Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.

0

240

"Is there any strong objection?"

0

290

Fear, Uncertainty, and Continuous Deployment

1

160

0

300

Strategies For Being On Call & Keeping Your Sanity At The Same Time

0

210

Engineering for Engineers

0

130

Other Decks in Technology

See All in Technology

USB PDを使ってスマホを労わりたい

0

180

V8のArray.prototype.flatを最大約20倍速くするまでと、巨大OSSへの大規模コミットの道のり

0

310

AI コーディングと人間の認知スループット / CTO Night & Day 2026 Morning Session / AI Coding and Human Cognitive Throughput

5

1.3k

SAENM ~Jupyterで残すPoCの試行錯誤、未来の自分と仲間を助ける記録！~

0

150

SharedWorkerを使ったキャッシュでネイティブアプリに負けないユーザー体験を目指す

PRO

1

110

宣言型部分更新は何を解決しどこへ向かうのか

1

490

会社紹介資料 / Sansan Company Profile

PRO

31

430k

Browser Spec: Testing Hotwire with Ruby

1

130

フロントエンドカンファレンス関西2026 - 速すぎるUIに、待ち時間を

PRO

0

200

Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces

PRO

0

140

Rust で関数型ドメインモデリング

0

170

スキマバイトという雇用形態は存在しないので、雇用期間1日のアルバイト契約を事業者とワーカーが結ぶ手続きを支えるのがタイミーなんです

0

210

Featured

See All Featured

Lightning Talk: Beautiful Slides for Beginners

PRO

2

730

How to build an LLM SEO readiness audit: a practical framework

2

950

Building the Perfect Custom Keyboard

2

910

Bioeconomy Workshop: Dr. Julius Ecuru, Opportunities for a Bioeconomy in West Africa

PRO

1

410

Skip the Path - Find Your Career Trail

1

260

AI: The stuff that nobody shows you

PRO

10

1.2k

The Success of Rails: Ensuring Growth for the Next 100 Years

47

8.4k

Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]

11

1.1k

HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy

PRO

0

760

PRO

2

880

Impact Scores and Hybrid Strategies: The future of link building

tamaranovitovic

0

460

187

17k

Transcript

Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of
breaking things in production, on purpose.
@esigler Obligatory disclaimer: This is what works for us. Take
away ideas, not dogmas.
@esigler
@esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016
2017
@esigler 2013 2014 2015 2016 2017
None
@esigler 2014: Expanding Scope 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2015: Automation 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Also 2016: Putting It All Together 2013 2014 2015
2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections
resulting in a public postmortem: 3
@esigler Simulated full AZ failures: 4 Simulated full Region failures:
3 Simulated partial Disaster Recovery: 2
@esigler Tickets created from Failure Friday: over 225 Distinct services
that had faults injected: 49
@esigler
@esigler Optimized for learning first, tooling second Built the toolchain
to enable other teams Distributed chaos engineering knowledge
@esigler