Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ensuring Success During Disaster - SRECon 2015

Doug Barth
March 16, 2015
Technology
0
240

Ensuring Success During Disaster - SRECon 2015

Video will be at SRECon Program: https://www.usenix.org/conference/srecon15/program/presentation/barth

Surviving a large scale outage requires more than just standing up a few extra servers. Validation and capacity planning can mean the difference between proper mitigation, or just a bunch of wasted effort. This talk will explore how to ensure DR success, gleaned from PagerDuty's production systems.

Doug Barth

March 16, 2015
Tweet

More Decks by Doug Barth

See All by Doug Barth
Zero Trust Networks: In Theory and In Practice
dougbarth
1
470
Culture from Chaos
dougbarth
0
87
IPsec mesh network: Perfect for the cloud?
dougbarth
1
1.5k
Failure Friday: Start Injecting Failure Today! (DevOpsDays Austin 2015)
dougbarth
0
620
Making PagerDuty More Reliable Using XtraDB Cluster
dougbarth
0
190
Minimum Viable Ops
dougbarth
0
79
Policy as Code
dougbarth
1
290
Living without scheduled maintenance
dougbarth
2
1.1k
Failure Friday: Start Injecting Failure Today!
dougbarth
0
120

Other Decks in Technology

See All in Technology
Snowflake女子会#3 Snowpipeの良さを5分で語るよ
lana2548
0
220
OpenAIの蒸留機能(Model Distillation)を使用して運用中のLLMのコストを削減する取り組み
pharma_x_tech
4
550
20241214_WACATE2024冬_テスト設計技法をチョット俯瞰してみよう
kzsuzuki
3
440
5分でわかるDuckDB
chanyou0311
10
3.2k
複雑性の高いオブジェクト編集に向き合う: プラガブルなReactフォーム設計
righttouch
PRO
0
110
私なりのAIのご紹介 [2024年版]
qt_luigi
1
120
LINEスキマニにおけるフロントエンド開発
lycorptech_jp
PRO
0
330
Postman と API セキュリティ / Postman and API Security
yokawasa
0
200
PHPからGoへのマイグレーション for DMMアフィリエイト
yabakokobayashi
1
160
re:Invent をおうちで楽しんでみた ～CloudWatch のオブザーバビリティ機能がスゴい！/ Enjoyed AWS re:Invent from Home and CloudWatch Observability Feature is Amazing!
yuj1osm
0
120
AWS re:Invent 2024 ふりかえり
kongmingstrap
0
130
組織に自動テストを書く文化を根付かせる戦略（2024冬版） / Building Automated Test Culture 2024 Winter Edition
twada
PRO
12
3.5k

Featured

See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
28
900
Building a Modern Day 
E-commerce SEO Strategy
aleyda
38
7k
Making Projects Easy
brettharned
116
5.9k
Rails Girls Zürich Keynote
gr2m
94
13k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
232
17k
VelocityConf: Rendering Performance Case Studies
addyosmani
326
24k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
26
1.9k
Code Review Best Practice
trishagee
65
17k
Adopting Sorbet at Scale
ufuk
73
9.1k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
10
810
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
28
4.4k

Transcript

  1. 3/20/15 Ensuring success during disaster SRECON 2015

  2. 3/20/15 ENSURING SUCCESS DURING DISASTER @dougbarth

  3. 3/20/15 ENSURING SUCCESS DURING DISASTER

  4. 3/20/15 ENSURING SUCCESS DURING DISASTER Dev Ops Consider killing this

  5. 3/20/15 ENSURING SUCCESS DURING DISASTER How is babby PagerDuty formed?

  6. 3/20/15 ENSURING SUCCESS DURING DISASTER

  7. 3/20/15 ENSURING SUCCESS DURING DISASTER Reliability

  8. 3/20/15 ENSURING SUCCESS DURING DISASTER Agenda

  9. 3/20/15 ENSURING SUCCESS DURING DISASTER DR vs. HA Data DR

    Failover Active/Active Legacy Systems Q&A

  10. 3/20/15 ENSURING SUCCESS DURING DISASTER Disaster Recovery

  11. 3/20/15 ENSURING SUCCESS DURING DISASTER A plan for surviving rare

    failure events that threaten our ability to continue operating

  12. 3/20/15 ENSURING SUCCESS DURING DISASTER High availability

  13. 3/20/15 ENSURING SUCCESS DURING DISASTER DR HA Rare failures Common

    failures Slow recovery Fast recovery

  14. 3/20/15 ENSURING SUCCESS DURING DISASTER Latency

  15. 3/20/15 ENSURING SUCCESS DURING DISASTER DR needs to exist

  16. 3/20/15 ENSURING SUCCESS DURING DISASTER DR needs to be tested

  17. 3/20/15 ENSURING SUCCESS DURING DISASTER DR needs to be tested

    for correctness

  18. 3/20/15 ENSURING SUCCESS DURING DISASTER DR needs to be tested

    for capacity

  19. 3/20/15 ENSURING SUCCESS DURING DISASTER DR needs to be tested

    for execution

  20. 3/20/15 ENSURING SUCCESS DURING DISASTER Data DR

  21. 3/20/15 ENSURING SUCCESS DURING DISASTER You can never have too

    many copies

  22. 3/20/15 ENSURING SUCCESS DURING DISASTER Lose a disk? Lose a

    server?

  23. 3/20/15 ENSURING SUCCESS DURING DISASTER Secondary Primary Secondary

  24. 3/20/15 ENSURING SUCCESS DURING DISASTER Data corruption?

  25. 3/20/15 ENSURING SUCCESS DURING DISASTER Backups

  26. 3/20/15 ENSURING SUCCESS DURING DISASTER Test restorations

  27. 3/20/15 ENSURING SUCCESS DURING DISASTER DROP TABLE USERS; DELETE FROM

    USERS;

  28. 3/20/15 ENSURING SUCCESS DURING DISASTER Delayed secondary Primary Secondary (delayed

    2 hrs)

  29. 3/20/15 ENSURING SUCCESS DURING DISASTER Failover

  30. 3/20/15 ENSURING SUCCESS DURING DISASTER Primary DC & Secondary DC

  31. 3/20/15 ENSURING SUCCESS DURING DISASTER The flip

  32. 3/20/15 ENSURING SUCCESS DURING DISASTER Testing == outage

  33. 3/20/15 ENSURING SUCCESS DURING DISASTER So it goes untested

  34. 3/20/15 ENSURING SUCCESS DURING DISASTER Tested infrequently

  35. 3/20/15 ENSURING SUCCESS DURING DISASTER Flip test Flip test Breaking

    change Failure window

  36. 3/20/15 ENSURING SUCCESS DURING DISASTER Forgotten roles

  37. 3/20/15 ENSURING SUCCESS DURING DISASTER Low capacity

  38. 3/20/15 ENSURING SUCCESS DURING DISASTER Expensive

  39. 3/20/15 ENSURING SUCCESS DURING DISASTER Active/Active

  40. 3/20/15 ENSURING SUCCESS DURING DISASTER PagerDuty Active/Active

  41. 3/20/15 ENSURING SUCCESS DURING DISASTER Multiple regions Use all the

    AZs!

  42. 3/20/15 ENSURING SUCCESS DURING DISASTER

  43. 3/20/15 ENSURING SUCCESS DURING DISASTER Constant validation

  44. 3/20/15 ENSURING SUCCESS DURING DISASTER Failure Friday HTTPS://WWW.PAGERDUTY.COM/BLOG/FAILURE-FRIDAY-AT-PAGERDUTY/

  45. 3/20/15 ENSURING SUCCESS DURING DISASTER Trade latency for reliability

  46. 3/20/15 ENSURING SUCCESS DURING DISASTER 30ms RTT

  47. 3/20/15 ENSURING SUCCESS DURING DISASTER Legacy Systems

  48. 3/20/15 ENSURING SUCCESS DURING DISASTER Older systems still use failover

  49. 3/20/15 ENSURING SUCCESS DURING DISASTER Correctness Capacity Execution

  50. 3/20/15 ENSURING SUCCESS DURING DISASTER Correctness Capacity Execution

  51. 3/20/15 ENSURING SUCCESS DURING DISASTER 1% of requests go to

    DR site

  52. 3/20/15 ENSURING SUCCESS DURING DISASTER Finds issues immediately

  53. 3/20/15 ENSURING SUCCESS DURING DISASTER Requires DR uses production DB

    DR Prod App App App App App App

  54. 3/20/15 ENSURING SUCCESS DURING DISASTER 30ms per DB call

  55. 3/20/15 ENSURING SUCCESS DURING DISASTER Trade latency to test correctness

  56. 3/20/15 ENSURING SUCCESS DURING DISASTER Correctness Capacity Execution

  57. 3/20/15 ENSURING SUCCESS DURING DISASTER DR is a mirror of

    production

  58. 3/20/15 ENSURING SUCCESS DURING DISASTER

  59. 3/20/15 ENSURING SUCCESS DURING DISASTER dr-compare.rb

  60. 3/20/15 ENSURING SUCCESS DURING DISASTER Checks that DR footprint is

    the same as production

  61. 3/20/15 ENSURING SUCCESS DURING DISASTER Surfaces missing roles

  62. 3/20/15 ENSURING SUCCESS DURING DISASTER Correctness Capacity Execution

  63. 3/20/15 ENSURING SUCCESS DURING DISASTER The flip

  64. 3/20/15 ENSURING SUCCESS DURING DISASTER Production impact

  65. 3/20/15 ENSURING SUCCESS DURING DISASTER Scale up staging

  66. 3/20/15 ENSURING SUCCESS DURING DISASTER Practice flip there

  67. 3/20/15 ENSURING SUCCESS DURING DISASTER Tested infrequently

  68. 3/20/15 ENSURING SUCCESS DURING DISASTER PagerDuty Active/Active

  69. 3/20/15 ENSURING SUCCESS DURING DISASTER Correctness Capacity Execution

  70. 3/20/15 ENSURING SUCCESS DURING DISASTER pagerduty.com/jobs [email protected]

  71. 3/20/15 [email protected] Questions?