Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ensuring Success During Disaster - SRECon 2015

Doug Barth
March 16, 2015
Technology
0
230

Ensuring Success During Disaster - SRECon 2015

Video will be at SRECon Program: https://www.usenix.org/conference/srecon15/program/presentation/barth

Surviving a large scale outage requires more than just standing up a few extra servers. Validation and capacity planning can mean the difference between proper mitigation, or just a bunch of wasted effort. This talk will explore how to ensure DR success, gleaned from PagerDuty's production systems.

Doug Barth

March 16, 2015
Tweet

More Decks by Doug Barth

See All by Doug Barth
Zero Trust Networks: In Theory and In Practice
dougbarth
1
470
Culture from Chaos
dougbarth
0
87
IPsec mesh network: Perfect for the cloud?
dougbarth
1
1.5k
Failure Friday: Start Injecting Failure Today! (DevOpsDays Austin 2015)
dougbarth
0
610
Making PagerDuty More Reliable Using XtraDB Cluster
dougbarth
0
190
Minimum Viable Ops
dougbarth
0
79
Policy as Code
dougbarth
1
290
Living without scheduled maintenance
dougbarth
2
1.1k
Failure Friday: Start Injecting Failure Today!
dougbarth
0
120

Other Decks in Technology

See All in Technology
サイバーセキュリティと認知バイアス：対策の隙を埋める心理学的アプローチ
shumei_ito
0
380
100 名超が参加した日経グループ横断の競技型 AWS 学習イベント「Nikkei Group AWS GameDay」の紹介/mediajaws202411
nikkei_engineer_recruiting
1
170
TanStack Routerに移行するのかい しないのかい、どっちなんだい！ / Are you going to migrate to TanStack Router or not? Which one is it?
kaminashi
0
580
rootlessコンテナのすゝめ - 研究室サーバーでもできる安全なコンテナ管理
kitsuya0828
3
380
Python(PYNQ)がテーマのAMD主催のFPGAコンテストに参加してきた
iotengineer22
0
470
【若手エンジニア応援LT会】ソフトウェアを学んできた私がインフラエンジニアを目指した理由
kazushi_ohata
0
150
Can We Measure Developer Productivity?
ewolff
1
150
いざ、BSC討伐の旅
nikinusu
2
780
OCI Security サービス 概要
oracle4engineer
PRO
0
6.5k
テストコード品質を高めるためにMutation Testingライブラリ・Strykerを実戦導入してみた話
ysknsid25
7
2.6k
B2B SaaSから見た最近のC#/.NETの進化
sansantech
PRO
0
680
dev 補講: プロダクトセキュリティ / Product security overview
wa6sn
1
2.3k

Featured

See All Featured
Documentation Writing (for coders)
carmenintech
65
4.4k
Happy Clients
brianwarren
98
6.7k
Practical Orchestrator
shlominoach
186
10k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
126
18k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
860
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.5k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
31
2.7k
Visualization
eitanlees
145
15k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.5k
Bash Introduction
62gerente
608
210k

Transcript

  1. 3/20/15 Ensuring success during disaster SRECON 2015

  2. 3/20/15 ENSURING SUCCESS DURING DISASTER @dougbarth

  3. 3/20/15 ENSURING SUCCESS DURING DISASTER

  4. 3/20/15 ENSURING SUCCESS DURING DISASTER Dev Ops Consider killing this

  5. 3/20/15 ENSURING SUCCESS DURING DISASTER How is babby PagerDuty formed?

  6. 3/20/15 ENSURING SUCCESS DURING DISASTER

  7. 3/20/15 ENSURING SUCCESS DURING DISASTER Reliability

  8. 3/20/15 ENSURING SUCCESS DURING DISASTER Agenda

  9. 3/20/15 ENSURING SUCCESS DURING DISASTER DR vs. HA Data DR

    Failover Active/Active Legacy Systems Q&A

  10. 3/20/15 ENSURING SUCCESS DURING DISASTER Disaster Recovery

  11. 3/20/15 ENSURING SUCCESS DURING DISASTER A plan for surviving rare

    failure events that threaten our ability to continue operating

  12. 3/20/15 ENSURING SUCCESS DURING DISASTER High availability

  13. 3/20/15 ENSURING SUCCESS DURING DISASTER DR HA Rare failures Common

    failures Slow recovery Fast recovery

  14. 3/20/15 ENSURING SUCCESS DURING DISASTER Latency

  15. 3/20/15 ENSURING SUCCESS DURING DISASTER DR needs to exist

  16. 3/20/15 ENSURING SUCCESS DURING DISASTER DR needs to be tested

  17. 3/20/15 ENSURING SUCCESS DURING DISASTER DR needs to be tested

    for correctness

  18. 3/20/15 ENSURING SUCCESS DURING DISASTER DR needs to be tested

    for capacity

  19. 3/20/15 ENSURING SUCCESS DURING DISASTER DR needs to be tested

    for execution

  20. 3/20/15 ENSURING SUCCESS DURING DISASTER Data DR

  21. 3/20/15 ENSURING SUCCESS DURING DISASTER You can never have too

    many copies

  22. 3/20/15 ENSURING SUCCESS DURING DISASTER Lose a disk? Lose a

    server?

  23. 3/20/15 ENSURING SUCCESS DURING DISASTER Secondary Primary Secondary

  24. 3/20/15 ENSURING SUCCESS DURING DISASTER Data corruption?

  25. 3/20/15 ENSURING SUCCESS DURING DISASTER Backups

  26. 3/20/15 ENSURING SUCCESS DURING DISASTER Test restorations

  27. 3/20/15 ENSURING SUCCESS DURING DISASTER DROP TABLE USERS; DELETE FROM

    USERS;

  28. 3/20/15 ENSURING SUCCESS DURING DISASTER Delayed secondary Primary Secondary (delayed

    2 hrs)

  29. 3/20/15 ENSURING SUCCESS DURING DISASTER Failover

  30. 3/20/15 ENSURING SUCCESS DURING DISASTER Primary DC & Secondary DC

  31. 3/20/15 ENSURING SUCCESS DURING DISASTER The flip

  32. 3/20/15 ENSURING SUCCESS DURING DISASTER Testing == outage

  33. 3/20/15 ENSURING SUCCESS DURING DISASTER So it goes untested

  34. 3/20/15 ENSURING SUCCESS DURING DISASTER Tested infrequently

  35. 3/20/15 ENSURING SUCCESS DURING DISASTER Flip test Flip test Breaking

    change Failure window

  36. 3/20/15 ENSURING SUCCESS DURING DISASTER Forgotten roles

  37. 3/20/15 ENSURING SUCCESS DURING DISASTER Low capacity

  38. 3/20/15 ENSURING SUCCESS DURING DISASTER Expensive

  39. 3/20/15 ENSURING SUCCESS DURING DISASTER Active/Active

  40. 3/20/15 ENSURING SUCCESS DURING DISASTER PagerDuty Active/Active

  41. 3/20/15 ENSURING SUCCESS DURING DISASTER Multiple regions Use all the

    AZs!

  42. 3/20/15 ENSURING SUCCESS DURING DISASTER

  43. 3/20/15 ENSURING SUCCESS DURING DISASTER Constant validation

  44. 3/20/15 ENSURING SUCCESS DURING DISASTER Failure Friday HTTPS://WWW.PAGERDUTY.COM/BLOG/FAILURE-FRIDAY-AT-PAGERDUTY/

  45. 3/20/15 ENSURING SUCCESS DURING DISASTER Trade latency for reliability

  46. 3/20/15 ENSURING SUCCESS DURING DISASTER 30ms RTT

  47. 3/20/15 ENSURING SUCCESS DURING DISASTER Legacy Systems

  48. 3/20/15 ENSURING SUCCESS DURING DISASTER Older systems still use failover

  49. 3/20/15 ENSURING SUCCESS DURING DISASTER Correctness Capacity Execution

  50. 3/20/15 ENSURING SUCCESS DURING DISASTER Correctness Capacity Execution

  51. 3/20/15 ENSURING SUCCESS DURING DISASTER 1% of requests go to

    DR site

  52. 3/20/15 ENSURING SUCCESS DURING DISASTER Finds issues immediately

  53. 3/20/15 ENSURING SUCCESS DURING DISASTER Requires DR uses production DB

    DR Prod App App App App App App

  54. 3/20/15 ENSURING SUCCESS DURING DISASTER 30ms per DB call

  55. 3/20/15 ENSURING SUCCESS DURING DISASTER Trade latency to test correctness

  56. 3/20/15 ENSURING SUCCESS DURING DISASTER Correctness Capacity Execution

  57. 3/20/15 ENSURING SUCCESS DURING DISASTER DR is a mirror of

    production

  58. 3/20/15 ENSURING SUCCESS DURING DISASTER

  59. 3/20/15 ENSURING SUCCESS DURING DISASTER dr-compare.rb

  60. 3/20/15 ENSURING SUCCESS DURING DISASTER Checks that DR footprint is

    the same as production

  61. 3/20/15 ENSURING SUCCESS DURING DISASTER Surfaces missing roles

  62. 3/20/15 ENSURING SUCCESS DURING DISASTER Correctness Capacity Execution

  63. 3/20/15 ENSURING SUCCESS DURING DISASTER The flip

  64. 3/20/15 ENSURING SUCCESS DURING DISASTER Production impact

  65. 3/20/15 ENSURING SUCCESS DURING DISASTER Scale up staging

  66. 3/20/15 ENSURING SUCCESS DURING DISASTER Practice flip there

  67. 3/20/15 ENSURING SUCCESS DURING DISASTER Tested infrequently

  68. 3/20/15 ENSURING SUCCESS DURING DISASTER PagerDuty Active/Active

  69. 3/20/15 ENSURING SUCCESS DURING DISASTER Correctness Capacity Execution

  70. 3/20/15 ENSURING SUCCESS DURING DISASTER pagerduty.com/jobs [email protected]

  71. 3/20/15 [email protected] Questions?