The Overnight Failure

The Overnight Failure

This talk is based on a true horror story.

It is very likely that you too have created a big problem in production at some point in your career, wether by creating was a bug or running the wrong command. Here I share the story of how I did it this time and the lessons I learned from this experience.

Presented at: Euruko 2017

92d08794b535e41a4082c57ea547546e?s=128

Sebastian Sogamoso

September 30, 2017
Tweet

Transcript

  1. 1.
  2. 3.

    !

  3. 4.
  4. 5.
  5. 6.
  6. 8.
  7. 12.
  8. 13.
  9. 14.
  10. 15.
  11. 20.
  12. 21.
  13. 22.
  14. 23.
  15. 24.

    A B

  16. 25.

    A B

  17. 26.

    A B

  18. 27.

    A B

  19. 28.
  20. 29.
  21. 30.
  22. 31.
  23. 32.
  24. 33.
  25. 34.
  26. 35.
  27. 36.
  28. 37.
  29. 38.
  30. 39.
  31. 41.
  32. 42.
  33. 43.
  34. 48.
  35. 54.

    Recap • Users carpooled everyday • The payment process ran

    once a week • Passengers were charged • Drivers were paid
  36. 55.
  37. 59.
  38. 60.
  39. 61.
  40. 62.
  41. 66.

    Black Saturday Wekly process was ran 06:00 06:25 User couldn’t

    pay for breakafast 06:34 Users reported bug
  42. 67.
  43. 68.
  44. 69.

    Boss: hey, sorry to call you this early but we

    have a problem with payments in production and a lot of customers are complaining about it
  45. 73.

    Black Saturday Wekly process was ran 06:00 06:25 06:43 User

    couldn’t pay for breakafast 06:34 Users reported bug Manager woke me up
  46. 83.

    Black Saturday Wekly process was ran 06:00 06:25 07:28 06:43

    User couldn’t pay for breakafast 06:34 Users reported bug Manager woke me up Problem contained
  47. 84.
  48. 85.
  49. 87.

    Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID:

    9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 0 0 0
  50. 92.

    Black Saturday Wekly process was ran 06:00 06:25 06:43 User

    couldn’t pay for breakafast 06:34 Users reported bug Manager woke me up 22:50 Deployed a fix to production Problem contained 07:28
  51. 93.

    Black Saturday Wekly process was ran 06:00 06:25 06:43 User

    couldn’t pay for breakafast 06:34 Users reported bug Manager woke me up 22:50 Deployed a fix to production 22:55 Started looking for a new job Problem contained 07:28
  52. 94.

    Black Saturday Wekly process was ran 06:00 06:25 06:43 User

    couldn’t pay for breakafast 06:34 Users reported bug Manager woke me up 22:50 Deployed a fix to production Problem contained 07:28
  53. 95.
  54. 96.

    Thousands of users affected by the bug Users were charged

    up-to 200 times A single user was charged over $5k Maxed out credit cards. Emptied bank accounts
  55. 100.
  56. 109.