The overnight failure

This talk is based on a true story.

Here I share the story of how I created a big bug and the lessons I learned from this experience. In hope that you can learn from them, so that when it happens to you (it eventually will), you are better prepared.

Presented at: RubyConf 2017

Transcript

1. The Overnight Failure
3. ☠
5. My name is Sebastián Tweet to me @sebasoga
6. cookpad.com
7. Why?
8. We all have broken the internet
9. We normally don’t talk about it in public
10. Impostor syndrome
15. No matter the scale, our bugs affect our users lives
16. We learn more from failure than success
17. What’s the worst thing that could happen to you at work?
18. The Overnight Failure
19. How?
23. A B
24. A B
25. A B
26. A B
32. A B 0
33. A B A B 1
34. A B A B 1
35. A B 2
38. A B 9
39. Once a week
43. Once a week
44. Once a week
45. Once a week
46. Once a week
48. Passenger Driver Total for trips
49. Once a week
50. Once a week
51. Once a week
52. Once a week
53. Recap • Users carpooled everyday • The payment process ran once a week • Passengers were charged • Drivers were paid
54. What?
55. Once a week 6:00am
56. Saturday Weekly process was ran 06:00
57. Black Saturday
61. 6:25am
62. Black Saturday Weekly process was ran 06:00 06:25 User couldn’t pay for breakfast
63. @#$&! @#$&!
64. 6:25am @#$&! @#$&!! @#$&! @#$&! @#$&!!!
65. Black Saturday Weekly process was ran 06:00 06:25 User couldn’t pay for breakfast 06:34 Users reported bug 22:50
67. 6:34am
68. Boss: hey, sorry to call you this early but we have a problem with payments in production and a lot of customers are complaining about it
69. Me: Sure, I’ll take a look right away. Let’s talk over chat
70. Me: (thinking I ended the call) Me: F********k!!!!!!
71. Me: (thinking I ended the call) Me: F********k!!!!!! Boss: I’m still in the line
72. Black Saturday Weekly process was ran 06:00 06:25 06:43 User couldn’t pay for breakfast 06:34 Users reported bug Manager woke me up
73. Lots of duplicated charges
74. Once a week
75. Once a week
76. Once a week
77. Once a week
78. Refunded charges
79. Once a week
80. Reversed transfers
81. Once a week 7:28am
82. Black Saturday Weekly process was ran 06:00 06:25 06:43 User couldn’t pay for breakfast 06:34 Users reported bug Manager woke me up Problem contained 07:28
85. Passenger: UserID: 9 Driver: User ID: 100 $10.00
86. Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 Passenger: UserID: 9 Driver: User ID: 100 $10.00 0 0 0
87. Once a week
88. Once a week
89. Once a week
90. Wrote tests, a fix and deployed it 10:50pm
91. Black Saturday Weekly process was ran 06:00 06:25 06:43 User couldn’t pay for breakfast 06:34 Users reported bug Manager woke me up 22:50 Deployed a fix to production Problem contained 07:28
93. Black Saturday 06:25 06:34 Users reported bug 22:55 Started looking for a new job Problem contained 07:28 Weekly process was ran 06:00 06:25 06:43 User couldn’t pay for breakfast 06:34 Users reported bug Manager woke me up 22:50 Deployed a fix to production Problem contained 07:28 Started looking for a new job
94. Black Saturday Weekly process was ran 06:00 06:25 06:43 User couldn’t pay for breakfast 06:34 Users reported bug Manager woke me up 22:50 Deployed a fix to production Problem contained 07:28
95. Thousands of users affected by the bug Users were charged up-to 200 times A single user was charged over $5k Maxed out credit cards. Emptied bank accounts
96. Refunds take up- to 5 business days
97. Reached out to users to offer an expedited reimbursement option
98. Why?
99. Embarrassing things happen
100. Tests won’t save you
101. Code review won’t save you
102. QA won’t save you
103. Software is built by humans
104. We need to make admiting mistakes easy
105. Trust that we won’t be judged
106. Make sure you understand what happened
107. Move slooooow
108. Document the problem
109. Document the fix
110. Document the lesson learnt
111. Don’t git blame
112. You are not your failures
113. No one will care about the bugs you create…
114. When you die
115. It’s all temporary
116. #RubyConf
117. We are hiring
118. The end @sebasoga