Rollback: The Impossible Dream

Roll back doesn’t exist. It’s not real. It’s a fantasy, a dream, a delusion. Any vendor who tells you they have a roll back capability is lying to you. And lying to you in a downright dangerous way that will come back to haunt you at 4am in a war room when someone says:

“We can’t fix this. Let’s roll back the deployment.”

This talk is designed to explain and demonstrate to Operations staff:

Why roll back is a fantasy and explained with a dash of Werner Heisenberg
Why it is dangerous and how you can recognize when you’re about to get trapped
How you can avoid falling into that trap of considering it an appropriate compensating control.
It’ll also explain what you can actually do operationally instead of “rolling back”. This will cover other alternative compensating controls that can help you get running again and resolve your outage whilst still allowing you to find root cause.

Transcript

1. Rollback
2. About Me VP Technical Operations at Puppet Labs Puppet guy Ruby guy Talks funny
3. A show of hands
4. Who thinks they know what rollback is?
5. Last set of hands
6. YMMV
10. Fact or Fiction?
11. Accept certain constraints
13. Constraint #2
14. Constraint #3
15. Constraint #4
16. A Philosophical Digression
17. If  I  know  where  I  am   I  don’t  know  how  I  got  there   If  I  know  how  I  got  there   I  don’t  know  where  I  am      
18. Very few “systems” are truly deterministic
19. A Mathematical Digression
20. On system rollback and totalised fields
21. So what’s wrong with rollback?
22. Risk
23. Learning from mistakes
24. Complex systems are
25. Human error
27. What is the problem YOU are trying to solve?
28. So how can we mitigate Rollback shortcomings?
31. Increase  Resilience    
32. Opera8onal   Intelligence  
33. A  li:le  bit  of  DevOps  in   every  byte…  
34. Small, iterative changes
35. Accept that failure happens
36. “We can’t test that? Okay we can roll it back if it breaks…”
37. Assump8on  is  the   mother  of  all  fuckups*  
38. “But  the  system  can’t   be  {run|upgraded| deployed}  like  that   because…”
39. Conclusions
40. Rollback is possible but not probable
41. If you have to have “rollback” accept constraints
42. You can mitigate the need for it
43. Thank you! Questions/Insults? jamtur01 @ github kartar @ twitter jamesturnbull on freenode james @ puppetlabs.com