Moving Fast (Without Crashing)

Pushpay operates in the payments industry so we are very risk adverse, yet we are able to deploy to production up to a dozen times per day.

This talk will discuss the culture that enables this, and the tools and processes that have been built over time to support our fast pace. We will show how we can maintain a high quality codebase that has minimal technical debt, while simultaneously growing the team by over 800% in the last 2 years. Finally we will explain how our culture helps support us in those rare instances when something does go wrong.

https://www.devopsdays.org/events/2017-auckland/

Transcript

1. MOVING FAST (WITHOUT CRASHING) Marcus Bristol @mightymuke
3. TEXT
4. Just Culture http://sidneydekker.com/just-culture/
5. A just culture is a culture of… trust learning accountability
6. balance safety and accountability
7. A culture that allows the boss to hear bad news
8. Retributive What rule is broken? Who did it? How bad was the breach, and
 what should the consequences be? Who gets to decide this?
9. Retributive
10. Retributive Meet hurt with more hurt
11. Restorative Who is hurt? What do they need? Who's obligation is it to meet that need? How do you involve the community in
 this conversation?
12. Restorative
13. Restorative Meet hurt with healing
14. Blameless Postmortem Engineers give a detailed account of… what actions they took at what time what effects they observed expectations they had assumptions they had made their understanding of timeline of events
15. Blameless Postmortem they can give this detailed account without fear of punishment or retribution
16. Blameless Postmortem When do we do one? Opportunity to learn Impact to production Near miss
17. Blameless Postmortem Written rather than spoken
19. Timeline
21. Impact Assessment
23. Discussion
25. Mitigations
27. Our Process Feature Flags
29. Our Process Feature Flags Science Lots of small PR’s Code Review WOMM Automated Testing Pollination Bots
42. Etsy - Blameless Postmortem https://codeascraft.com/2012/05/22/blameless-postmortems/ Marcus Bristol (@mightymuke) https://speakerdeck.com/mightymuke/moving-fast-without-crashing Spotify Engineering Culture https://www.youtube.com/watch?v=dtzPtFi8jiQ https://www.youtube.com/watch?v=X3rGdmoTjDc Pushpay - Blameless Postmortem https://channel9.msdn.com/Shows/DevOps-Dimension/6--Blameless-Postmortems-with-PushPay Just Culture http://sidneydekker.com/just-culture/