How Square Stays Up

I talk I gave on the tools and processes Square uses to stay stable and available

Transcript

1. How Square Stays Up Tools and Processes Square Uses to Maintain Stability and Availability
2. @pui_ling Erica Kwan
3. 1 2 3 4 Developing Deploying Monitoring On-calling
4. Developing 1
5. We pair program (sometimes)
6. We solo, then get a code review (other times)
7. Why?
8. PCI Compliance Read all about it: http://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard
9. It is also good practice
10. git checkout -b topic-branch do work* git checkout master git merge --no-ff topic-branch
11. A clean commit history helps
12. A super good git workflow: http://sandofsky.com/blog/git-workflow.html
13. git rebase --interactive
14. git rebase protip: config rebase.autosquash = true
15. git commit -m “squash! Monkeys”
16. pick 8374d8e Monkeys squash 8374d8e squash! Monkeys pick 259a7e6 Better monkeys
17. Deploying 2
18. We deploy lots
19. but there are processes around deploys
20. Some history
21. We do canary deploys
23. Our full deploys do rolling restarts
24. And automatically run integration tests
25. Monitoring 3
26. We use common monitoring tools
27. We have application level checks
28. We have custom metrics dashboards
29. Graphite (whisper) + Cubism.js http://square.github.com/cubism/ http://d3js.org/ More info:
30. Horizon Graph http://vis.berkeley.edu/papers/horizon/
32. On-Calling 4
33. Engineers are responsible for their work
34. Ad-hoc at first
35. First real on-call rotations were simple
36. Original escalation path:
37. Engineer 1
38. Engineer 2
39. @jack
40. General on-call could not be responsible for everything
41. Now, every engineering team has an on-call rotation
42. Process is still evolving
43. Do these 4 things well all the time
44. @pui_ling /pui