Automate or Die

Explaining Paperless Post's deploy methodology and how we got there. When I Ship, You Ship, We Ship.

Transcript

1. Automate or die Aaron Quint / DevOpsDays NYC / 1/2013 Friday, January 18, 13
2. It’s me, @AQ HeYO Friday, January 18, 13
3. Friday, January 18, 13
4. • Food Obsessed (CodeFoodBar) Friday, January 18, 13
5. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) Friday, January 18, 13
6. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • BROOKLYNNNNNNNNN (get hype) Friday, January 18, 13
7. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • BROOKLYNNNNNNNNN (get hype) • twitter: @aq Friday, January 18, 13
8. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS (Sammy.js, et al) • BROOKLYNNNNNNNNN (get hype) • twitter: @aq • github: quirkey Friday, January 18, 13
9. CTO of Friday, January 18, 13
10. Chief Taco Officer CTO of Friday, January 18, 13
11. www.paperlesspost.com Friday, January 18, 13
12. Friday, January 18, 13
13. • Most individual user-generated cards sent in a single week: ~2 million • Registered user growth over December 2011: 211% (3.1x) • Revenue growth over December 2011: 135% (2.3x) • Number of commits to main app git repo: 44,545 • Number of commits to chef git repo: 4,447 • Peak traffic served: ~300 Mbps • Number of production deploys across applications: 1,631 • Number of staging deploys across applications: 10,600 • Unique animated gifs posted in Campfire: 3,208 Friday, January 18, 13
14. DA DIP Friday, January 18, 13
15. DA SHIP Friday, January 18, 13
16. WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13
17. It’s about being efficient Automation is not about being lazy Friday, January 18, 13
18. Developers aren’t slow, they’re ... Friday, January 18, 13
19. Distracted Friday, January 18, 13
20. blocked Friday, January 18, 13
21. churning Friday, January 18, 13
22. Complexity = Not just large, but distributed applications get more complex over time Friday, January 18, 13
23. how to allow for growing complexity while not slowing down Friday, January 18, 13
24. - A DevOps Oath I, [YOU], Do solemnly swear to help developers solve their problems, quickly and without [too many] yak- shaves and make their lives easier, so they can ship code, so that the company can improve, so that we all can be successful. I will do what it takes to avoid repeated tasks, help unblock the blocked, and generally make things work better. Friday, January 18, 13
25. You want Friday, January 18, 13
26. Friday, January 18, 13
27. • Notifications Friday, January 18, 13
28. • Notifications • One click deploys Friday, January 18, 13
29. • Notifications • One click deploys • Continuous integration Friday, January 18, 13
30. • Notifications • One click deploys • Continuous integration • Distributed/fast deploys Friday, January 18, 13
31. • Notifications • One click deploys • Continuous integration • Distributed/fast deploys • Zero config Dev environment Friday, January 18, 13
32. • Notifications • One click deploys • Continuous integration • Distributed/fast deploys • Zero config Dev environment • Developer Metrics Friday, January 18, 13
33. • Notifications • One click deploys • Continuous integration • Distributed/fast deploys • Zero config Dev environment • Developer Metrics • Deploy History/Tracking Friday, January 18, 13
34. • Notifications • One click deploys • Continuous integration • Distributed/fast deploys • Zero config Dev environment • Developer Metrics • Deploy History/Tracking • Integrated Troubleshooting Friday, January 18, 13
35. You HAVE Friday, January 18, 13
36. A: Start Small Q: But how to start? Friday, January 18, 13
37. It’s daunting, and incremental improvements reap rewards DON’t DO IT ALL AT ONCE Friday, January 18, 13
38. Distill into a perfect complainy-developer cocktail Start with the whining Friday, January 18, 13
39. Deploys are too complicated, so I end up not doing them ಠ_ಠ Friday, January 18, 13
40. $ git checkout staging $ git pull root staging $ git merge my_branch $ git push root staging $ cap staging deploy:long Friday, January 18, 13
41. WE COULD • Build a web ui to handle deploys • Make a simple script that wraps the common commands • Ignore them, haters gonna hate Friday, January 18, 13
42. Friday, January 18, 13
43. ಠ_ಠ Friday, January 18, 13
44. I keep losing my changes or forget to commit ಠ_ಠ Friday, January 18, 13
45. I keep losing my changes or forget to commit ಠ_ಠ ಠ_ಠ Friday, January 18, 13
46. I keep losing my changes or forget to commit ಠ_ಠ I want notifications when someone deploys ಠ_ಠ Friday, January 18, 13
47. I keep losing my changes or forget to commit ಠ_ಠ I want notifications when someone deploys ಠ_ಠ ಠ_ಠ Friday, January 18, 13
48. I keep losing my changes or forget to commit ಠ_ಠ I want notifications when someone deploys I want to deploy multiple branches at once ಠ_ಠ ಠ_ಠ Friday, January 18, 13
49. For developers it’s a familiar interface Lots of things can be solved with command line tools Friday, January 18, 13
50. Devtools was born Friday, January 18, 13
51. Design an aPI around your common tasks Friday, January 18, 13
52. $ pp rails deploy earth quirkey/branch1 mrb/f2 # pp <app> deploy <environment> [branches, ...] Friday, January 18, 13
53. • Does the git flow (checkout, reset, merge, push) • Handles conflicts and stashing changes • Runs the deploy • Sends notifications to email list/ campfire rooms Friday, January 18, 13
54. $ pp boot renderer restart # pp boot <service> [stop|start|restart|status] Friday, January 18, 13
55. • Handles booting local services • Has all powerful boot/restart commands • Monitors service status Friday, January 18, 13
56. • Ruby gem (on internal gemserver) • Self-updating (pp devtools update) • Built on top of Thor • Made to be extensible (easy to add apps, tasks, recipes) DEvTOOLS Friday, January 18, 13
57. WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13
58. Friday, January 18, 13
59. ಠ_ಠ Friday, January 18, 13
60. I need to deploy but I don’t have ssh credentials ಠ_ಠ Friday, January 18, 13
61. I need to deploy but I don’t have ssh credentials ಠ_ಠ ಠ_ಠ Friday, January 18, 13
62. I need to deploy but I don’t have ssh credentials ಠ_ಠ It takes too long to run the test builds ಠ_ಠ Friday, January 18, 13
63. I need to deploy but I don’t have ssh credentials ಠ_ಠ It takes too long to run the test builds ಠ_ಠ ಠ_ಠ Friday, January 18, 13
64. I need to deploy but I don’t have ssh credentials ಠ_ಠ It takes too long to run the test builds Are we deploying? What step are we at? ಠ_ಠ ಠ_ಠ Friday, January 18, 13
65. ; http://jenkins-ci.org Jenkins! Friday, January 18, 13
66. It just needs to be told what to deploy (It has an HTTP API) Jenkins can just run devtools Friday, January 18, 13
67. Text # On your machine $ pp rails deploy earth quirkey/branch1 mrb/f2 # Does git merging, sends request to jenkins # On Jenkins, in a `deploy` job $ pp $APP deploy $ENV $BRANCHES --local # Skips the git stuff, just does the actual # deploy and sends the notifications Friday, January 18, 13
68. Jenkins • Global visibility (changed the game) • No fear of losing network connectivity during a deploy • Single point of orchestration • Being able to see the deploys and builds made the team want to make them faster and better Friday, January 18, 13
69. WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13
70. Friday, January 18, 13
71. ಠ_ಠ Friday, January 18, 13
72. What branches are deployed to staging? ಠ_ಠ Friday, January 18, 13
73. What branches are deployed to staging? ಠ_ಠ ಠ_ಠ Friday, January 18, 13
74. What branches are deployed to staging? ಠ_ಠ What just got deployed to production? ಠ_ಠ Friday, January 18, 13
75. What branches are deployed to staging? ಠ_ಠ What just got deployed to production? ಠ_ಠ ಠ_ಠ Friday, January 18, 13
76. What branches are deployed to staging? ಠ_ಠ What just got deployed to production? Johnny just said he pushed something, but I can’t find it ಠ_ಠ ಠ_ಠ Friday, January 18, 13
77. Now we just need to collect the data We unified the entry points Friday, January 18, 13
78. A web UI and database for devtools Enter: PUSH PARTY Friday, January 18, 13
79. Text Friday, January 18, 13
80. And separated them from the CLI We extracted the “tools” of Devtools Friday, January 18, 13
81. > require 'paperless-devtools' > app = PaperlessDevtools.app :rails > app.tools(:git).current_branch #=> "master" > app.tools(:jenkins).start_build("master") #=> send build to jenkins Friday, January 18, 13
82. Rails, Postgres, Devtools, Compass Then we built a WebApp around it Friday, January 18, 13
83. Friday, January 18, 13
84. devtools makes api calls to pushparty Friday, January 18, 13
85. devtools makes api calls to pushparty which records deploys in postgres Friday, January 18, 13
86. devtools makes api calls to pushparty which records deploys in postgres then uses devtools to send them to jenkins Friday, January 18, 13
87. devtools makes api calls to pushparty which records deploys in postgres then uses devtools to send them to jenkins which uses devtools to send progress and status to pushparty Friday, January 18, 13
88. devtools makes api calls to pushparty which records deploys in postgres then uses devtools to send them to jenkins which uses devtools to send progress and status to pushparty which uses devtools to announce the completed deploy in campfire Friday, January 18, 13
89. devtools makes api calls to pushparty which records deploys in postgres then uses devtools to send them to jenkins which uses devtools to send progress and status to pushparty which uses devtools to announce the completed deploy in campfire Friday, January 18, 13
90. Extremely easy to see what was merged/ deployed, in what state and at what time Now we have DATA Friday, January 18, 13
91. BRANCHES FLOW Friday, January 18, 13
92. WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13
93. Working with chef [with multiple team members working on multiple projects in multiple staging environments] is a pain in the butt. ಠ_ಠ Friday, January 18, 13
94. $ pp chef deploy earth johnnydtan/graphite_cluster # git merging # cookbook testing # cookbook uploading # environment munging Friday, January 18, 13
95. http://www.meetup.com/Chef-NYC/ Come to #CHEF NYC Friday, January 18, 13
96. WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13
97. We have all this data in graphite, chef, sensu, pushparty, but when theres an issue it’s hard to coalate the problems ಠ_ಠ Friday, January 18, 13
98. Devtools wrappers for different tools, pulled into a single web ui HOTPOT Friday, January 18, 13
99. Friday, January 18, 13
100. Awesome things I don’t have time to talk about Friday, January 18, 13
101. Awesome things I don’t have time to talk about • pp chef create_node Friday, January 18, 13
102. Awesome things I don’t have time to talk about • pp chef create_node • pp vagrant setup Friday, January 18, 13
103. Awesome things I don’t have time to talk about • pp chef create_node • pp vagrant setup • pushparty bashrc’s Friday, January 18, 13
104. Awesome things I don’t have time to talk about • pp chef create_node • pp vagrant setup • pushparty bashrc’s • graphite integration Friday, January 18, 13
105. Awesome things I don’t have time to talk about • pp chef create_node • pp vagrant setup • pushparty bashrc’s • graphite integration • deploy build time graphs Friday, January 18, 13
106. Awesome things I don’t have time to talk about • pp chef create_node • pp vagrant setup • pushparty bashrc’s • graphite integration • deploy build time graphs • more every day Friday, January 18, 13
107. Things to remember Friday, January 18, 13
108. Build for your team for right now, not for every team at any size Don’t expect to build the perfect system Friday, January 18, 13
109. Not the other way around Start with a workflow then tool around it Friday, January 18, 13
110. Then add the cool shit that they didn’t know they needed Build what your team needs first Friday, January 18, 13
111. WHEN I SHIP YOU SHIP WE SHIP Friday, January 18, 13
112. github.com/quirkey twitter.com/aq quirkey.com paperlesspost.com THANKS! Friday, January 18, 13