Taming the Kraken - How Operations enables developer productivity

Are you having trouble launching new features because of friction between development and operations? At CustomInk, we've reduced this friction by making changes to our teams, processes, and tools. Come find out what we've been up to and learn how you can implement similar changes in your own environment.

There's always a bit of tension when getting features from idea to production. In this talk, we'll look at some of the changes CustomInk has made to reduce this friction and keep the new features coming. Gone are the days of bi-monthly deploys, office pools dedicated to guessing when this deploy will be rolled back, and the ceremony surrounding the deploy-rollback-fix-deploy cycle. Today, ideas flow from product managers to developers to production with ease thanks to a number of changes that we've made to our teams, processes and tools.

Transcript

1. Taming the Kraken How operations enables developer productivity Nathen Harvey CustomInk @nathenharvey @nathenharvey
2. CustomInk @nathenharvey
3. CustomInk @nathenharvey
4. CustomInk @nathenharvey
5. CustomInk @nathenharvey
6. CustomInk Founded in 1999 40+ million requests to our app servers each week Over 1 million customers @nathenharvey
7. How it used to be One team working on a few applications simultaneously Two-week sprints Operations team deploys the code at the end of each sprint @nathenharvey
8. It worked relatively well except... There were often merge conflicts which lead to... merge conflicts Small changes sat for days waiting to be deployed @nathenharvey
9. It worked relatively well except... Deploys were often rolled back because of an issue with one part of the new code Our team grew @nathenharvey
10. You might have a problem if... A deploy is like the High School Prom @nathenharvey
11. Deploy is like Prom if... Happens infrequently @nathenharvey
12. Deploy is like Prom if... The build-up is bigger than the result @nathenharvey
13. Deploy is like Prom if... Lots of ceremony @nathenharvey
14. Other problem indicators... "All hands on deck" @nathenharvey
15. Other problem indicators... Office pools for when cap deploy:rollback will happen @nathenharvey
16. Other problem indicators... You've considered ordering custom t-shirts for the team to celebrate the deploy @nathenharvey
17. How can you fix this? @nathenharvey
18. Ship it! Deploy small, independent changes Deploy when change is ready @nathenharvey
19. Change the Process Redefine "done" Move from sprints to Kanban @nathenharvey
20. Change the Process Emphasize minimizing work in process Product managers help prioritize the work to be done @nathenharvey
21. Disposable Servers Staging and verifying many small changes in one or two staging environments doesn't work Each topic branch must have its own staging environment @nathenharvey
22. Disposable Servers Short-lived branches need short-lived staging environments Only feasible way is via automation @nathenharvey
23. Infrastructure as Code Enable the reconstruction of the business from nothing but a source code repository an application data backup and bare metal resources -Jesse Robins, Opscode @nathenharvey
24. Infrastructure as Code Custom scripts Custom scripts + Capistrano CFEngine, Puppet Chef @nathenharvey
25. Chef First, come up with your policy / specification Abstract the resources in your spec Write recipes Package recipes in cookbooks Apply recipes to nodes @nathenharvey
26. Getting better... Change the process Automation for disposable servers @nathenharvey
27. Development Environments Vagrant @nathenharvey
28. Let the Vagrants gather! Hobo Jungle One git repository with a Vagrantfile for launching each application @nathenharvey
29. Testing every topic branch "master" is ALWAYS deployable We use Jenkins to test every branch Custom Capistrano scripts make it easy! cap jenkins:create cap jenkins:build cap jenkins:status cap jenkins:console cap jenkins:delete @nathenharvey
30. Greenscreen Big Visible Chart (BVC) to monitor our builds Sinatra app with simple configuration @nathenharvey
31. Green Screen @nathenharvey
32. Green Screen @nathenharvey
33. Getting better... Change the process Automation for disposable servers Easy development environments Testing every branch @nathenharvey
34. Deploying the Apps Wait, how many apps does it take to make a custom t-shirt? @nathenharvey
35. CustomInk Apps @nathenharvey
36. Deploying the Apps Most of the apps are Rails Deployed with Capistrano @nathenharvey
37. Deploying the Apps Lots of identical deploy logic in each application Inconsistent deploy behavior Need a way to streamline this @nathenharvey
38. Caphub Centralized Capistrano for deploying multiple apps cap apps:ecommerce:production deploy cap apps:fulfillment:production deploy @nathenharvey
39. Deploying Merge Build Deploy Verify Get out of the way @nathenharvey
40. Stay on "topic" Use the "Topic" of our Campfire room to sequence deploys Product manager help with sequencing, when it's important @nathenharvey
41. Announcing the deploy Still some ceremony here Automate the ceremony Changelog and compare URL posted to Campfire Blog updated Email sent @nathenharvey
42. Getting better... Change the process Automation for disposable servers Easy development environments Testing every branch Stay on "topic" Automate the ceremony @nathenharvey
43. What's changed? Minimized work in process Shorter cycle times Fewer integration bugs @nathenharvey
44. What's changed? People get agitated if / when deploys slow down Deploy one thing at a time Everyone deploys Rollbacks are now the exception @nathenharvey
45. How can YOU get started? Look at the process Look at the culture Automate, automate, automate Enable everyone Use the right tools! @nathenharvey
46. The right tools @nathenharvey
47. The right tools Culture Conversation Collaboration @nathenharvey
48. Other tools that we use git / GitHub - github.com Vagrant - vagrantup.com Chef - opscode.com @nathenharvey
49. Other tools that we use Jenkins - jenkins-ci.org Greenscreen - CustomInk's GitHub Campfire - 37signals @nathenharvey
50. Other tools that we use Capistrano - GitHub Caphub - Railsware's GitHub @nathenharvey
51. Thanks! @nathenharvey @nathenharvey