$30 off During Our Annual Pro Sale. View Details »

Taming the Kraken - How Operations enables developer productivity

Nathen Harvey
April 24, 2012

Taming the Kraken - How Operations enables developer productivity

Are you having trouble launching new features because of friction between development and operations? At CustomInk, we've reduced this friction by making changes to our teams, processes, and tools. Come find out what we've been up to and learn how you can implement similar changes in your own environment.

There's always a bit of tension when getting features from idea to production. In this talk, we'll look at some of the changes CustomInk has made to reduce this friction and keep the new features coming. Gone are the days of bi-monthly deploys, office pools dedicated to guessing when this deploy will be rolled back, and the ceremony surrounding the deploy-rollback-fix-deploy cycle. Today, ideas flow from product managers to developers to production with ease thanks to a number of changes that we've made to our teams, processes and tools.

Nathen Harvey

April 24, 2012


  1. Taming the Kraken How operations enables developer productivity Nathen Harvey

    CustomInk @nathenharvey @nathenharvey
  2. CustomInk @nathenharvey

  3. CustomInk @nathenharvey

  4. CustomInk @nathenharvey

  5. CustomInk @nathenharvey

  6. CustomInk Founded in 1999 40+ million requests to our app

    servers each week Over 1 million customers @nathenharvey
  7. How it used to be One team working on a

    few applications simultaneously Two-week sprints Operations team deploys the code at the end of each sprint @nathenharvey
  8. It worked relatively well except... There were often merge conflicts

    which lead to... merge conflicts Small changes sat for days waiting to be deployed @nathenharvey
  9. It worked relatively well except... Deploys were often rolled back

    because of an issue with one part of the new code Our team grew @nathenharvey
  10. You might have a problem if... A deploy is like

    the High School Prom @nathenharvey
  11. Deploy is like Prom if... Happens infrequently @nathenharvey

  12. Deploy is like Prom if... The build-up is bigger than

    the result @nathenharvey
  13. Deploy is like Prom if... Lots of ceremony @nathenharvey

  14. Other problem indicators... "All hands on deck" @nathenharvey

  15. Other problem indicators... Office pools for when cap deploy:rollback will

    happen @nathenharvey
  16. Other problem indicators... You've considered ordering custom t-shirts for the

    team to celebrate the deploy @nathenharvey
  17. How can you fix this? @nathenharvey

  18. Ship it! Deploy small, independent changes Deploy when change is

    ready @nathenharvey
  19. Change the Process Redefine "done" Move from sprints to Kanban

  20. Change the Process Emphasize minimizing work in process Product managers

    help prioritize the work to be done @nathenharvey
  21. Disposable Servers Staging and verifying many small changes in one

    or two staging environments doesn't work Each topic branch must have its own staging environment @nathenharvey
  22. Disposable Servers Short-lived branches need short-lived staging environments Only feasible

    way is via automation @nathenharvey
  23. Infrastructure as Code Enable the reconstruction of the business from

    nothing but a source code repository an application data backup and bare metal resources -Jesse Robins, Opscode @nathenharvey
  24. Infrastructure as Code Custom scripts Custom scripts + Capistrano CFEngine,

    Puppet Chef @nathenharvey
  25. Chef First, come up with your policy / specification Abstract

    the resources in your spec Write recipes Package recipes in cookbooks Apply recipes to nodes @nathenharvey
  26. Getting better... Change the process Automation for disposable servers @nathenharvey

  27. Development Environments Vagrant @nathenharvey

  28. Let the Vagrants gather! Hobo Jungle One git repository with

    a Vagrantfile for launching each application @nathenharvey
  29. Testing every topic branch "master" is ALWAYS deployable We use

    Jenkins to test every branch Custom Capistrano scripts make it easy! cap jenkins:create cap jenkins:build cap jenkins:status cap jenkins:console cap jenkins:delete @nathenharvey
  30. Greenscreen Big Visible Chart (BVC) to monitor our builds Sinatra

    app with simple configuration @nathenharvey
  31. Green Screen @nathenharvey

  32. Green Screen @nathenharvey

  33. Getting better... Change the process Automation for disposable servers Easy

    development environments Testing every branch @nathenharvey
  34. Deploying the Apps Wait, how many apps does it take

    to make a custom t-shirt? @nathenharvey
  35. CustomInk Apps @nathenharvey

  36. Deploying the Apps Most of the apps are Rails Deployed

    with Capistrano @nathenharvey
  37. Deploying the Apps Lots of identical deploy logic in each

    application Inconsistent deploy behavior Need a way to streamline this @nathenharvey
  38. Caphub Centralized Capistrano for deploying multiple apps cap apps:ecommerce:production deploy

    cap apps:fulfillment:production deploy @nathenharvey
  39. Deploying Merge Build Deploy Verify Get out of the way

  40. Stay on "topic" Use the "Topic" of our Campfire room

    to sequence deploys Product manager help with sequencing, when it's important @nathenharvey
  41. Announcing the deploy Still some ceremony here Automate the ceremony

    Changelog and compare URL posted to Campfire Blog updated Email sent @nathenharvey
  42. Getting better... Change the process Automation for disposable servers Easy

    development environments Testing every branch Stay on "topic" Automate the ceremony @nathenharvey
  43. What's changed? Minimized work in process Shorter cycle times Fewer

    integration bugs @nathenharvey
  44. What's changed? People get agitated if / when deploys slow

    down Deploy one thing at a time Everyone deploys Rollbacks are now the exception @nathenharvey
  45. How can YOU get started? Look at the process Look

    at the culture Automate, automate, automate Enable everyone Use the right tools! @nathenharvey
  46. The right tools @nathenharvey

  47. The right tools Culture Conversation Collaboration @nathenharvey

  48. Other tools that we use git / GitHub - github.com

    Vagrant - vagrantup.com Chef - opscode.com @nathenharvey
  49. Other tools that we use Jenkins - jenkins-ci.org Greenscreen -

    CustomInk's GitHub Campfire - 37signals @nathenharvey
  50. Other tools that we use Capistrano - GitHub Caphub -

    Railsware's GitHub @nathenharvey
  51. Thanks! @nathenharvey @nathenharvey