$30 off During Our Annual Pro Sale. View Details »

Taming the Kraken - How Operations enables developer productivity

Nathen Harvey
April 24, 2012
6.1k

Taming the Kraken - How Operations enables developer productivity

Are you having trouble launching new features because of friction between development and operations? At CustomInk, we've reduced this friction by making changes to our teams, processes, and tools. Come find out what we've been up to and learn how you can implement similar changes in your own environment.

There's always a bit of tension when getting features from idea to production. In this talk, we'll look at some of the changes CustomInk has made to reduce this friction and keep the new features coming. Gone are the days of bi-monthly deploys, office pools dedicated to guessing when this deploy will be rolled back, and the ceremony surrounding the deploy-rollback-fix-deploy cycle. Today, ideas flow from product managers to developers to production with ease thanks to a number of changes that we've made to our teams, processes and tools.

Nathen Harvey

April 24, 2012
Tweet

Transcript

  1. Taming the Kraken
    How operations enables developer productivity
    Nathen Harvey
    CustomInk
    @nathenharvey
    @nathenharvey

    View Slide

  2. CustomInk
    @nathenharvey

    View Slide

  3. CustomInk
    @nathenharvey

    View Slide

  4. CustomInk
    @nathenharvey

    View Slide

  5. CustomInk
    @nathenharvey

    View Slide

  6. CustomInk
    Founded in 1999
    40+ million requests to our app servers
    each week
    Over 1 million customers
    @nathenharvey

    View Slide

  7. How it used to be
    One team working on a few applications
    simultaneously
    Two-week sprints
    Operations team deploys the code at the
    end of each sprint
    @nathenharvey

    View Slide

  8. It worked relatively well
    except...
    There were often merge conflicts which
    lead to... merge conflicts
    Small changes sat for days waiting to be
    deployed
    @nathenharvey

    View Slide

  9. It worked relatively well
    except...
    Deploys were often rolled back because
    of an issue with one part of the new code
    Our team grew
    @nathenharvey

    View Slide

  10. You might have a problem
    if...
    A deploy is like the High School Prom
    @nathenharvey

    View Slide

  11. Deploy is like Prom if...
    Happens infrequently
    @nathenharvey

    View Slide

  12. Deploy is like Prom if...
    The build-up is bigger than the result
    @nathenharvey

    View Slide

  13. Deploy is like Prom if...
    Lots of ceremony
    @nathenharvey

    View Slide

  14. Other problem indicators...
    "All hands on deck"
    @nathenharvey

    View Slide

  15. Other problem indicators...
    Office pools for when cap deploy:rollback will happen
    @nathenharvey

    View Slide

  16. Other problem indicators...
    You've considered ordering custom t-shirts for the team to celebrate
    the deploy
    @nathenharvey

    View Slide

  17. How can you fix this?
    @nathenharvey

    View Slide

  18. Ship it!
    Deploy small, independent changes
    Deploy when change is ready
    @nathenharvey

    View Slide

  19. Change the Process
    Redefine "done"
    Move from sprints to Kanban
    @nathenharvey

    View Slide

  20. Change the Process
    Emphasize minimizing work in process
    Product managers help prioritize the work
    to be done
    @nathenharvey

    View Slide

  21. Disposable Servers
    Staging and verifying many small
    changes in one or two staging
    environments doesn't work
    Each topic branch must have its own
    staging environment
    @nathenharvey

    View Slide

  22. Disposable Servers
    Short-lived branches need short-lived
    staging environments
    Only feasible way is via automation
    @nathenharvey

    View Slide

  23. Infrastructure as Code
    Enable the reconstruction of the business
    from nothing but
    a source code repository
    an application data backup
    and bare metal resources
    -Jesse Robins, Opscode
    @nathenharvey

    View Slide

  24. Infrastructure as Code
    Custom scripts
    Custom scripts + Capistrano
    CFEngine, Puppet
    Chef
    @nathenharvey

    View Slide

  25. Chef
    First, come up with your policy /
    specification
    Abstract the resources in your spec
    Write recipes
    Package recipes in cookbooks
    Apply recipes to nodes
    @nathenharvey

    View Slide

  26. Getting better...
    Change the process
    Automation for disposable servers
    @nathenharvey

    View Slide

  27. Development Environments
    Vagrant
    @nathenharvey

    View Slide

  28. Let the Vagrants gather!
    Hobo Jungle
    One git repository with a Vagrantfile for
    launching each application
    @nathenharvey

    View Slide

  29. Testing every topic branch
    "master" is ALWAYS deployable
    We use Jenkins to test every branch
    Custom Capistrano scripts make it easy!
    cap jenkins:create
    cap jenkins:build
    cap jenkins:status
    cap jenkins:console
    cap jenkins:delete
    @nathenharvey

    View Slide

  30. Greenscreen
    Big Visible Chart (BVC) to monitor our
    builds
    Sinatra app with simple configuration
    @nathenharvey

    View Slide

  31. Green Screen
    @nathenharvey

    View Slide

  32. Green Screen
    @nathenharvey

    View Slide

  33. Getting better...
    Change the process
    Automation for disposable servers
    Easy development environments
    Testing every branch
    @nathenharvey

    View Slide

  34. Deploying the Apps
    Wait, how many apps does it take to
    make a custom t-shirt?
    @nathenharvey

    View Slide

  35. CustomInk Apps
    @nathenharvey

    View Slide

  36. Deploying the Apps
    Most of the apps are Rails
    Deployed with Capistrano
    @nathenharvey

    View Slide

  37. Deploying the Apps
    Lots of identical deploy logic in each
    application
    Inconsistent deploy behavior
    Need a way to streamline this
    @nathenharvey

    View Slide

  38. Caphub
    Centralized Capistrano for deploying multiple apps
    cap apps:ecommerce:production deploy
    cap apps:fulfillment:production deploy
    @nathenharvey

    View Slide

  39. Deploying
    Merge
    Build
    Deploy
    Verify
    Get out of the way
    @nathenharvey

    View Slide

  40. Stay on "topic"
    Use the "Topic" of our Campfire room to
    sequence deploys
    Product manager help with sequencing,
    when it's important
    @nathenharvey

    View Slide

  41. Announcing the deploy
    Still some ceremony here
    Automate the ceremony
    Changelog and compare URL posted
    to Campfire
    Blog updated
    Email sent
    @nathenharvey

    View Slide

  42. Getting better...
    Change the process
    Automation for disposable servers
    Easy development environments
    Testing every branch
    Stay on "topic"
    Automate the ceremony
    @nathenharvey

    View Slide

  43. What's changed?
    Minimized work in process
    Shorter cycle times
    Fewer integration bugs
    @nathenharvey

    View Slide

  44. What's changed?
    People get agitated if / when deploys slow
    down
    Deploy one thing at a time
    Everyone deploys
    Rollbacks are now the exception
    @nathenharvey

    View Slide

  45. How can YOU get started?
    Look at the process
    Look at the culture
    Automate, automate, automate
    Enable everyone
    Use the right tools!
    @nathenharvey

    View Slide

  46. The right tools
    @nathenharvey

    View Slide

  47. The right tools
    Culture
    Conversation
    Collaboration
    @nathenharvey

    View Slide

  48. Other tools that we use
    git / GitHub - github.com
    Vagrant - vagrantup.com
    Chef - opscode.com
    @nathenharvey

    View Slide

  49. Other tools that we use
    Jenkins - jenkins-ci.org
    Greenscreen - CustomInk's GitHub
    Campfire - 37signals
    @nathenharvey

    View Slide

  50. Other tools that we use
    Capistrano - GitHub
    Caphub - Railsware's GitHub
    @nathenharvey

    View Slide

  51. Thanks!
    @nathenharvey @nathenharvey

    View Slide