Share
Embed
Slide 1
Slide 1 text
Taming the Kraken
How operations enables developer productivity
Nathen Harvey
CustomInk
@nathenharvey
@nathenharvey
Slide 2
Slide 2 text
CustomInk
@nathenharvey
Slide 3
Slide 3 text
CustomInk
@nathenharvey
Slide 4
Slide 4 text
CustomInk
@nathenharvey
Slide 5
Slide 5 text
CustomInk
@nathenharvey
Slide 6
Slide 6 text
CustomInk
Founded in 1999
40+ million requests to our app servers
each week
Over 1 million customers
@nathenharvey
Slide 7
Slide 7 text
How it used to be
One team working on a few applications
simultaneously
Two-week sprints
Operations team deploys the code at the
end of each sprint
@nathenharvey
Slide 8
Slide 8 text
It worked relatively well
except...
There were often merge conflicts which
lead to... merge conflicts
Small changes sat for days waiting to be
deployed
@nathenharvey
Slide 9
Slide 9 text
It worked relatively well
except...
Deploys were often rolled back because
of an issue with one part of the new code
Our team grew
@nathenharvey
Slide 10
Slide 10 text
You might have a problem
if...
A deploy is like the High School Prom
@nathenharvey
Slide 11
Slide 11 text
Deploy is like Prom if...
Happens infrequently
@nathenharvey
Slide 12
Slide 12 text
Deploy is like Prom if...
The build-up is bigger than the result
@nathenharvey
Slide 13
Slide 13 text
Deploy is like Prom if...
Lots of ceremony
@nathenharvey
Slide 14
Slide 14 text
Other problem indicators...
"All hands on deck"
@nathenharvey
Slide 15
Slide 15 text
Other problem indicators...
Office pools for when cap deploy:rollback will happen
@nathenharvey
Slide 16
Slide 16 text
Other problem indicators...
You've considered ordering custom t-shirts for the team to celebrate
the deploy
@nathenharvey
Slide 17
Slide 17 text
How can you fix this?
@nathenharvey
Slide 18
Slide 18 text
Ship it!
Deploy small, independent changes
Deploy when change is ready
@nathenharvey
Slide 19
Slide 19 text
Change the Process
Redefine "done"
Move from sprints to Kanban
@nathenharvey
Slide 20
Slide 20 text
Change the Process
Emphasize minimizing work in process
Product managers help prioritize the work
to be done
@nathenharvey
Slide 21
Slide 21 text
Disposable Servers
Staging and verifying many small
changes in one or two staging
environments doesn't work
Each topic branch must have its own
staging environment
@nathenharvey
Slide 22
Slide 22 text
Disposable Servers
Short-lived branches need short-lived
staging environments
Only feasible way is via automation
@nathenharvey
Slide 23
Slide 23 text
Infrastructure as Code
Enable the reconstruction of the business
from nothing but
a source code repository
an application data backup
and bare metal resources
-Jesse Robins, Opscode
@nathenharvey
Slide 24
Slide 24 text
Infrastructure as Code
Custom scripts
Custom scripts + Capistrano
CFEngine, Puppet
Chef
@nathenharvey
Slide 25
Slide 25 text
Chef
First, come up with your policy /
specification
Abstract the resources in your spec
Write recipes
Package recipes in cookbooks
Apply recipes to nodes
@nathenharvey
Slide 26
Slide 26 text
Getting better...
Change the process
Automation for disposable servers
@nathenharvey
Slide 27
Slide 27 text
Development Environments
Vagrant
@nathenharvey
Slide 28
Slide 28 text
Let the Vagrants gather!
Hobo Jungle
One git repository with a Vagrantfile for
launching each application
@nathenharvey
Slide 29
Slide 29 text
Testing every topic branch
"master" is ALWAYS deployable
We use Jenkins to test every branch
Custom Capistrano scripts make it easy!
cap jenkins:create
cap jenkins:build
cap jenkins:status
cap jenkins:console
cap jenkins:delete
@nathenharvey
Slide 30
Slide 30 text
Greenscreen
Big Visible Chart (BVC) to monitor our
builds
Sinatra app with simple configuration
@nathenharvey
Slide 31
Slide 31 text
Green Screen
@nathenharvey
Slide 32
Slide 32 text
Green Screen
@nathenharvey
Slide 33
Slide 33 text
Getting better...
Change the process
Automation for disposable servers
Easy development environments
Testing every branch
@nathenharvey
Slide 34
Slide 34 text
Deploying the Apps
Wait, how many apps does it take to
make a custom t-shirt?
@nathenharvey
Slide 35
Slide 35 text
CustomInk Apps
@nathenharvey
Slide 36
Slide 36 text
Deploying the Apps
Most of the apps are Rails
Deployed with Capistrano
@nathenharvey
Slide 37
Slide 37 text
Deploying the Apps
Lots of identical deploy logic in each
application
Inconsistent deploy behavior
Need a way to streamline this
@nathenharvey
Slide 38
Slide 38 text
Caphub
Centralized Capistrano for deploying multiple apps
cap apps:ecommerce:production deploy
cap apps:fulfillment:production deploy
@nathenharvey
Slide 39
Slide 39 text
Deploying
Merge
Build
Deploy
Verify
Get out of the way
@nathenharvey
Slide 40
Slide 40 text
Stay on "topic"
Use the "Topic" of our Campfire room to
sequence deploys
Product manager help with sequencing,
when it's important
@nathenharvey
Slide 41
Slide 41 text
Announcing the deploy
Still some ceremony here
Automate the ceremony
Changelog and compare URL posted
to Campfire
Blog updated
Email sent
@nathenharvey
Slide 42
Slide 42 text
Getting better...
Change the process
Automation for disposable servers
Easy development environments
Testing every branch
Stay on "topic"
Automate the ceremony
@nathenharvey
Slide 43
Slide 43 text
What's changed?
Minimized work in process
Shorter cycle times
Fewer integration bugs
@nathenharvey
Slide 44
Slide 44 text
What's changed?
People get agitated if / when deploys slow
down
Deploy one thing at a time
Everyone deploys
Rollbacks are now the exception
@nathenharvey
Slide 45
Slide 45 text
How can YOU get started?
Look at the process
Look at the culture
Automate, automate, automate
Enable everyone
Use the right tools!
@nathenharvey
Slide 46
Slide 46 text
The right tools
@nathenharvey
Slide 47
Slide 47 text
The right tools
Culture
Conversation
Collaboration
@nathenharvey
Slide 48
Slide 48 text
Other tools that we use
git / GitHub - github.com
Vagrant - vagrantup.com
Chef - opscode.com
@nathenharvey
Slide 49
Slide 49 text
Other tools that we use
Jenkins - jenkins-ci.org
Greenscreen - CustomInk's GitHub
Campfire - 37signals
@nathenharvey
Slide 50
Slide 50 text
Other tools that we use
Capistrano - GitHub
Caphub - Railsware's GitHub
@nathenharvey
Slide 51
Slide 51 text
Thanks!
@nathenharvey @nathenharvey