MARK CROSSFIELD, OCTOBER 2014
“AUTO TRADER: FROM
CONTINUOUS INTEGRATION TO
CONTINUOUS DELIVERY”
Slide 2
Slide 2 text
I’m Mark Crossfield
DEVELOPER, ENGINEER
Seven Years at Auto Trader
Slide 3
Slide 3 text
(definitely not a DevOps team)
I work in our
CONTINUOUS DELIVERY TEAM
Slide 4
Slide 4 text
THIS IS THE HONEST
STORY OF THE LAST YEAR
Slide 5
Slide 5 text
Background
This time last year
Continuous Delivery
Deployment
Monitoring
Self Contained Applications
RPM Builders
Yum Repo Indexing
The Future
CONTENTS
Slide 6
Slide 6 text
But first…
A LITTLE INFORMATION
about Auto Trader
Slide 7
Slide 7 text
Our traffic peaks at
70 MILLION
page views / day
Slide 8
Slide 8 text
We have more stock than ever…
435,000
and that’s just cars!
Slide 9
Slide 9 text
Each month we see
14.5 MILLION
unique users
Slide 10
Slide 10 text
In our First Street offices we have
323
product & technology staff
Slide 11
Slide 11 text
Our continuous build environment reads from
250
code bases
Slide 12
Slide 12 text
Website launched in 1996
Slide 13
Slide 13 text
In June 2013 we completed our
PRINT TO DIGITAL TRANSITION
We achieved continued growth of the business throughout.
Slide 14
Slide 14 text
Over the last five years
WE’VE INVESTED HEAVILY IN
OUR PRODUCTS
but not so much in our infrastructure
Slide 15
Slide 15 text
make up most of our business revenue, not consumers
TRADE RETAILERS
Slide 16
Slide 16 text
We have approximately
12,000
trade customers.
Slide 17
Slide 17 text
Our bespoke website platform serves around
4,500
websites for retail customers.
Slide 18
Slide 18 text
LARGE PORTFOLIO OF RETAIL
MANAGEMENT SOFTWARE
The website really is the tip of the iceberg
Slide 19
Slide 19 text
Of course…
IT IS EASY TO SEE HOW TO
IMPROVE WITH HINDSIGHT
Slide 20
Slide 20 text
Over the last year
WE’VE KEPT THE LIGHTS ON
whilst freeing up staff from manual deployments.
http://www.thoughtworks.com/continuous-integration
Continuous Integration
Continuous Integration (CI) is a development
practice that requires developers to integrate code
into a shared repository several times a day. Each
check-in is then verified by an automated build,
allowing teams to detect problems early.
!
By integrating regularly, you can detect errors
quickly, and locate them more easily.
Slide 45
Slide 45 text
WHAT DOES THAT
MEAN FOR
INFRASTRUCTURE?
Slide 46
Slide 46 text
1. YOUR
CONFIGURATION IS
CODE
Slide 47
Slide 47 text
2. YOU ONLY MAKE
CHANGES BY
CHANGING YOUR CODE
Slide 48
Slide 48 text
3. YOUR CODE IS TESTED
IN A DIFFERENT
ENVIRONMENT BEFORE IT
IS USED
Slide 49
Slide 49 text
CONTINUOUS
DELIVERY
… getting the value out to your customers early and often.
BASH UNOFFICIAL
STRICT MODE
An easy way to make it safer:
set -euo pipefail; IFS=$'\n\t';
http://redsymbol.net/articles/unofficial-bash-strict-mode/
Slide 106
Slide 106 text
EXIT IMMEDIATELY ON
ERROR
set -e
Slide 107
Slide 107 text
EXIT IMMEDIATELY ON
UNDEFINED VARIABLES
set -u
Slide 108
Slide 108 text
EXIT THE PIPE ON FAILURE
—DO NOT MASK ERRORS
set -o pipefail
Slide 109
Slide 109 text
RESTRICT THE INTER-
FIELD SEPARATOR
Produces more sensible looping behaviour.
IFS=$'\n\t'
Slide 110
Slide 110 text
26 SELF CONTAINED APPS
Good progress:
Slide 111
Slide 111 text
No content
Slide 112
Slide 112 text
RPM BUILDERS
Act IV:
Slide 113
Slide 113 text
WE USE RPMS A *LOT*
=
It’s fair to say…
Slide 114
Slide 114 text
VERSIONED, DEPENDENCIES,
FITS IN WITH REDHAT
=
It’s worked well for us
Slide 115
Slide 115 text
BUILDING RPMS IS
FORGETTABLE
Mostly because it is something we automate.
Slide 116
Slide 116 text
This is a spec file
#
Generated
from
bundler-‐1.6.5.gem
by
gem2rpm
-‐*-‐
rpm-‐spec
-‐*-‐
%global
gemname
bundler
!
%global
gemdir
%(ruby
-‐rubygems
-‐e
'puts
Gem::dir'
2>/dev/null)
%global
geminstdir
%{gemdir}/gems/%{gemname}-‐%{version}
%global
rubyabi
1.8
!
Summary:
The
best
way
to
manage
your
application's
dependencies
Name:
rubygem-‐%{gemname}
Version:
1.6.5
Release:
1%{?dist}
Group:
Development/Languages
License:
closed
URL:
http://bundler.io
Source0:
http://rubygems.org/gems/%{gemname}-‐%{version}.gem
Requires:
ruby(abi)
=
%{rubyabi}
Requires:
ruby(rubygems)
>=
1.3.6
Slide 117
Slide 117 text
This is how you call rpmbuild
rpmbuild
-‐v
-‐V
-‐-‐buildroot
"$TOP_DIR/BUILD"
-‐-‐define
"_topdir
$TOP_DIR"
-‐-‐define
"_tmppath
$TMP_PATH"
-‐-‐define
"_build_name_fmt
%%{NAME}-‐%%{VERSION}-‐%%{RELEASE}.%%{ARCH}.rpm"
-‐-‐define
"_rpmdir
$PROJECT_ROOT/$outputDir"
-‐-‐define
"source_dir
$PROJECT_ROOT"
-‐-‐
define
"in_version
$version"
src/some-‐app-‐name.spec
Slide 118
Slide 118 text
CREATE TOOLS TO
BUILD THEM
Solution:
Slide 119
Slide 119 text
Application Launcher Builder
Meta Package Builder
Config Override Builder
Confidential Config Override Builder
BUILDERS WE’VE CREATED
Slide 120
Slide 120 text
RHEL VS
CENTOS VS
OSX RPMBUILD
Each is different in it’s own special way.
Slide 121
Slide 121 text
WHAT DID
WE LEARN?
Slide 122
Slide 122 text
I HATE
RPMBUILD
Slide 123
Slide 123 text
SLOPPY RPM NAMING
CONVENTIONS
The version / release cannot contain a dash “-“, but this is not enforced.
name-version-release.architecture.rpm
Slide 124
Slide 124 text
REDLINE
platform independent JVM building using the JVM
There is a better way:
Slide 125
Slide 125 text
No content
Slide 126
Slide 126 text
YUM INDEXING
In which Mark spends four fifths of the time packaging his script
Act V:
Slide 127
Slide 127 text
YUM IS THE CENTOS
PACKAGE MANAGER
Yellowdog Updater, Modified
For those who don’t know
Slide 128
Slide 128 text
TYPICALLY
USED FOR OS
PACKAGES
Slide 129
Slide 129 text
WE DEPLOY
OUR
APPLICATIONS
USING YUM
Slide 130
Slide 130 text
YUM SUFFERS A SHORT
OUTAGE WHILE INDEXING
Which is a problem when you are publishing more and more packages.
The problem:
Slide 131
Slide 131 text
MORE CHURN ON YUM MEANS
MORE FAILED DEPLOYMENTS
Slide 132
Slide 132 text
RETRIES HAD BEEN
MASKING THE ISSUE
This had always been the case. But…
Slide 133
Slide 133 text
WE TURNED OFF YUM’S
CLIENT CACHE
It’s eviction policy is simply time based, which prevented deployments.
In addition:
Slide 134
Slide 134 text
WE THOUGHT THIS WAS
AWESOME
Repositories decouple build environments from deployment.
Go had recently introduced package polling
Slide 135
Slide 135 text
RUBY
Made a different choice of language:
Slide 136
Slide 136 text
HOW DO WE DEPLOY
RUBY?
After writing the script, we started to look into:
Slide 137
Slide 137 text
YET ANOTHER PACKAGE
MANAGER?
The community would suggest we should use:
Slide 138
Slide 138 text
GEM2RPM
Generates your .spec file from your .gemspec file.
Building a Gem server didn’t seem sensible.
Slide 139
Slide 139 text
DEPENDENCIES
But gem2rpm doesn’t deal with
Up to you to package them yourself…
Slide 140
Slide 140 text
WHAT DID
WE LEARN?
Slide 141
Slide 141 text
I HATE
YUM
Slide 142
Slide 142 text
I HATE
CREATEREPO
Slide 143
Slide 143 text
I HATE
RHEL 5
Lack of modern packages means old & slow createrepo.
Slide 144
Slide 144 text
No content
Slide 145
Slide 145 text
THE FUTURE
Act VI:
Slide 146
Slide 146 text
TEAMS
SELF
SERVING
Slide 147
Slide 147 text
PROVISIONING
& DEPLOYMENT
Slide 148
Slide 148 text
PRIVATE
CLOUD
Slide 149
Slide 149 text
ENVIRONMENT
MONITORING
Slide 150
Slide 150 text
GIT &
GITHUB
Slide 151
Slide 151 text
No quick wins—everything takes longer than you’d expect
Avoid Bash, use Shellcheck and Unofficial Bash Strict Mode
Don’t deploy your apps with RPMs / don’t use rpmbuild
Remember rpmbuild differs between Linux / UNIX flavours
Tackling provisioning can make deployment go away
Don’t model deployments with your CI tool
JVM is good to you—portability is easily taken for granted
CONCLUSIONS
Slide 152
Slide 152 text
IT IS EASY TO SEE HOW TO
IMPROVE WITH HINDSIGHT
We can only see better options now because of the pain we’ve gone through.