Auto Trader: from Contininuous Integration to Continuous Delivery

Here is a talk I gave at DevOpsManc about our Continuous Delivery team at Auto Trader and their activities facilitating automated deployment, preparing for a move into a private cloud and encouraging a culture of Continuous Delivery.

1. MARK CROSSFIELD, OCTOBER 2014 “AUTO TRADER: FROM CONTINUOUS INTEGRATION TO CONTINUOUS DELIVERY”
2. I’m Mark Crossfield DEVELOPER, ENGINEER Seven Years at Auto Trader
3. (definitely not a DevOps team) I work in our CONTINUOUS DELIVERY TEAM
4. THIS IS THE HONEST STORY OF THE LAST YEAR
5. Background This time last year Continuous Delivery Deployment Monitoring Self Contained Applications RPM Builders Yum Repo Indexing The Future CONTENTS
6. But first… A LITTLE INFORMATION about Auto Trader
7. Our traffic peaks at 70 MILLION page views / day
8. We have more stock than ever… 435,000 and that’s just cars!
9. Each month we see 14.5 MILLION unique users
10. In our First Street offices we have 323 product & technology staff
11. Our continuous build environment reads from 250 code bases
12. Website launched in 1996
13. In June 2013 we completed our PRINT TO DIGITAL TRANSITION We achieved continued growth of the business throughout.
14. Over the last five years WE’VE INVESTED HEAVILY IN OUR PRODUCTS but not so much in our infrastructure
15. make up most of our business revenue, not consumers TRADE RETAILERS
16. We have approximately 12,000 trade customers.
17. Our bespoke website platform serves around 4,500 websites for retail customers.
18. LARGE PORTFOLIO OF RETAIL MANAGEMENT SOFTWARE The website really is the tip of the iceberg
19. Of course… IT IS EASY TO SEE HOW TO IMPROVE WITH HINDSIGHT
20. Over the last year WE’VE KEPT THE LIGHTS ON whilst freeing up staff from manual deployments.
21. THIS TIME LAST YEAR
22. 46 DEPLOYS / WEEK
23. CARBON CONSTRAINT
24. LEGACY PERL
25. 75 APPS
26. 350 GO PIPELINES
27. 2000 VIRTUAL SERVERS
28. 920 RUN RHEL 5.4
29. RHEL 5.4 IS 5 YRS OLD
31. TWO DATA CENTRES
33. HOW OUR PIPELINES LOOKED
35. STRIPES, STRUTS, JERSEY! JETTY, JBOSS! JAVA! APACHE! PUPPET! CENTOS, RHEL! VMWARE, AWS / RIGHTSCALE! HP BLADE SERVERS
36. ORACLE, ENDECA In addition:
37. HOW WE DELIVER
39. JUST MOVED TO SPOTIFY MODEL
40. https://dl.dropboxusercontent.com/u/1018963/Articles/SpotifyScaling.pdf
42. WHAT IS CONTINUOUS DELIVERY?
44. http://www.thoughtworks.com/continuous-integration Continuous Integration Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early. ! By integrating regularly, you can detect errors quickly, and locate them more easily.
45. WHAT DOES THAT MEAN FOR INFRASTRUCTURE?
46. 1. YOUR CONFIGURATION IS CODE
47. 2. YOU ONLY MAKE CHANGES BY CHANGING YOUR CODE
48. 3. YOUR CODE IS TESTED IN A DIFFERENT ENVIRONMENT BEFORE IT IS USED
49. CONTINUOUS DELIVERY … getting the value out to your customers early and often.
50. PRIORITIES:! 1. DEPLOYMENT ! 2. MONITORING! 3. PROVISIONING! 4. DEPLOYMENT
52. DEPLOYMENT Act I
53. PERL SCRIPT HORROR
54. ADJUSTS LOAD BALANCER APP POOLS, STOPS / STARTS SERVICES, INVOKES YUM
55. SNOWFLAKE DEPLOYMENT SERVER
56. NO TESTS! NO VERSIONING! NO PACKAGING
57. NOT WRITTEN TO BE TESTABLE IN ISOLATION
58. NEEDED SIGNIFICANT CHANGES
59. TEST LOCALLY with stubbed out integration points Felt like we should be able to
60. PERL ISN’T PORTABLE
61. TESTS RUN AGAINST AN INTEGRATION ENVIRONMENT Used a CentOS virtual machine to run the tests locally. We compromised:
62. DEPLOYMENT PIPELINE TEMPLATE
63. Deployment pipeline dependencies
64. HOW OUR PIPELINES LOOKED
65. How our pipelines looked
66. HOW OUR PIPELINES LOOK NOW
68. 38 APPS AUTO DEPLOYED
69. 53 RELEASES PER WEEK (+7 YOY)
70. Auto Trader deployments broken down by month
71. STILL RUN BY OPS Releases
72. BUILT BY OPS Release pipelines
73. AUTO DEPLOY TO QA it takes a little longer Now that we
74. STILL ONE PER DAY Although many apps used to be none per day We’re more flexible over schedules
75. MANUAL RELEASES down from three full time One team member spends 1/3 of their time doing
76. 45 MINUTE DEPLOYMENT … down from two hours. Our main website
77. 13 MINUTES from passing quality control. Fastest deployments take
79. MONITORING Act III
80. SCALABLE GRAPHITE
81. DASHING
82. LOGSTASH, ELASTICSEARCH, KIBANA
83. METRICS 2.0
84. STATSD
85. GRAFANA
87. SELF CONTAINED APPLICATIONS Act III
88. MULTIPLE VERSIONS OF APPLICATION CONTAINERS … running on different versions in dev, qa & production. The problem:
89. SIMPLIFY APP SERVERS BY EMBEDDING THE CONTAINER That way the team owns their application container. The goal:
90. CREATE A STANDARD APPLICATION INIT SCRIPT Approach:
91. ESTABLISHES CONTRACT WITH SERVER Simplifying the interface with the OS.
92. LOCKING, KILLING, WAITING, RECOVERING This brings improvements…
93. TESTED WITH STUBBED OUT COMMANDS self shunt functions override commands to provide inversion of control Chose to write it in Bash
94. SUPPLYING DEPENDENCIES TO THINGS UNDER TEST Inversion of control Stubbed dependencies create a controlled environment
95. APPLICATION ENVIRONMENT CONFIGURATION Identified a great opportunity to tackle
96. DEVELOPERS HAVE STRONG OPINIONS We shouldn’t really have been surprised about that. Turns out…
97. ENVIRONMENT VARIABLES Application sees no file system changes. Provide environment specific variation using
98. INEVITABLY ONLY ONE ENVIRONMENT VARIABLE However…
99. Inevitably, we end up with only one environment variable… AT_CONFIGURATION_RUNTIME_ENVIRONMENT=QA
100. WHAT HAVE WE LEARNED?
101. I HATE BASH
102. ECHO RETURNS, BUT RETURN SETS $? Things I hate about Bash:
103. WORD SPLITTING after parameter expansion / command substitution Things I hate about Bash:
104. https://twitter.com/benjammingh/status/451506083128152064
105. BASH UNOFFICIAL STRICT MODE An easy way to make it safer: set -euo pipefail; IFS=$'\n\t'; http://redsymbol.net/articles/unofficial-bash-strict-mode/
106. EXIT IMMEDIATELY ON ERROR set -e
107. EXIT IMMEDIATELY ON UNDEFINED VARIABLES set -u
108. EXIT THE PIPE ON FAILURE —DO NOT MASK ERRORS set -o pipefail
109. RESTRICT THE INTER- FIELD SEPARATOR Produces more sensible looping behaviour. IFS=$'\n\t'
110. 26 SELF CONTAINED APPS Good progress:
112. RPM BUILDERS Act IV:
113. WE USE RPMS A *LOT* = It’s fair to say…
114. VERSIONED, DEPENDENCIES, FITS IN WITH REDHAT = It’s worked well for us
115. BUILDING RPMS IS FORGETTABLE Mostly because it is something we automate.
116. This is a spec file #  Generated  from  bundler-­‐1.6.5.gem  by  gem2rpm  -­‐*-­‐  rpm-­‐spec  -­‐*-­‐   %global  gemname  bundler   ! %global  gemdir  %(ruby  -­‐rubygems  -­‐e  'puts  Gem::dir'  2>/dev/null)   %global  geminstdir  %{gemdir}/gems/%{gemname}-­‐%{version}   %global  rubyabi  1.8   ! Summary:  The  best  way  to  manage  your  application's  dependencies   Name:  rubygem-­‐%{gemname}   Version:  1.6.5   Release:  1%{?dist}   Group:  Development/Languages   License:  closed   URL:  http://bundler.io   Source0:  http://rubygems.org/gems/%{gemname}-­‐%{version}.gem   Requires:  ruby(abi)  =  %{rubyabi}   Requires:  ruby(rubygems)  >=  1.3.6
117. This is how you call rpmbuild rpmbuild  -­‐v  -­‐V  -­‐-­‐buildroot  "$TOP_DIR/BUILD"  -­‐-­‐define  "_topdir   $TOP_DIR"  -­‐-­‐define  "_tmppath  $TMP_PATH"  -­‐-­‐define  "_build_name_fmt   %%{NAME}-­‐%%{VERSION}-­‐%%{RELEASE}.%%{ARCH}.rpm"  -­‐-­‐define  "_rpmdir   $PROJECT_ROOT/$outputDir"  -­‐-­‐define  "source_dir  $PROJECT_ROOT"  -­‐-­‐ define  "in_version  $version"  src/some-­‐app-­‐name.spec
118. CREATE TOOLS TO BUILD THEM Solution:
119. Application Launcher Builder Meta Package Builder Config Override Builder Confidential Config Override Builder BUILDERS WE’VE CREATED
120. RHEL VS CENTOS VS OSX RPMBUILD Each is different in it’s own special way.
121. WHAT DID WE LEARN?
122. I HATE RPMBUILD
123. SLOPPY RPM NAMING CONVENTIONS The version / release cannot contain a dash “-“, but this is not enforced. name-version-release.architecture.rpm
124. REDLINE platform independent JVM building using the JVM There is a better way:
126. YUM INDEXING In which Mark spends four fifths of the time packaging his script Act V:
127. YUM IS THE CENTOS PACKAGE MANAGER Yellowdog Updater, Modified For those who don’t know
128. TYPICALLY USED FOR OS PACKAGES
129. WE DEPLOY OUR APPLICATIONS USING YUM
130. YUM SUFFERS A SHORT OUTAGE WHILE INDEXING Which is a problem when you are publishing more and more packages. The problem:
131. MORE CHURN ON YUM MEANS MORE FAILED DEPLOYMENTS
132. RETRIES HAD BEEN MASKING THE ISSUE This had always been the case. But…
133. WE TURNED OFF YUM’S CLIENT CACHE It’s eviction policy is simply time based, which prevented deployments. In addition:
134. WE THOUGHT THIS WAS AWESOME Repositories decouple build environments from deployment. Go had recently introduced package polling
135. RUBY Made a different choice of language:
136. HOW DO WE DEPLOY RUBY? After writing the script, we started to look into:
137. YET ANOTHER PACKAGE MANAGER? The community would suggest we should use:
138. GEM2RPM Generates your .spec file from your .gemspec file. Building a Gem server didn’t seem sensible.
139. DEPENDENCIES But gem2rpm doesn’t deal with Up to you to package them yourself…
140. WHAT DID WE LEARN?
141. I HATE YUM
142. I HATE CREATEREPO
143. I HATE RHEL 5 Lack of modern packages means old & slow createrepo.
145. THE FUTURE Act VI:
146. TEAMS SELF SERVING
147. PROVISIONING & DEPLOYMENT
148. PRIVATE CLOUD
149. ENVIRONMENT MONITORING
150. GIT & GITHUB
151. No quick wins—everything takes longer than you’d expect Avoid Bash, use Shellcheck and Unofficial Bash Strict Mode Don’t deploy your apps with RPMs / don’t use rpmbuild Remember rpmbuild differs between Linux / UNIX flavours Tackling provisioning can make deployment go away Don’t model deployments with your CI tool JVM is good to you—portability is easily taken for granted CONCLUSIONS
152. IT IS EASY TO SEE HOW TO IMPROVE WITH HINDSIGHT We can only see better options now because of the pain we’ve gone through.
153. QUESTIONS?
154. Hiring! http://careers.autotrader.co.uk/