Automated Deployment at Auto Trader: How We Put Developers In Control & What We Learned

This is a talk I delivered about Automated Deployment at Auto Trader for a DevOps Edinburgh meet up on Monday 7th December at the Skyscanner offices in Edinburgh.

It covers covers:

• Auto Trader’s deployment history
• Our project which delivered control of deployment to developers
• What we learned from the experience
• Our in-progress product which will be the future of deployment at Auto Trader

1. AUTOMATED DEPLOYMENT AT AUTO TRADER:
 HOW WE PUT DEVELOPERS IN CONTROL & WHAT WE LEARNED MARK CROSSFIELD, DECEMBER 2015
2. AUTO TRADER Hiring! http://careers.autotrader.co.uk/ 340km away: 4hr by car or just 65m by plane © OpenStreetMap contributors
3. About Auto Trader and me Some deployment history Automated Deployment v1: Pipelines! What did we learn? Automated Deployment v2: Private Cloud STRUCTURE
4. Tech Lead in Operations @mrmanc / mark@markcrossfield.co.uk http://markcrossfield.co.uk http://www.devopsmanchester.co.uk/ Fascination for underground features http://hidden-manchester.org.uk/ MARK CROSSFIELD
5. DEVELOPER AT HEART but I care deeply about Operations
6. TWO YEARS IN OPS Eight years at Auto Trader
8. was a route into strong Operational empathy
9. LOTS OF PEOPLE have contributed to this over the course of years Although I’m telling the story…
10. JOURNEY This was, is and continues to be a
11. CLASSIFIED ADVERTISING We don’t sell cars… Auto Trader’s business is
12. Website launched in 1996
13. FINAL PRINT EDITION June 2013
14. AUTO TRADER PLC Investor owned Today…
15. 12K RETAILER CUSTOMERS make up most of our business revenue, not consumers
16. 4 MILLION PAGE VIEWS / HR At peak users generate data taken from Google Analytics
17. Multi faceted & geographic search over ~480k vehicles Large volume of traffic Large number of applications behind the scenes Lack of vehicle sale visibility masks business metrics CORE CHALLENGES
18. 2 x data centre suites each containing: F5 BIG-IP DNS Global Traffic Management F5 Local Traffic Management ~2,400 virtual servers using VMware ~200 HP Blades NetApp Storage INFRASTRUCTURE
20. Java: Spring Boot / Dropwizard Solr, Oracle, Mongo, MySQL, SQL Server, Vertica Jetty / JBoss Apache CentOS STACK
21. Where We Are
22. Continuous Delivery & Continuous Learning Tens of experiments per day WHERE ARE WE GOING?
23. 2007: Copied .war files 2008: Standardised script built RPMs, Perl deployment script, hired team of release engineers, VMware 2009: Redeveloped autotrader.co.uk built own RPM, introduced Cruise / Go CD 2010: Another project builds own RPM, release engineers absorbed 2011: Further projects build own RPM 2013: New CEO DEPLOYMENT HISTORY
24. bit.ly/SquadsAndTribes
25. SEPT 2013: DEPLOYMENT AUTOMATION
26. GOAL: TO CREATE BREATHING SPACE
27. SO WHAT DID WE DO?
30. LAYERED RPMS allowed developers to roll out config themselves
32. PIPELINE TEMPLATES & DEPENDENCIES allowed us to compose deployment processes from reusable templates
33. Deployment Pipelines
34. SELF CONTAINED APPS
35. HAVE A CLEAN CONTRACT WITH THE UNDERLYING OPERATING SYSTEM, OFFERING MAXIMUM PORTABILITY BETWEEN EXECUTION ENVIRONMENTS http://12factor.net/
36. THE TWELVE-FACTOR APP STORES CONFIG IN ENVIRONMENT VARIABLES… ENV VARS ARE EASY TO CHANGE BETWEEN DEPLOYS…, THEY ARE A LANGUAGE—AND OS—AGNOSTIC STANDARD. http://12factor.net/
37. Started to roll out Auto Deploy Formed new team in Ops
39. AUG 2015: DEVELOPERS KICK OFF ALL AUTO DEPLOYMENTS !
41. ! Config in the hands of developers ! Deployments are repeatable ! Deployments are faster ! Less people tied up ! No longer constrained by release slots RECAP
42. AWESOME, SO WHAT WAS PAINFUL?
49. THREE THINGS I WOULD CHANGE
50. (MIS)USING OUR CI TOOL
51. THE DEATH STAR How our pipelines looked before: Dec 2013
52. THE DEATH STAR How our pipelines looked during: Oct 2014
53. THE DEATH STAR How our pipelines look now: Dec 2015 https://flic.kr/p/qngNWb
54. RPMS FOR EVERYTHING
55. THE DOCUMENTATION
56. Stretching continuous integration environments Over-using complicated packaging format Overly complicated documentation is a smell PAIN RECAP
57. THOSE PITFALLS HELPED US REACH OUR GOAL
58. # Senior Leadership Support # Starting with frequent deployers # Piloting with flexible team # Automating existing processes # Building tools and build plugins # Focussed multi skilled team with CI mindset THINGS WHICH HELPED
59. 2015: PRIVATE CLOUD
60. Decouple devs from infrastructure using a clean API Make deployments easy and visible Make deployments faster Reduce config drift Make infrastructure change visible Flow regular updates through teams’ pipelines Facilitate emergency patches Remove innovation bottlenecks GOALS
61. FASTER DEPLOYMENTS DEPLOYING DEPLOYING! https://xkcd.com/303/
62. FASTER BACK OUT https://commons.wikimedia.org/wiki/File:Panic_button.jpg
63. EXISTING CLOUDSTACK & SCALR ENVIRONMENT
64. DEPLOYMENT SERVICE PROVIDES DECOUPLED API
65. AUTOMATED TESTING OF INFRASTRUCTURE CODE
66. BERKSHELF PACKAGES COOKBOOKS
68. SAME SPECIFICATION IN EACH ENVIRONMENT
69. EARLY DAYS
70. CLOUD READINESS
72. Get leadership buy in Dedicate people with the right skills and attitude Involve your customers & shout about progress Focus on the MVP Look for shortcuts Reorganise around products Keep it simple, stupid CONCLUSIONS
73. Resources Continuous Delivery
 Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble and David Farley http://www.amazon.co.uk/dp/0321601912 A manifesto for applications running on Heroku, and applicable as a set of best practices for cloud native applications with some great wisdom http://12factor.net/ Berkshelf: Chef Cookbook packaging (part of Chef DK) http://berkshelf.com/ https://www.go.cd/ https://downloads.chef.io/chef-dk/ Scaling Agile @ Spotify http://bit.ly/SquadsAndTribes