Automated Deployment at Auto Trader:  How We Put Developers In Control & What We Learned

Slide 1

Slide 1 text

AUTOMATED DEPLOYMENT AT AUTO TRADER:  HOW WE PUT DEVELOPERS IN CONTROL & WHAT WE LEARNED MARK CROSSFIELD, DECEMBER 2015

Slide 2

Slide 2 text

AUTO TRADER Hiring! http://careers.autotrader.co.uk/ 340km away: 4hr by car or just 65m by plane © OpenStreetMap contributors

Slide 3

Slide 3 text

About Auto Trader and me Some deployment history Automated Deployment v1: Pipelines! What did we learn? Automated Deployment v2: Private Cloud STRUCTURE

Slide 4

Slide 4 text

Tech Lead in Operations @mrmanc / [email protected] http://markcrossfield.co.uk http://www.devopsmanchester.co.uk/ Fascination for underground features http://hidden-manchester.org.uk/ MARK CROSSFIELD

Slide 5

Slide 5 text

DEVELOPER AT HEART but I care deeply about Operations

Slide 6

Slide 6 text

TWO YEARS IN OPS Eight years at Auto Trader

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

was a route into strong Operational empathy

Slide 9

Slide 9 text

LOTS OF PEOPLE have contributed to this over the course of years Although I’m telling the story…

Slide 10

Slide 10 text

JOURNEY This was, is and continues to be a

Slide 11

Slide 11 text

CLASSIFIED ADVERTISING We don’t sell cars… Auto Trader’s business is

Slide 12

Slide 12 text

Website launched in 1996

Slide 13

Slide 13 text

FINAL PRINT EDITION June 2013

Slide 14

Slide 14 text

AUTO TRADER PLC Investor owned Today…

Slide 15

Slide 15 text

12K RETAILER CUSTOMERS make up most of our business revenue, not consumers

Slide 16

Slide 16 text

4 MILLION PAGE VIEWS / HR At peak users generate data taken from Google Analytics

Slide 17

Slide 17 text

Multi faceted & geographic search over ~480k vehicles Large volume of traffic Large number of applications behind the scenes Lack of vehicle sale visibility masks business metrics CORE CHALLENGES

Slide 18

Slide 18 text

2 x data centre suites each containing: F5 BIG-IP DNS Global Traffic Management F5 Local Traffic Management ~2,400 virtual servers using VMware ~200 HP Blades NetApp Storage INFRASTRUCTURE

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Java: Spring Boot / Dropwizard Solr, Oracle, Mongo, MySQL, SQL Server, Vertica Jetty / JBoss Apache CentOS STACK

Slide 21

Slide 21 text

Where We Are

Slide 22

Slide 22 text

Continuous Delivery & Continuous Learning Tens of experiments per day WHERE ARE WE GOING?

Slide 23

Slide 23 text

2007: Copied .war files 2008: Standardised script built RPMs, Perl deployment script, hired team of release engineers, VMware 2009: Redeveloped autotrader.co.uk built own RPM, introduced Cruise / Go CD 2010: Another project builds own RPM, release engineers absorbed 2011: Further projects build own RPM 2013: New CEO DEPLOYMENT HISTORY

Slide 24

Slide 24 text

bit.ly/SquadsAndTribes

Slide 25

Slide 25 text

SEPT 2013: DEPLOYMENT AUTOMATION

Slide 26

Slide 26 text

GOAL: TO CREATE BREATHING SPACE

Slide 27

Slide 27 text

SO WHAT DID WE DO?

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

LAYERED RPMS allowed developers to roll out config themselves

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

PIPELINE TEMPLATES & DEPENDENCIES allowed us to compose deployment processes from reusable templates

Slide 33

Slide 33 text

Deployment Pipelines

Slide 34

Slide 34 text

SELF CONTAINED APPS

Slide 35

Slide 35 text

HAVE A CLEAN CONTRACT WITH THE UNDERLYING OPERATING SYSTEM, OFFERING MAXIMUM PORTABILITY BETWEEN EXECUTION ENVIRONMENTS http://12factor.net/

Slide 36

Slide 36 text

THE TWELVE-FACTOR APP STORES CONFIG IN ENVIRONMENT VARIABLES… ENV VARS ARE EASY TO CHANGE BETWEEN DEPLOYS…, THEY ARE A LANGUAGE—AND OS—AGNOSTIC STANDARD. http://12factor.net/

Slide 37

Slide 37 text

Started to roll out Auto Deploy Formed new team in Ops

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

AUG 2015: DEVELOPERS KICK OFF ALL AUTO DEPLOYMENTS !

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

! Config in the hands of developers ! Deployments are repeatable ! Deployments are faster ! Less people tied up ! No longer constrained by release slots RECAP

Slide 42

Slide 42 text

AWESOME, SO WHAT WAS PAINFUL?

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

THREE THINGS I WOULD CHANGE

Slide 50

Slide 50 text

(MIS)USING OUR CI TOOL

Slide 51

Slide 51 text

THE DEATH STAR How our pipelines looked before: Dec 2013

Slide 52

Slide 52 text

THE DEATH STAR How our pipelines looked during: Oct 2014

Slide 53

Slide 53 text

THE DEATH STAR How our pipelines look now: Dec 2015 https://ﬂic.kr/p/qngNWb

Slide 54

Slide 54 text

RPMS FOR EVERYTHING

Slide 55

Slide 55 text

THE DOCUMENTATION

Slide 56

Slide 56 text

Stretching continuous integration environments Over-using complicated packaging format Overly complicated documentation is a smell PAIN RECAP

Slide 57

Slide 57 text

THOSE PITFALLS HELPED US REACH OUR GOAL

Slide 58

Slide 58 text

# Senior Leadership Support # Starting with frequent deployers # Piloting with flexible team # Automating existing processes # Building tools and build plugins # Focussed multi skilled team with CI mindset THINGS WHICH HELPED

Slide 59

Slide 59 text

2015: PRIVATE CLOUD

Slide 60

Slide 60 text

Decouple devs from infrastructure using a clean API Make deployments easy and visible Make deployments faster Reduce config drift Make infrastructure change visible Flow regular updates through teams’ pipelines Facilitate emergency patches Remove innovation bottlenecks GOALS

Slide 61

Slide 61 text

FASTER DEPLOYMENTS DEPLOYING DEPLOYING! https://xkcd.com/303/

Slide 62

Slide 62 text

FASTER BACK OUT https://commons.wikimedia.org/wiki/File:Panic_button.jpg

Slide 63

Slide 63 text

EXISTING CLOUDSTACK & SCALR ENVIRONMENT

Slide 64

Slide 64 text

DEPLOYMENT SERVICE PROVIDES DECOUPLED API

Slide 65

Slide 65 text

AUTOMATED TESTING OF INFRASTRUCTURE CODE

Slide 66

Slide 66 text

BERKSHELF PACKAGES COOKBOOKS

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

SAME SPECIFICATION IN EACH ENVIRONMENT

Slide 69

Slide 69 text

EARLY DAYS

Slide 70

Slide 70 text

CLOUD READINESS

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

Get leadership buy in Dedicate people with the right skills and attitude Involve your customers & shout about progress Focus on the MVP Look for shortcuts Reorganise around products Keep it simple, stupid CONCLUSIONS

Slide 73

Slide 73 text

Resources Continuous Delivery  Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble and David Farley http://www.amazon.co.uk/dp/0321601912 A manifesto for applications running on Heroku, and applicable as a set of best practices for cloud native applications with some great wisdom http://12factor.net/ Berkshelf: Chef Cookbook packaging (part of Chef DK) http://berkshelf.com/ https://www.go.cd/ https://downloads.chef.io/chef-dk/ Scaling Agile @ Spotify http://bit.ly/SquadsAndTribes