Continuous Deployment for Dummies

Slide 1

Slide 1 text

Continuous Deployment for Dummies @rantav

Slide 2

Slide 2 text

WTF is Continuous Deployment?

Slide 3

Slide 3 text

Continuous Deployment is... ● A Lean Startup technique ● Code written ○ Immediately deployed ● Code deployed many times a day ● Continuous Integration => the next step

Slide 4

Slide 4 text

Continuous Deployment is NOT... ● ... is not Low Quality code ○ Quite the contrary, very high quality ● ... is not Cowboy Coding ○ CD requires discipline. Culture is key ● ... is not Time Consuming or requires massive investment up front ○ Small initial investment ○ Actually saves time ● ... is not Just for web services ○ although it's easier for webs

Slide 5

Slide 5 text

What was wrong before?

Slide 6

Slide 6 text

No one wants this...

Slide 7

Slide 7 text

PM Goal: Minimize feedback loop ● Developers know REPL ○ Read Evaluate Print Loop ● Product managers need fast feedback too ○ CD is a step forward

Slide 8

Slide 8 text

Does anyone want that?

Slide 9

Slide 9 text

Dev Goal: Less process, More fun ●Less waits ○Wait for QA ○Wait for end of cycle ○Wait for code merge ○Wait for other devs ○Wait for... ●Less context switches ○Do one task, end it, next

Slide 10

Slide 10 text

Of course no one wants this...

Slide 11

Slide 11 text

Ops Goal: Increase Stability ●Small changes - small risks ●Large changes - large risks ●You can spill a bucket, but you can't spill a hose / Kent Beck

Slide 12

Slide 12 text

CD in five easy steps - step 1 ●Step 1: Continuous Integration ○Jenkins (aka Hudson) ○Team City ○Build Bot ○... Continuous deployment in 5 easy steps

Slide 13

Slide 13 text

CD in five easy steps - step 2 ●Step 2: Post Commit Hooks ○svn: post-commit ○git: post-receive ○cvs: post-commit ○perforce... Continuous deployment in 5 easy steps

Slide 14

Slide 14 text

CD in five easy steps - step 3 ●Step 3: Simple Deployment Script ○Start simple ○Build as you grow ■Don't start with the almighty ○Add Immune system as you grow Continuous deployment in 5 easy steps

Slide 15

Slide 15 text

CD in five easy steps - step 4 ●Step 4: Real Time Alerting ○Nagios ○OpenNMS ○... Continuous deployment in 5 easy steps

Slide 16

Slide 16 text

CD in five easy steps - step 5 ●Step 5: Root Cause Analysis ○Five Whys ○Fail fast, learn fast, improve Continuous deployment in 5 easy steps

Slide 17

Slide 17 text

Extra ● Mechanics ○ Are you crazy? No, I'm immune ● Ingredients ○ Culture ○ Trunk stable ○ Automated Tests ○ Infrastructure Automation ○ Deployer ○ Servicization ○ The Immune system ○ Visibility ○ 5 Ys ● Workflow

Slide 18

Slide 18 text

Mechanics Commit => new code in production

Slide 19

Slide 19 text

Are you crazy? What? Are you Crazy? Deploy? Just like that? No QA? No nothing?...

Slide 20

Slide 20 text

The immune system Let me present: The Immune System

Slide 21

Slide 21 text

The Immune System ● Coding ○ Review ○ Testing ● Monitoring ○ We use Nagios for the vitals ○ Regularly check servers via instrumentation ○ KeyNote ○ Test Production (selenium,APT) ● Instrumentation ○ Self-Test ○ Performance ● Monitoring KPIs ○ PVs, ○ 3 different CTRs, ○ clicks, revenue, RPM ○ etc...

Slide 22

Slide 22 text

The Ingredients

Slide 23

Slide 23 text

Culture

Slide 24

Slide 24 text

Culture ● Everyone need to care about everything! ○ build ○ tests ○ quality ○ production ○ monitoring ○ business

Slide 25

Slide 25 text

Culture ● No Broken Windows! Broken Windows Theory

Slide 26

Slide 26 text

Culture “What's the worse that could happen?”

Slide 27

Slide 27 text

Trunk Stable ●Everyone run tests before commit ●No branches, really ●Trunk may get released any moment. If you commit now, users will see it really soon. => Test your code, really good ●Use feature flags, but only if you must ●Forward and backward compatibility

Slide 28

Slide 28 text

Automated Tests ●We have about 2000 test cases ●They run in < 4 minutes ●We use TeamCity ●We regularly test production Desired state: No manual QA whatsoever.

Slide 29

Slide 29 text

Infrastructure Automation "Infrastructure as code" ●Using kickstart script to install OS and Chef agent. ●All infrastructure is deployed by Chef. ●All apps are deployed by Glu ●All scripts, and chef cookbooks, glu configs are SVNed. ●Very easy to deploy large number of machines.

Slide 30

Slide 30 text

Deployer

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Servicization ● At outbrain we have ~ 25 services ● Each service is deployed at > 1 server ● If there's damage, it's contained ● It's easy to make small changes ● If there's an error it's easy to find it ● If rollback is needed it's easier ● Everything is either proxies (HAProxy) or queued (AMQ) ● Challenge: It's sometimes hard to find the right balance: ○ number of services that need maintanance ○ complexity ○ performance ○ api conformance

Slide 34

Slide 34 text

Slide 35

Slide 35 text

The Immune System ● The line of defense will always be broken => Multiple lines of defense

Slide 36

Slide 36 text

Visibilty ● Everyone monitor the services ● When there's a deployment, you see it on the graph

Slide 37

Slide 37 text

Visibilty ● Yammer

Slide 38

Slide 38 text

Visibilty ● svn changelog ● glu changelog ● glu audit log

Slide 39

Slide 39 text

Learn Fast, Adapt Fast ● 5 Ys

Slide 40

Slide 40 text

WORKFLOW

Slide 41

Slide 41 text

WORKFLOW

Slide 42

Slide 42 text

Who's in?

Slide 43

Slide 43 text

Fun Numbers ● 5-50 production changes a day!!! ● More then 2000 code tests running in less then 4 minutes. ● More then 600 production services tests runs every 10 minutes. ● It takes ~30 minutes from code complete to ~100 machines deployed.

Slide 44

Slide 44 text

References ● Why Continuous Deployment / Eric Ries ● Continuous Deployment at outbrain / Ran Tavory ● Deployment Infrastructure for Continuous Deployment / WealthFront ● Continuous Deployment presentation / Eishay Smith ● Quantum of Deployment / Etsy ● Chrome Release Cycle / Anthony Laforge