Continuous Deployment for Dummies

Transcript

1. Continuous Deployment for Dummies @rantav
2. WTF is Continuous Deployment?
3. Continuous Deployment is... • A Lean Startup technique • Code written ◦ Immediately deployed • Code deployed many times a day • Continuous Integration => the next step
4. Continuous Deployment is NOT... • ... is not Low Quality code ◦ Quite the contrary, very high quality • ... is not Cowboy Coding ◦ CD requires discipline. Culture is key • ... is not Time Consuming or requires massive investment up front ◦ Small initial investment ◦ Actually saves time • ... is not Just for web services ◦ although it's easier for webs
5. What was wrong before?
6. No one wants this...
7. PM Goal: Minimize feedback loop • Developers know REPL ◦ Read Evaluate Print Loop • Product managers need fast feedback too ◦ CD is a step forward
8. Does anyone want that?
9. Dev Goal: Less process, More fun •Less waits ◦Wait for QA ◦Wait for end of cycle ◦Wait for code merge ◦Wait for other devs ◦Wait for... •Less context switches ◦Do one task, end it, next
10. Of course no one wants this...
11. Ops Goal: Increase Stability •Small changes - small risks •Large changes - large risks •You can spill a bucket, but you can't spill a hose / Kent Beck
12. CD in five easy steps - step 1 •Step 1: Continuous Integration ◦Jenkins (aka Hudson) ◦Team City ◦Build Bot ◦... Continuous deployment in 5 easy steps
13. CD in five easy steps - step 2 •Step 2: Post Commit Hooks ◦svn: post-commit ◦git: post-receive ◦cvs: post-commit ◦perforce... Continuous deployment in 5 easy steps
14. CD in five easy steps - step 3 •Step 3: Simple Deployment Script ◦Start simple ◦Build as you grow ▪Don't start with the almighty ◦Add Immune system as you grow Continuous deployment in 5 easy steps
15. CD in five easy steps - step 4 •Step 4: Real Time Alerting ◦Nagios ◦OpenNMS ◦... Continuous deployment in 5 easy steps
16. CD in five easy steps - step 5 •Step 5: Root Cause Analysis ◦Five Whys ◦Fail fast, learn fast, improve Continuous deployment in 5 easy steps
17. Extra • Mechanics ◦ Are you crazy? No, I'm immune • Ingredients ◦ Culture ◦ Trunk stable ◦ Automated Tests ◦ Infrastructure Automation ◦ Deployer ◦ Servicization ◦ The Immune system ◦ Visibility ◦ 5 Ys • Workflow
18. Mechanics Commit => new code in production
19. Are you crazy? What? Are you Crazy? Deploy? Just like that? No QA? No nothing?...
20. The immune system Let me present: The Immune System
21. The Immune System • Coding ◦ Review ◦ Testing • Monitoring ◦ We use Nagios for the vitals ◦ Regularly check servers via instrumentation ◦ KeyNote ◦ Test Production (selenium,APT) • Instrumentation ◦ Self-Test ◦ Performance • Monitoring KPIs ◦ PVs, ◦ 3 different CTRs, ◦ clicks, revenue, RPM ◦ etc...
22. The Ingredients
23. Culture
24. Culture • Everyone need to care about everything! ◦ build ◦ tests ◦ quality ◦ production ◦ monitoring ◦ business
25. Culture • No Broken Windows! Broken Windows Theory
26. Culture “What's the worse that could happen?”
27. Trunk Stable •Everyone run tests before commit •No branches, really •Trunk may get released any moment. If you commit now, users will see it really soon. => Test your code, really good •Use feature flags, but only if you must •Forward and backward compatibility
28. Automated Tests •We have about 2000 test cases •They run in < 4 minutes •We use TeamCity •We regularly test production Desired state: No manual QA whatsoever.
29. Infrastructure Automation "Infrastructure as code" •Using kickstart script to install OS and Chef agent. •All infrastructure is deployed by Chef. •All apps are deployed by Glu •All scripts, and chef cookbooks, glu configs are SVNed. •Very easy to deploy large number of machines.
30. Deployer
33. Servicization • At outbrain we have ~ 25 services • Each service is deployed at > 1 server • If there's damage, it's contained • It's easy to make small changes • If there's an error it's easy to find it • If rollback is needed it's easier • Everything is either proxies (HAProxy) or queued (AMQ) • Challenge: It's sometimes hard to find the right balance: ◦ number of services that need maintanance ◦ complexity ◦ performance ◦ api conformance
34. The Immune System • Coding ◦ Review ◦ Testing • Monitoring ◦ We use Nagios for the vitals ◦ Regularly check servers via instrumentation ◦ KeyNote ◦ Test Production (selenium,APT) • Instrumentation ◦ Self-Test ◦ Performance • Monitoring KPIs ◦ PVs, ◦ 3 different CTRs, ◦ clicks, revenue, RPM ◦ etc...
35. The Immune System • The line of defense will always be broken => Multiple lines of defense
36. Visibilty • Everyone monitor the services • When there's a deployment, you see it on the graph
37. Visibilty • Yammer
38. Visibilty • svn changelog • glu changelog • glu audit log
39. Learn Fast, Adapt Fast • 5 Ys
40. WORKFLOW
41. WORKFLOW
42. Who's in?
43. Fun Numbers • 5-50 production changes a day!!! • More then 2000 code tests running in less then 4 minutes. • More then 600 production services tests runs every 10 minutes. • It takes ~30 minutes from code complete to ~100 machines deployed.
44. References • Why Continuous Deployment / Eric Ries • Continuous Deployment at outbrain / Ran Tavory • Deployment Infrastructure for Continuous Deployment / WealthFront • Continuous Deployment presentation / Eishay Smith • Quantum of Deployment / Etsy • Chrome Release Cycle / Anthony Laforge