Continuous Deployment - IEEE

Tools and techniques used at outbrain for CD.
Presentation for IEEE intl conference June 2012 Herzliya

Transcript

1. Continuous Deployment Tools and Techniques @outbrain Ran Tavory @rantav Monday, June 11, 2012
2. How is it done at outbrain? svn ci -m "Implement the super-sharp image scaler. #deloy:ImageServer #to:ny" Monday, June 11, 2012
3. In Five Easy Steps Monday, June 11, 2012
4. In Five Easy Steps Monday, June 11, 2012
5. In Five Easy Steps 1. Continuous Integration Monday, June 11, 2012
6. In Five Easy Steps 1. Continuous Integration 2. Post Commit Hook Monday, June 11, 2012
7. In Five Easy Steps 1. Continuous Integration 2. Post Commit Hook 3. Deployment Script Monday, June 11, 2012
8. In Five Easy Steps 1. Continuous Integration 2. Post Commit Hook 3. Deployment Script 4. Realtime Alerting (monitoring) Monday, June 11, 2012
9. In Five Easy Steps 1. Continuous Integration 2. Post Commit Hook 3. Deployment Script 4. Realtime Alerting (monitoring) 5. Root Cause Analysis Monday, June 11, 2012
10. Outbrain’s Arsenal • Java: Tomcat, Spring, SpringMVC, Struts2 • JavaScript: ExtJS and homemade • Data: MySQL, Cassandra, HDFS, MogileFS, Memcached • Message queues: ActiveMQ Monday, June 11, 2012
11. Outbrain’s Arsenal • Service multiplicity: HAProxy • Monitoring: Nagios, Keynote, Pingdom • Infra: CentOS, Kickstart, Chef, Glu, RPM +YUM • Testing: JUnit, TeamCity, Selenium, Staging Monday, June 11, 2012
12. Fun Numbers • 5-50 deployments / day • 4800 tests / every commit • 1600 production tests / hour • 47 services • 100s of servers / 3 Datacenters • 5-12 minutes from commit to production Monday, June 11, 2012
13. Tools and Techniques Monday, June 11, 2012
14. Tools and Techniques • Trunk Stable Monday, June 11, 2012
15. Tools and Techniques • Trunk Stable • Continuous Integration Monday, June 11, 2012
16. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests Monday, June 11, 2012
17. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation Monday, June 11, 2012
18. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation • Deployer Monday, June 11, 2012
19. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments Monday, June 11, 2012
20. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments • Servicization Monday, June 11, 2012
21. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments • Servicization • Immune System Monday, June 11, 2012
22. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments • Servicization • Immune System • Visibility Monday, June 11, 2012
23. Trunk Stable • Trunk always builds cleanly • (if broken - commit freeze) • No SCM branches • But there are feature flags • Tests always green • Trunk may always be deployed (not just by you) • => You better have your features well tested • Forward and backwards compatibility (DB) Covered by Itai Monday, June 11, 2012
24. Automated Testing • 4800 unit tests • They run in 5 minutes • 1600 production tests • Run hourly and after deployments Monday, June 11, 2012
25. Infrastructure Automation •Infrastructure as Code • Kickstart: install OS and Chef • Chef: to configure host and install services • Glu: to install applications • Infrastructure code is in trunk Monday, June 11, 2012
26. Deployer Command line Web UI Monday, June 11, 2012
27. Deployer - Log Monday, June 11, 2012
28. Deployer (Glu) - State Monday, June 11, 2012
29. Deployer (Glu) - Deploy Log Monday, June 11, 2012
30. Gradual Deployment • Deploy first Canary • Self Test • Then deploy 1/3 of each DC in parallel • Self Test all • Continue until done Monday, June 11, 2012
31. Servicization • 47 different services • At least one replica for each service • Everything either load-balanced (HAProxy) or queued (ActiveMQ) • Services talk REST Monday, June 11, 2012
32. Servicization What do we get from it? • Easy to make small incremental changes • If there’s damage, it’s contained (memory leak, handle leak etc) • Rollback (if desired) is possible. Roll forward is preferred Monday, June 11, 2012
33. Servicization Challanges • Challenge: Find the correct balance: • Service Coherency • Number of services that need operational maintenance • Interop complexity • Performance Monday, June 11, 2012
34. The Immune System • The line of defense will always be broken => Multiple lines of defense Monday, June 11, 2012
35. The Immune System Coding • Code Review • Pairing • Best Practices • Static analysis • Unit Test Monday, June 11, 2012
36. The Immune System Production Time • Nagios + Graphite • KeyNote, pingdom • Test Production (selenium, APT, 1800 tests hourly) Monday, June 11, 2012
37. The Immune System Instrumentation • Self-Test • JMX • Performance • metrics • speed4j Monday, June 11, 2012
38. The Immune System Monitoring KPIs • PVs, • 3 different CTRs, • clicks, revenue • etc... Monday, June 11, 2012
39. Visibility • Everyone monitor the services • Deployment events on graph • Yammer • Email • Changelog • svn, glu, yammer Monday, June 11, 2012
40. Monday, June 11, 2012
41. References • Why Continuous Deployment / Eric Ries http://www.startuplessonslearned.com/2009/06/why-continuous-deployment.html • Continuous Deployment in five easy steps / Eric Ries http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html • Continuous Deployment at outbrain / Ran Tavory http://prettyprint.me/2011/01/24/continuous-deployment-at-outbrain/ • Deployment Infrastructure for Continuous Deployment / WealthFront http://eng.wealthfront.com/2010/05/deployment-infrastructure-for.html • Continuous Deployment presentation / Eishay Smith http://prezi.com/5zm8xplapff2/continuous-deployment/ • Quantum of Deployment / Etsy http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/ • Chrome Release Cycle / Anthony Laforge https://docs.google.com/present/view?id=dg63dpc6_4d7vkk6ch&pli=1 Monday, June 11, 2012