Continuous Deployment

Transcript

1. Continuous Deployment Tools and Techniques @outbrain Ran Tavory @rantav Tuesday, June 19, 2012
2. How much time does it take your code to meet the users? Tuesday, June 19, 2012
3. How much time does it take your code to meet the users? • More than a year? Tuesday, June 19, 2012
4. How much time does it take your code to meet the users? • More than a year? • 6-12 months? Tuesday, June 19, 2012
5. How much time does it take your code to meet the users? • More than a year? • 6-12 months? • 1-6 months? Tuesday, June 19, 2012
6. How much time does it take your code to meet the users? • More than a year? • 6-12 months? • 1-6 months? • 2-4 weeks? Tuesday, June 19, 2012
7. How much time does it take your code to meet the users? • More than a year? • 6-12 months? • 1-6 months? • 2-4 weeks? • 1-14 days? Tuesday, June 19, 2012
8. How much time does it take your code to meet the users? • More than a year? • 6-12 months? • 1-6 months? • 2-4 weeks? • 1-14 days? • 1-24 hours? Tuesday, June 19, 2012
9. How much time does it take your code to meet the users? • More than a year? • 6-12 months? • 1-6 months? • 2-4 weeks? • 1-14 days? • 1-24 hours? •10 minutes? Tuesday, June 19, 2012
10. How is it done at outbrain? svn ci -m "Implement the super-sharp image scaler. #deloy:ImageServer #to:ny" Tuesday, June 19, 2012
11. Martin Fowler Eric Ries Kent Beck ...and the world Tuesday, June 19, 2012
12. What was wrong? Tuesday, June 19, 2012
13. For the business What is a startup? a startup is always on a quest to find a product-market fit => You have to iterate fast ... before running out of money Tuesday, June 19, 2012
14. For the business • Feedback loop speed is important • Developers have REPL • But what do Product Managers have? Tuesday, June 19, 2012
15. For the developers • What's used to be the case before: • Inefficient Waits (wait for QA, wait for other features etc) • Inefficient Context Switch • Feature Delayed - b/c of other features • Big Changes – Big Problems. Tuesday, June 19, 2012
16. Fun Numbers • 5-50 deployments / day • 4800 tests / every commit • 1600 production tests / hour • 47 services • 100s of servers / 3 Datacenters • 5-12 minutes from commit to production Tuesday, June 19, 2012
17. How? Tuesday, June 19, 2012
18. In Five Easy Steps Tuesday, June 19, 2012
19. In Five Easy Steps Tuesday, June 19, 2012
20. In Five Easy Steps 1. Continuous Integration Tuesday, June 19, 2012
21. In Five Easy Steps 1. Continuous Integration 2. Post Commit Hook Tuesday, June 19, 2012
22. In Five Easy Steps 1. Continuous Integration 2. Post Commit Hook 3. Deployment Script Tuesday, June 19, 2012
23. In Five Easy Steps 1. Continuous Integration 2. Post Commit Hook 3. Deployment Script 4. Realtime Alerting (monitoring) Tuesday, June 19, 2012
24. In Five Easy Steps 1. Continuous Integration 2. Post Commit Hook 3. Deployment Script 4. Realtime Alerting (monitoring) 5. Root Cause Analysis Tuesday, June 19, 2012
25. Culture Tuesday, June 19, 2012
26. CD Themes Release is a Marketing decision Deployment is an Engineering decision Tuesday, June 19, 2012
27. CD Themes Small changes reduce risks Kent Beck: You can spill a bucket, but you cannot spill a hose Tuesday, June 19, 2012
28. CD Themes Deploy often, deploy fast Fast turnaround => Happy customers ... and happy developers Tuesday, June 19, 2012
29. CD Themes Deploy when you are ready Do not have to wait for other teams Tuesday, June 19, 2012
30. CD Themes Everyone cares about Everything • build • test • quality • production • monitoring • business Tuesday, June 19, 2012
31. Culture No broken windows! Tuesday, June 19, 2012
32. Culture Management Support: What’s the worst that could happen? Tuesday, June 19, 2012
33. Workflow Tuesday, June 19, 2012
34. Workflow Tuesday, June 19, 2012
35. Workflow Tuesday, June 19, 2012
36. Workflow Tuesday, June 19, 2012
37. Workflow Tuesday, June 19, 2012
38. Workflow Minimize Work in Progress Tuesday, June 19, 2012
39. Workflow Definition of Done: Released and monitoring shows value Tuesday, June 19, 2012
40. Focus on Production Measure everything Some even eliminate test, stg etc Tuesday, June 19, 2012
41. Tools and Techniques Tuesday, June 19, 2012
42. Tools and Techniques • Trunk Stable Tuesday, June 19, 2012
43. Tools and Techniques • Trunk Stable • Continuous Integration Tuesday, June 19, 2012
44. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests Tuesday, June 19, 2012
45. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation Tuesday, June 19, 2012
46. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation • Deployer Tuesday, June 19, 2012
47. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments Tuesday, June 19, 2012
48. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments • Servicization Tuesday, June 19, 2012
49. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments • Servicization • Immune System Tuesday, June 19, 2012
50. Tools and Techniques • Trunk Stable • Continuous Integration • Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments • Servicization • Immune System • Visibility Tuesday, June 19, 2012
51. Trunk Stable • Trunk always builds cleanly • (if broken - commit freeze) • No SCM branches • But there are feature flags • Tests always green • Trunk may always be deployed (not just by you) • => You better have your features well tested • Forward and backwards compatibility (DB) Tuesday, June 19, 2012
52. Automated Testing • 4800 unit tests • They run in 5 minutes • 1600 production tests • Run hourly and after deployments Tuesday, June 19, 2012
53. Infrastructure Automation •Infrastructure as Code • Kickstart: install OS and Chef • Chef: to configure host and install services • Glu: to install applications • Infrastructure code is in trunk Tuesday, June 19, 2012
54. Deployer Command line Web UI Tuesday, June 19, 2012
55. Deployer - Log Tuesday, June 19, 2012
56. Deployer (Glu) - State Tuesday, June 19, 2012
57. Deployer (Glu) - Deploy Log Tuesday, June 19, 2012
58. Gradual Deployment • Deploy first Canary • Self Test • Then deploy 1/3 of each DC in parallel • Self Test all • Continue until done Tuesday, June 19, 2012
59. Servicization • 47 different services • At least one replica for each service • Everything either load-balanced (HAProxy) or queued (ActiveMQ) • Services talk REST Tuesday, June 19, 2012
60. Servicization What do we get from it? • Easy to make small incremental changes • If there’s damage, it’s contained (memory leak, handle leak etc) • Rollback (if desired) is possible. Roll forward is preferred Tuesday, June 19, 2012
61. Servicization Challanges • Challenge: Find the correct balance: • Service Coherency • Number of services that need operational maintenance • Interop complexity • Performance Tuesday, June 19, 2012
62. The Immune System • The line of defense will always be broken => Multiple lines of defense Tuesday, June 19, 2012
63. The Immune System Coding • Code Review • Pairing • Best Practices • Static analysis • Unit Test Tuesday, June 19, 2012
64. The Immune System Production Time • Nagios + Graphite • KeyNote, pingdom • Test Production (selenium, APT, 1800 tests hourly) Tuesday, June 19, 2012
65. The Immune System Instrumentation • Self-Test • JMX • Performance • metrics • speed4j Tuesday, June 19, 2012
66. The Immune System Monitoring KPIs • PVs, • 3 different CTRs, • clicks, revenue • etc... Tuesday, June 19, 2012
67. Visibility • Everyone monitor the services • Deployment events on graph • Yammer • Email • Changelog • svn, glu, yammer Tuesday, June 19, 2012
68. Tuesday, June 19, 2012
69. Outbrain’s Arsenal • Java: Tomcat, Spring, SpringMVC, Struts2 • JavaScript: ExtJS and homemade • Data: MySQL, Cassandra, HDFS, MogileFS, Memcached • Message queues: ActiveMQ Tuesday, June 19, 2012
70. Outbrain’s Arsenal • Service multiplicity: HAProxy • Monitoring: Nagios, Keynote, Pingdom • Infra: CentOS, Kickstart, Chef, Glu, RPM +YUM • Testing: JUnit, TeamCity, Selenium, Staging Tuesday, June 19, 2012
71. References • Why Continuous Deployment / Eric Ries http://www.startuplessonslearned.com/2009/06/why-continuous-deployment.html • Continuous Deployment in five easy steps / Eric Ries http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html • Continuous Deployment at outbrain / Ran Tavory http://prettyprint.me/2011/01/24/continuous-deployment-at-outbrain/ • Deployment Infrastructure for Continuous Deployment / WealthFront http://eng.wealthfront.com/2010/05/deployment-infrastructure-for.html • Continuous Deployment presentation / Eishay Smith http://prezi.com/5zm8xplapff2/continuous-deployment/ • Quantum of Deployment / Etsy http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/ • Chrome Release Cycle / Anthony Laforge https://docs.google.com/present/view?id=dg63dpc6_4d7vkk6ch&pli=1 Tuesday, June 19, 2012
72. This Presentation https://speakerdeck.com/u/rantav/p/ continuous-deployment Tuesday, June 19, 2012