Share
Embed
Transcript
72 slides
-
How much time does it take your code to meet
the users? Tuesday, June 19, 2012 -
How much time does it take your code to meet
the users? • More than a year? Tuesday, June 19, 2012 -
How much time does it take your code to meet
the users? • More than a year? • 6-12 months? Tuesday, June 19, 2012 -
How much time does it take your code to meet
the users? • More than a year? • 6-12 months? • 1-6 months? Tuesday, June 19, 2012 -
How much time does it take your code to meet
the users? • More than a year? • 6-12 months? • 1-6 months? • 2-4 weeks? Tuesday, June 19, 2012 -
How much time does it take your code to meet
the users? • More than a year? • 6-12 months? • 1-6 months? • 2-4 weeks? • 1-14 days? Tuesday, June 19, 2012 -
How much time does it take your code to meet
the users? • More than a year? • 6-12 months? • 1-6 months? • 2-4 weeks? • 1-14 days? • 1-24 hours? Tuesday, June 19, 2012 -
How much time does it take your code to meet
the users? • More than a year? • 6-12 months? • 1-6 months? • 2-4 weeks? • 1-14 days? • 1-24 hours? •10 minutes? Tuesday, June 19, 2012 -
How is it done at outbrain? svn ci -m "Implement
the super-sharp image scaler. #deloy:ImageServer #to:ny" Tuesday, June 19, 2012 -
For the business What is a startup? a startup is
always on a quest to find a product-market fit => You have to iterate fast ... before running out of money Tuesday, June 19, 2012 -
For the business • Feedback loop speed is important •
Developers have REPL • But what do Product Managers have? Tuesday, June 19, 2012 -
For the developers • What's used to be the case
before: • Inefficient Waits (wait for QA, wait for other features etc) • Inefficient Context Switch • Feature Delayed - b/c of other features • Big Changes – Big Problems. Tuesday, June 19, 2012 -
Fun Numbers • 5-50 deployments / day • 4800 tests
/ every commit • 1600 production tests / hour • 47 services • 100s of servers / 3 Datacenters • 5-12 minutes from commit to production Tuesday, June 19, 2012 -
In Five Easy Steps 1. Continuous Integration 2. Post Commit
Hook Tuesday, June 19, 2012 -
In Five Easy Steps 1. Continuous Integration 2. Post Commit
Hook 3. Deployment Script Tuesday, June 19, 2012 -
In Five Easy Steps 1. Continuous Integration 2. Post Commit
Hook 3. Deployment Script 4. Realtime Alerting (monitoring) Tuesday, June 19, 2012 -
In Five Easy Steps 1. Continuous Integration 2. Post Commit
Hook 3. Deployment Script 4. Realtime Alerting (monitoring) 5. Root Cause Analysis Tuesday, June 19, 2012 -
CD Themes Release is a Marketing decision Deployment is an
Engineering decision Tuesday, June 19, 2012 -
CD Themes Small changes reduce risks Kent Beck: You can
spill a bucket, but you cannot spill a hose Tuesday, June 19, 2012 -
CD Themes Deploy often, deploy fast Fast turnaround => Happy
customers ... and happy developers Tuesday, June 19, 2012 -
CD Themes Deploy when you are ready Do not have
to wait for other teams Tuesday, June 19, 2012 -
CD Themes Everyone cares about Everything • build • test
• quality • production • monitoring • business Tuesday, June 19, 2012 -
Focus on Production Measure everything Some even eliminate test, stg
etc Tuesday, June 19, 2012 -
Tools and Techniques • Trunk Stable • Continuous Integration •
Fully Automated Tests Tuesday, June 19, 2012 -
Tools and Techniques • Trunk Stable • Continuous Integration •
Fully Automated Tests • Infrastructure Automation Tuesday, June 19, 2012 -
Tools and Techniques • Trunk Stable • Continuous Integration •
Fully Automated Tests • Infrastructure Automation • Deployer Tuesday, June 19, 2012 -
Tools and Techniques • Trunk Stable • Continuous Integration •
Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments Tuesday, June 19, 2012 -
Tools and Techniques • Trunk Stable • Continuous Integration •
Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments • Servicization Tuesday, June 19, 2012 -
Tools and Techniques • Trunk Stable • Continuous Integration •
Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments • Servicization • Immune System Tuesday, June 19, 2012 -
Tools and Techniques • Trunk Stable • Continuous Integration •
Fully Automated Tests • Infrastructure Automation • Deployer • Gradual Deployments • Servicization • Immune System • Visibility Tuesday, June 19, 2012 -
Trunk Stable • Trunk always builds cleanly • (if broken
- commit freeze) • No SCM branches • But there are feature flags • Tests always green • Trunk may always be deployed (not just by you) • => You better have your features well tested • Forward and backwards compatibility (DB) Tuesday, June 19, 2012 -
Automated Testing • 4800 unit tests • They run in
5 minutes • 1600 production tests • Run hourly and after deployments Tuesday, June 19, 2012 -
Infrastructure Automation •Infrastructure as Code • Kickstart: install OS and
Chef • Chef: to configure host and install services • Glu: to install applications • Infrastructure code is in trunk Tuesday, June 19, 2012 -
Gradual Deployment • Deploy first Canary • Self Test •
Then deploy 1/3 of each DC in parallel • Self Test all • Continue until done Tuesday, June 19, 2012 -
Servicization • 47 different services • At least one replica
for each service • Everything either load-balanced (HAProxy) or queued (ActiveMQ) • Services talk REST Tuesday, June 19, 2012 -
Servicization What do we get from it? • Easy to
make small incremental changes • If there’s damage, it’s contained (memory leak, handle leak etc) • Rollback (if desired) is possible. Roll forward is preferred Tuesday, June 19, 2012 -
Servicization Challanges • Challenge: Find the correct balance: • Service
Coherency • Number of services that need operational maintenance • Interop complexity • Performance Tuesday, June 19, 2012 -
The Immune System • The line of defense will always
be broken => Multiple lines of defense Tuesday, June 19, 2012 -
The Immune System Coding • Code Review • Pairing •
Best Practices • Static analysis • Unit Test Tuesday, June 19, 2012 -
The Immune System Production Time • Nagios + Graphite •
KeyNote, pingdom • Test Production (selenium, APT, 1800 tests hourly) Tuesday, June 19, 2012 -
The Immune System Instrumentation • Self-Test • JMX • Performance
• metrics • speed4j Tuesday, June 19, 2012 -
The Immune System Monitoring KPIs • PVs, • 3 different
CTRs, • clicks, revenue • etc... Tuesday, June 19, 2012 -
Visibility • Everyone monitor the services • Deployment events on
graph • Yammer • Email • Changelog • svn, glu, yammer Tuesday, June 19, 2012 -
Outbrain’s Arsenal • Java: Tomcat, Spring, SpringMVC, Struts2 • JavaScript:
ExtJS and homemade • Data: MySQL, Cassandra, HDFS, MogileFS, Memcached • Message queues: ActiveMQ Tuesday, June 19, 2012 -
Outbrain’s Arsenal • Service multiplicity: HAProxy • Monitoring: Nagios, Keynote,
Pingdom • Infra: CentOS, Kickstart, Chef, Glu, RPM +YUM • Testing: JUnit, TeamCity, Selenium, Staging Tuesday, June 19, 2012 -
References • Why Continuous Deployment / Eric Ries http://www.startuplessonslearned.com/2009/06/why-continuous-deployment.html •
Continuous Deployment in five easy steps / Eric Ries http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html • Continuous Deployment at outbrain / Ran Tavory http://prettyprint.me/2011/01/24/continuous-deployment-at-outbrain/ • Deployment Infrastructure for Continuous Deployment / WealthFront http://eng.wealthfront.com/2010/05/deployment-infrastructure-for.html • Continuous Deployment presentation / Eishay Smith http://prezi.com/5zm8xplapff2/continuous-deployment/ • Quantum of Deployment / Etsy http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/ • Chrome Release Cycle / Anthony Laforge https://docs.google.com/present/view?id=dg63dpc6_4d7vkk6ch&pli=1 Tuesday, June 19, 2012