where you build software in such a way that the software can be released to production at any time. -- Martin Fowler Code Build Test Release Deploy Monitor Idea Customer
on April 25, 2013, Quora released new versions of the site 46 times. This was a normal day for us.” - Quora / April 2013 “Deployment every 11.6 seconds, 1,079 max in one hour. 10,000 mean number of hosts per deployment, with 30,000 maximum” - Amazon.com / November 2012 “On the Google Consumer Surveys team, 8 minutes after you commit code it's live in production.” - Google / June 2014 “10+ deploys per day.” - John Allspaw / June 2009
handmade and vintage items. Etsy.com employs 500+ people and has grown to a community of more than 25 million members and 1,000,000+ online shops who generated 1.35$ billion of sales in 2013. extfiles.etsy.com/progress-report/2013-Etsy-Progress-Report.pdf Deploying etsy.com 40+ times every day by a team of 175+ committers supporting 60+ million unique monthly visitors. youtube.com/watch?v=4pVOXLvoUyE infoq.com/news/2014/03/etsy-deploy-50-times-a-day slideshare.net/beamrider9/continuous-deployment-at-etsy-a-tale-of-two-approaches etsy.com
accounts. 1+ million active retail partners worldwide. Between 100-150 services are accessed to build a page. fool.com/investing/general/2014/05/24/how-many-customers-does-amazon-have. aspx Deployment 1,079 max deploys per hour affecting 30,000 hosts simultaneous receiving a deployment. youtube.com/watch?v=dxk8b9rSKOo
billion hours of video per month. Two billion requests per day to the API, more than a dozen billion backend requests. Developers deploy when they want. • They also manage their own capacity and autoscaling. • And fix anything that breaks at 4am. Deploys a hundred times per day, without the use of Chef or Puppet, without a quality assurance department and without release engineers. infoq.com/presentations/netflix-architecture-resilience / QCon SF 2013 infoq.com/news/2013/06/netflix netflix.com
Before: monthly releases, 2 week qa cycle, late night incidents, 12 month lead time → After: bi-weekly release, <1 day qa, single-button rollback, 1 week lead time SAP.com
of software, serving 94 of Fortune 100 companies. Third largest company in software revenue in the world. en.wikipedia.org/wiki/HP_Software_Division Deployment 1000+ builds per day, 825,000+ change requests in a year, 100,000+ tests run daily. h30614.www3.hp.com/collateral/Barcelona2014/presentations/B5756.pdf hp.com
Before: 30 days per feature delivery, 300min prod build, 1440min full test → After: 7 days per feature delivery, 10min prod build, 360min full test Ticketmaster - youtube.com/watch?v=01gfNOwhnro Before: Bi-monthyl/Quarterly releases → After: 109 deployments in September Target.com - youtube.com/watch?v=exrjV9V9vhY 30+ APIs, 80 deployments per week, <10 incidents per month Devops Enterprise Summit
more frequent deployments • 8000x faster lead times than peers • 12x faster Mean Time to Recover (MTTR) • 50% faster market cap. growth State of Devops 2014 Report
time to restore service change fail rate How long is the delay between a request for a change, and a production system operating with that change implemented? How long does it take for an abnormal behavior in the system to be restored to the normal standard agreed way of operation? How many changes and features are being released to production in a fixed period of time? How often the system fails or service is disrupted?
with great signal to noise ratio. e.g. Google SRE notes & StackExchange Alerts ◦ standard procedures and checklists. e.g. AWS Operational Checklist ◦ practice recovery from system failures. e.g. Netflix downtime & Xbox downtime ◦ practice backup with restore to dev/test. e.g. Netflix Priam ◦ infrastructure as code & auto-healing. e.g. Antifragile Systems ◦ simplicity is prerequisite for reliability. e.g. Forrester Devops & Simple Made Easy ◦