Small Batch Deployments 2019-08-10

99d4c0ed3c8027ea1c37b53d6441d042?s=47 sullis
August 10, 2019

Small Batch Deployments 2019-08-10

#deployments
#ContinuousDelivery

99d4c0ed3c8027ea1c37b53d6441d042?s=128

sullis

August 10, 2019
Tweet

Transcript

  1. Small Batch Deployments Sean Sullivan August 10, 2019

  2. Principles of Continuous Delivery continuousdelivery.com

  3. 1986: Fred Brooks “No Silver Bullet” paper Growing software organically,

    adding more and more function to systems as they are run, used, and tested. 

  4. 1999: Kent Beck * first edition, published October 1999 We

    need to control the development of software by making many small adjustments, not by making a few large adjustments, kind of like driving a car.
  5. 1999: Kent Beck * first edition, published October 1999 Big

    changes made all at once just don't work. Any problem is solved with a series of the smallest changes that make a difference.
  6. 2001: Agile Manifesto * published February 2001 Deliver working software

    frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. https://agilemanifesto.org/principles.html
  7. 2003: Poppendieck * book published May 2003 Remember the rule

    of small batches: If you integrate changes in small batches, it will be infinitely easier to detect and fix problems.
  8. 2004: Kent Beck * second edition, published November 2004 Software

    development has long delivered value in big chunks. Many teams make the problem worse by tending to respond to stress by making the chunks of value bigger, from deploying software less frequently to integrating less often.
  9. 2004: Kent Beck * second edition, published November 2004 The

    more things are deferred, the larger the chunk, the higher the risk. In contrast, the principle of flow suggests that for improvement, deploy smaller increments of value ever more frequently.
  10. 2004: Kent Beck * second edition, published November 2004 It's

    always tempting to make big changes in big steps. Momentous change taken all at once is dangerous.
  11. 2004: Kent Beck * second edition, published November 2004 Baby

    steps acknowledge that the overhead of small steps is much less than when a team wastefully recoils from aborted big changes.
  12. 2004: Kent Beck * second edition, published November 2004 Baby

    steps are expressed in practices like test-first programming, which proceeds one test at a time, and continuous integration, which integrates and tests a few hours' worth of changes at a time.
  13. 2004: Kent Beck * second edition, published November 2004 Big

    deployments have a high risk and high human and economic costs.
  14. 2004: Kent Beck * second edition, published November 2004 Daily

    Deployment Put new software into production every night. Any gap between what is on a programmer's desk and what is in production is a risk.
  15. 2004: Kent Beck * second edition, published November 2004 The

    deployment tools must be automated, including the ability to roll out incrementally and roll back in case of failure.
  16. 2004: Kent Beck * second edition, published November 2004 There

    are many barriers to deploying frequently. Some are psychological or social, like a deployment process so stressful that people don't want to go through it twice as often.
  17. 2004: Kent Beck * second edition, published November 2004 Whatever

    the barrier, working to remove it and then letting more frequent deployment come as a natural consequence will help you improve development.
  18. 2006: Poppendieck * book published September 2006 We have been

    conditioned to think that the large-batch, all-or-nothing approach to software development is good. It’s time to recondition ourselves to think that this is the worst possible approach to good software development.
  19. 2006: Poppendieck * book published September 2006 Work in small

    batches Reduce project size. Shorten release cycles. Stabilize. Repeat.
  20. 2007: Michael Nygard * first edition, published March 2007 One

    solution — the easy but harmful one — is to slow down the release calendar. Like going to the dentist less frequently because it hurts, this response to the problem can only exacerbate the issue.
  21. 2007: Michael Nygard * first edition, published March 2007 The

    right response is to reduce the effort needed, remove people from the process, and make the whole thing more automated and standardized.
  22. http://www.startuplessonslearned.com/2008/09/new-version-of-joel-test-draft.html Do you work in small batches? Just like in

    lean manufacturing, it's generally more efficient to drive down the batch size. I try to encourage engineers to check in anytime they have the software in a working state in their sandbox. And it's way easier to deploy small bits of code, since if something goes wrong, the problem is automatically localized and easy to revert. 2008: Eric Ries * blog post, September 2008
  23. 2009: Donald Reinertsen * published January 2009

  24. 2009: Donald Reinertsen * published January 2009 Reducing batch size

    reduces cycle time
  25. 2009: Donald Reinertsen * published January 2009 Reducing batch sizes

    reduces variability in flow
  26. 2009: Donald Reinertsen * published January 2009 Reducing batch sizes

    accelerates feedback
  27. 2009: Donald Reinertsen * published January 2009 Reducing batch size

    reduces risk
  28. 2009: Donald Reinertsen * published January 2009 Reducing batch size

    reduces overhead
  29. 2009: Donald Reinertsen * published January 2009 Large batches inherently

    lower motivation and urgency
  30. 2009: Donald Reinertsen * published January 2009 Large batches lead

    to even larger batches
  31. 2009: Donald Reinertsen * published January 2009 Smaller changes reduce

    debug complexity and make debugging more efficient
  32. 2009: Donald Reinertsen * published January 2009 Investments in infrastructure

    enable small batches. Good infrastructure reduces transaction costs.
  33. http://www.startuplessonslearned.com/2009/02/work-in-small-batches.html software should be designed, written, and deployed in small

    batches 2009: Eric Ries * blog post, February 2009
  34. http://timothyfitz.com/2009/02/08/continuous-deployment/ Continuous Deployment is simple: just ship your code to

    customers as often as possible. 2009: Timothy Fitz * blog post, February 2009
  35. 2009: Flickr * Velocity Conference, June 2009 https://www.youtube.com/watch?v=LdOe18KhtT4

  36. 2009: Flickr * Velocity Conference, June 2009 https://www.youtube.com/watch?v=LdOe18KhtT4

  37. https://omniti.com/seeds/online-application-deployment-reducing-risk.html 2010: Theo Schlossnagle * published March 2010 micro-releases: releases

    must never get too large instead amortize risk with small, controlled releases
  38. 2010: Facebook * Velocity Conference, June 2010 a culture of

    making very frequent small changes if you've only changed one thing at a time, it is really easy to figure out what broke the site https://www.youtube.com/watch?v=nEmJ_5UHs1g Robert Johnson
  39. 2010: Etsy * Velocity Conference, June 2010 https://www.kitchensoap.com/2010/06/24/ops-meta-metrics-velocity-2010-slides/ John Allspaw

  40. 2010: Etsy * Velocity Conference, June 2010 https://www.kitchensoap.com/2010/06/24/ops-meta-metrics-velocity-2010-slides/ John Allspaw

  41. * book published August 2010 2010: Continuous Delivery Imagine that

    you released frequently, so the delta between what is currently in production and the new release is small. If that were true, the risk of release would be greatly diminished
  42. * blog post, March 2011 2011: Wealthfront we ship to

    production more than 30 times a day http://eng.wealthfront.com/2011/03/12/lean-startup-stage-at-sxsw/
  43. * book published September 2011 2011: Lean Startup Small Batches

    at IMVU IMVU makes about fifty changes to its product every single day. Just as with the Toyota Production System, the key to being able to operate this quickly is to check for defects immediately, thus preventing bigger problems later.
  44. * Ruby Midwest, November 2011 2011: GitHub We deploy 10

    to 40 times a day. We like very short changes. If something breaks, you know exactly what breaks, really quickly. https://www.youtube.com/watch?v=qyz3jkOBbQY Zach Holman
  45. incremental deployment 2012: gilt.com deploy early, deploy often * Portland

    Oregon, January 2012
  46. Work in small batches. With a small batch size, you

    are reducing the amount of complexity that has to be dealt with at any one time by the people working on the batch. Break down large releases into small units of deployment 2012: Damon Edwards http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches-improve-flow/ * published March 2012
  47. each release is a rollback point small incremental releases actually

    help reduce risk 2012: Sam Newman * GeeCon, May 2012
  48. https://www.joelonsoftware.com/2012/07/09/software-inventory/ * published July 2012 2012: Joel Spolsky If you’re

    already shipping monthly, figure out how to ship weekly. Keep pushing the bar on more and more frequent deployment of smaller and smaller changes.
  49. 2012: PuppetConf https://twitter.com/RealGeneKim/status/251457175128530944 * tweet, September 2012

  50. 2012: Michael Nygard The longer you go between deployments, the

    longer you build up ‘inventory’, the more opportunities there are for defects. https://www.youtube.com/watch?v=Luskg9ES9qI * GOTO Aarhus conference, October 2012
  51. 2012: Etsy https://www.youtube.com/watch?v=JR-ccCTmMKY * GOTO Aarhus conference, October 2012

  52. How long would it take your organization to deploy a

    change that involved just one single line of code? Do you do this on a repeatable, reliable basis? * Berlin, October 2012 2012: Jez Humble https://www.youtube.com/watch?v=skLJuksCRTw
  53. 2012: Donald Reinertsen https://www.youtube.com/watch?v=cAMyGVSj9m4 * December 2012

  54. 2012: Donald Reinertsen https://www.youtube.com/watch?v=cAMyGVSj9m4 * December 2012

  55. Shipping is your company’s heartbeat 2013: Intercom * blog post,

    May 2013 Shipping cadence defines your company https://www.intercom.com/blog/shipping-is-your-companys-heartbeat/
  56. 2013: Intercom * blog post, May 2013 https://www.intercom.com/blog/shipping-is-your-companys-heartbeat/

  57. And deploying smaller sets of features more frequently reduces the

    number of changes per deployment, which is an inherent benefit of continuous delivery and helps mitigate risk by making it easier to identify and triage problems if things go south during a deployment. https://medium.com/netflix-techblog/deploying-the-netflix-api-79b6176cc3f0 2013: Netflix * published August 2013
  58. 2013: gilt.com continuous delivery is critical Every release is small.

    You know what's changed in production. If it doesn't work, rollback is simple * Surge conference, September 2013 https://www.youtube.com/watch?v=8-6azNVq2X0
  59. https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/ 2014: Spotify * published March 2014

  60. 2014: Limoncelli et al * book published September 2014 Small

    batches means doing a lot of small releases with a few features rather than a small number of large releases with lots of features. To lower overall risk, it’s better to do many small releases containing only a few features each.
  61. 2014: Limoncelli et al * book published September 2014 The

    small batches principle is counter-intuitive because there is a human tendency to avoid risky behavior. Deploying software in production involves risk; therefore businesses traditionally minimize the frequency of deployments.
  62. 2014: Limoncelli et al * book published September 2014 While

    this makes them feel better, they actually are shooting themselves in the foot because the deployments that are done are bigger and riskier, and the team doing them is out of practice by the time the next one rolls around.
  63. We love to deploy new code incrementally hundreds of times

    a day. And there's good reason for that: it's safer overall. Incremental deploys are easier to understand and fix than one gigantic deploy once a year. https://zachholman.com/talk/move-fast-break-nothing/ 2014: Github * published October 2014
  64. Break down your feature into commits that are safe to

    push to production https://www.infoq.com/presentations/cd-linkedin/ 2014: LinkedIn * QCon San Francisco, November 2014 Jason Toy
  65. https://tech.ticketmaster.com/2014/11/14/fear-and-paranoia-in-a-devops-world/ 2014: Ticketmaster * blog post, November 2014 The premise

    of frequent small production deployments is that we minimize the risk in each deployment by keeping the change set small
  66. https://codeascraft.com/2015/02/20/re-introducing-deployinator-now-as-a-gem/ 2015: Etsy We deploy code about 40 times per

    day. This allows us to push smaller changes we are confident about and experiment at a fast rate. Jayson Paul * published February 2015
  67. in practice, large-impact, high- risk deployments end up happening infrequently

    due to understandable fear unfortunately, this means that our changes build up between releases 2015: Building Microservices * book published February 2015
  68. the bigger the delta between releases, the higher the risk

    that we’ll get something wrong * book published February 2015 2015: Building Microservices
  69. throughout the book, I promote the need to make small,

    incremental changes one of the key drivers is to understand the impact of each alteration * book published February 2015 2015: Building Microservices
  70. 2015: Lyft * DockerCon, June 2015 https://www.youtube.com/watch?v=iC2T3gJsB0g Matthew Leventi

  71. 2015: Gene Kim * published July 8 2015 Deploy smaller

    changes more frequently https://blog.newrelic.com/technology/gene-kim-devops-tips/ Practice deploying smaller changes, which dramatically reduces risk and improves MTTR
  72. 2015: gilt.com * QCon London, July 2015 simpler and easier

    deployments and rollbacks Yoni Goldberg teams can decide when they want to deploy deploy multiple times per day
  73. 2015: gilt.com * QCon London, July 2015 database schema changes

    are required to be incremental Yoni Goldberg database schema changes are released separately from service code
  74. 2016: Zalando * published February 2016 Strive for very short

    release cycles, optimally deploying daily Small releases tend to have fewer bugs. https://github.com/zalando/engineering-principles/blob/master/README.md
  75. 2016: Timothy Fitz * Agile India, March 2016 https://www.youtube.com/watch?v=qu89JWEshlU

  76. 2016: Instagram * SREcon, April 2016 Michael Gorven https://www.youtube.com/watch?v=XNJ30Vy5xSg

  77. 2016: Instagram * SREcon, April 2016 Michael Gorven https://www.youtube.com/watch?v=XNJ30Vy5xSg

  78. 2016: Instagram * SREcon, April 2016 Michael Gorven https://www.youtube.com/watch?v=XNJ30Vy5xSg

  79. Simple releases are generally better than complicated releases. 2016: Google

    * book published April 2016
  80. It is much easier to measure and understand the impact

    of a single change rather than a batch of changes released simultaneously. 2016: Google * book published April 2016
  81. If the release is performed in smaller batches, we can

    move faster with more confidence because each code change can be understood in isolation in the larger system. 2016: Google * book published April 2016
  82. 2016: Netflix * DevOpsDays Rockies, April 2016 https://www.youtube.com/watch?v=HmM4V33ReCw Dave Hahn

  83. 2016: Shopify * RailsConf, May 2016 https://www.youtube.com/watch?v=HNH7El_BEsw Kat Drobnjakovic We

    deploy Shopify an average 30 times each day.
  84. 2016: Danilo Sato * DevOps Singapore, May 2016 https://www.youtube.com/watch?v=eBRspUcDCTc

  85. 2016: Danilo Sato * DevOps Singapore, May 2016 https://www.youtube.com/watch?v=eBRspUcDCTc

  86. 2016: Uber * published June 2016 https://eng.uber.com/micro-deploy/

  87. 2016: Uber * published June 2016 https://eng.uber.com/micro-deploy/ Engineers write code,

    review it, test it, and put it into production the same day. Each week in 2016, thousands of engineers push to prod several thousand service builds
  88. * book published June 2016 2016: Microservice Architecture reducing the

    size of the deployment unit a notion we call “batch size reduction”
  89. emphasis on small batch sizes 2016: DevOps Handbook the principle

    of small batch sizes also applies to code reviews * book published October 2016
  90. make regular, small, incremental changes 2016: AWS re:Invent * Las

    Vegas - December 1, 2016 Tim DiLauro AWS Solutions Architect https://www.youtube.com/watch?v=8e2zCHx-K3M which allows you to ultimately move faster
  91. 18,000 production deployments a year 2017: Twilio * SaaStr conference,

    interview February 8, 2017 https://www.youtube.com/watch?v=h6KtHq2-6F0 Jeff Lawson Twilio CEO
  92. one practice that works very well operationally: deploying small units

    of code to production on a regular basis. 2017: Skyliner * published February 9, 2017 https://blog.skyliner.io/ship-small-diffs-741308bec0d1
  93. We cannot prevent all production problems. They will happen. And

    when they do, we’re better off when we’ve been pushing small changesets. 2017: Skyliner * published February 9, 2017 https://blog.skyliner.io/ship-small-diffs-741308bec0d1
  94. 2017: Uber * SRECon, March 2017 Sebastian Yates https://www.youtube.com/watch?v=4A4SLk1PRvg

  95. 2017: Uber * SRECon, March 2017 Sebastian Yates https://www.youtube.com/watch?v=4A4SLk1PRvg [service

    owners are] able to deploy at any schedule they like 40 deploys a day
  96. Our aim is always to move faster with smaller steps.

    2017: Intercom * blog post, June 2017 https://www.intercom.com/blog/moving-faster-with-smaller-steps/ Rather than shipping new product quarterly, monthly or weekly, we deploy new features to our customers up to 50 times a day.
  97. the heart of doing evolutionary architecture is to make small

    changes * book published September 2017 2017: Building Evolutionary Architectures
  98. an architecture that allows small, incremental changes is easier to

    evolve because developers have a smaller scope of change 2017: Building Evolutionary Architectures * book published September 2017
  99. with improved engineering practices comes decreased risk 2017: Building Evolutionary

    Architectures * book published September 2017
  100. Melanie Cebula 2017: Airbnb * FutureStack New York, September 2017

    https://www.youtube.com/watch?v=N1BWMW9NEQc
  101. 2017: Randy Shoup * DevOpsDays Cuba, October 2017 https://www.slideshare.net/RandyShoup/evolving-architecture-and-organization-lessons-from-google-and-ebay

  102. 2017: Randy Shoup * DevOpsDays Cuba, October 2017 https://www.slideshare.net/RandyShoup/evolving-architecture-and-organization-lessons-from-google-and-ebay

  103. our engineers are now deploying code 2000 times per day

    2017: Expedia * AWS re:Invent, November 2017 https://www.youtube.com/watch?v=1IxDLeFQKPk
  104. The smaller your releases the better — since the likelihood

    of a problem increases with every additional change. Furthermore, when something does go wrong, it is much easier to find the offending change from within a small batch size than a large one. 2017: Nextdoor * blog post, December 2017 https://engblog.nextdoor.com/3-hard-lessons-from-scaling-continuous-deployment-to-a-monolith-with-70-engineers-99fb6dfe3c38
  105. For years, IT teams in medium to large enterprises released

    updates to applications in large batches with the belief that this is more efficient, less impactful, and a better return on the investment. Because we believed this, we built processes, compliance checks, architecture guidelines, and measures to support the large sized, complex releases. But this belief is incorrect. 2017: Salesforce * blog post, December 2017 https://www.salesforce.com/blog/2017/12/smaller-batches-improves-application-roi.html
  106. * second edition, published January 2018 designing for deployment gives

    you the ability to make large changes in small steps 2018: Release It!
  107. * second edition, published January 2018 to be successful, your

    software will be deployed early and often 2018: Release It!
  108. * second edition, published January 2018 smaller, easier deployments mean

    you can make big changes over a series of small steps 2018: Release It!
  109. * DevOpsDays NYC, January 18 2018 2018: Thomas Limoncelli https://www.youtube.com/watch?v=3uVdRpjn8iA

  110. we deploy new code at a pace that others deem

    crazy or reckless (we ship new versions of Intercom almost 100 times per day) 2018: Intercom * blog post, February 2018 https://www.intercom.com/blog/run-less-software/
  111. 2018: Accelerate * book published March 2018

  112. 2018: Accelerate * book published March 2018

  113. Deploy in Small Batches, and Do It Often 2018: rollout.io

    * published March 13 2018 https://rollout.io/blog/deploy-to-production-5-tips-make-smoother/ Amplify feedback by increasing the number of times you deploy. It might sound risky, but it’s not. The fewer changes you make, the easier it will be to know what’s wrong.
  114. * tweet, April 2018 https://twitter.com/bridgetkromhout/status/983739341074845697 2018: Bridget Kromhout

  115. 2018: Michele Titolo Microservice deployments should be small. If you

    have small services, you are probably making small changes. * QCon New York, June 2018
  116. 2018: Michele Titolo One of the benefits of small changes

    is that you can release frequently * QCon New York, June 2018
  117. 2018: Flow Commerce * QCon New York, June 2018 Mike

    Bryzek
  118. 2018: Flow Commerce * QCon New York, June 2018 Mike

    Bryzek
  119. 2018: Charity Majors https://charity.wtf/2018/08/19/shipping-software-should-not-be-scary/ * published August 2018 suffice it

    to say that we now know that smaller and more frequent changes are much safer than larger and less frequent changes ship early, ship often, ship smaller change sets
  120. 2018: Hulu * QCon San Francisco, November 2018 https://www.youtube.com/watch?v=3vyLcbI16P8 Andrew

    McVeigh Granular deployment At Hulu, it was not uncommon for us to deploy 100 times in a day.
  121. 2018: Slack * QCon San Francisco, November 2018 https://www.youtube.com/watch?v=mwsfNio2Dho Holly

    Allen
  122. 2018: Atlassian * YOW! CTO Summit Sydney, November 28 2018

    https://www.youtube.com/watch?v=b-xRHOW4sEI Ben Mackie
  123. 2019: Financial Times * QCon London, March 2019 https://www.infoq.com/presentations/microservices-financial-times Sarah

    Wells
  124. 2019: Jez Humble * tweet, March 2019 https://twitter.com/jezhumble/status/1106297185119797248

  125. 2019: Starbucks * tweet, April 10 2019 https://twitter.com/davidbrunelle/status/1116169369879363584

  126. 2019: Starbucks * tweet, April 10 2019 https://twitter.com/davidbrunelle/status/1116169370911141888

  127. 2019: Charity Majors * tweet, April 15 2019 https://twitter.com/mipsytipsy/status/1117858831294320640

  128. 2019: Charity Majors * published May 1, 2019 https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/ Fear

    of deploys is the ultimate technical debt Do not batch up multiple people’s commits into a deploy Good teams ship often
  129. 2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8

  130. 2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8

  131. 2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8

  132. 2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8

  133. 2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8 Number

    of meetings per deploy: drive to zero You don’t have time to have 10 meetings a day for the 10 times you deploy today.
  134. 2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8 If

    you have a competitor that is updating 10 times a day and you are updating every 2 weeks, then you will fall behind.
  135. 2019: Jez Humble * CraftConf, May 9 2019 https://www.youtube.com/watch?v=CN6uhzNM4eA Working

    in small batches. Taking big ideas and splitting them into small chunks so they can be delivered in less than a week. And released frequently.
  136. 2019: Twilio * CraftConf, May 10 2019 https://www.youtube.com/watch?v=BIkDeTEdbSo Justin Kitagawa

  137. 2019: Bert Jan Schrijver * Devoxx UK, May 10 2019

    https://www.youtube.com/watch?v=tcQNK4R1tfs
  138. 2019: Dave Farley * JAX DevOps London, May 15 2019

    https://www.youtube.com/watch?v=NiTxzYA_qRQ
  139. 2019: Dave Farley * JAX DevOps London, May 15 2019

    https://www.youtube.com/watch?v=NiTxzYA_qRQ We could decide to write our software in big pieces and deploy infrequently. But that is a low quality solution. Each piece is complicated and risky.
  140. 2019: Dave Farley * JAX DevOps London, May 15 2019

    Or we could decide to break our problem down into small little pieces. Each piece is smaller, and simpler, and lower risk. https://www.youtube.com/watch?v=NiTxzYA_qRQ
  141. 2019: wix.com * Wix meetup, May 23 2019 https://www.youtube.com/watch?v=E_k1AteNWJI

  142. 2019: Wayfair * published June 17 2019 https://tech.wayfair.com/2019/06/history-of-integrator-scaling-software-deployment-automation/ On an

    average day, Wayfair deploys over 200 changes to its monolith, but on peak days (prepping for holiday and the like) we deploy over 500 unique changes
  143. 2019: Seth Vargo * DevOpsDays Amsterdam, June 28 2019 https://www.youtube.com/watch?v=SR6AvtsJmU8

    implement gradual change small changes isolated changes
  144. 2019: Twilio * Signal Conference, August 6 2019 https://www.youtube.com/watch?v=TTo37IRoMz4 one

    of the ways that we measure our progress [...] is our ability to release code to production we are on pace this year to release to production 125,000 times Jeff Lawson Twilio CEO
  145. Make It So small batches of work frequent deployments

  146. None