Small Batch Deployments 2019-06-04

99d4c0ed3c8027ea1c37b53d6441d042?s=47 sullis
June 04, 2019

Small Batch Deployments 2019-06-04

June 4, 2019

#deployments
#ContinuousDelivery

99d4c0ed3c8027ea1c37b53d6441d042?s=128

sullis

June 04, 2019
Tweet

Transcript

  1. 3.

    1986: Fred Brooks “No Silver Bullet” paper Growing software organically,

    adding more and more function to systems as they are run, used, and tested. 

  2. 4.

    1999: Kent Beck * first edition, published October 1999 We

    need to control the development of software by making many small adjustments, not by making a few large adjustments, kind of like driving a car.
  3. 5.

    1999: Kent Beck * first edition, published October 1999 Big

    changes made all at once just don't work. Any problem is solved with a series of the smallest changes that make a difference.
  4. 6.

    2001: Agile Manifesto * published February 2001 Deliver working software

    frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. https://agilemanifesto.org/principles.html
  5. 7.

    2003: Poppendieck * book published May 2003 Remember the rule

    of small batches: If you integrate changes in small batches, it will be infinitely easier to detect and fix problems.
  6. 8.

    2004: Kent Beck * second edition, published November 2004 Software

    development has long delivered value in big chunks. Many teams make the problem worse by tending to respond to stress by making the chunks of value bigger, from deploying software less frequently to integrating less often.
  7. 9.

    2004: Kent Beck * second edition, published November 2004 The

    more things are deferred, the larger the chunk, the higher the risk. In contrast, the principle of flow suggests that for improvement, deploy smaller increments of value ever more frequently.
  8. 10.

    2004: Kent Beck * second edition, published November 2004 It's

    always tempting to make big changes in big steps. Momentous change taken all at once is dangerous.
  9. 11.

    2004: Kent Beck * second edition, published November 2004 Baby

    steps acknowledge that the overhead of small steps is much less than when a team wastefully recoils from aborted big changes.
  10. 12.

    2004: Kent Beck * second edition, published November 2004 Baby

    steps are expressed in practices like test-first programming, which proceeds one test at a time, and continuous integration, which integrates and tests a few hours' worth of changes at a time.
  11. 13.

    2004: Kent Beck * second edition, published November 2004 Big

    deployments have a high risk and high human and economic costs.
  12. 14.

    2004: Kent Beck * second edition, published November 2004 Daily

    Deployment Put new software into production every night. Any gap between what is on a programmer's desk and what is in production is a risk.
  13. 15.

    2004: Kent Beck * second edition, published November 2004 The

    deployment tools must be automated, including the ability to roll out incrementally and roll back in case of failure.
  14. 16.

    2004: Kent Beck * second edition, published November 2004 There

    are many barriers to deploying frequently. Some are psychological or social, like a deployment process so stressful that people don't want to go through it twice as often.
  15. 17.

    2004: Kent Beck * second edition, published November 2004 Whatever

    the barrier, working to remove it and then letting more frequent deployment come as a natural consequence will help you improve development.
  16. 18.

    2006: Poppendieck * book published September 2006 We have been

    conditioned to think that the large-batch, all-or-nothing approach to software development is good. It’s time to recondition ourselves to think that this is the worst possible approach to good software development.
  17. 19.

    2006: Poppendieck * book published September 2006 Work in small

    batches Reduce project size. Shorten release cycles. Stabilize. Repeat.
  18. 20.

    2007: Michael Nygard * first edition, published March 2007 One

    solution — the easy but harmful one — is to slow down the release calendar. Like going to the dentist less frequently because it hurts, this response to the problem can only exacerbate the issue.
  19. 21.

    2007: Michael Nygard * first edition, published March 2007 The

    right response is to reduce the effort needed, remove people from the process, and make the whole thing more automated and standardized.
  20. 22.

    http://www.startuplessonslearned.com/2008/09/new-version-of-joel-test-draft.html Do you work in small batches? Just like in

    lean manufacturing, it's generally more efficient to drive down the batch size. I try to encourage engineers to check in anytime they have the software in a working state in their sandbox. And it's way easier to deploy small bits of code, since if something goes wrong, the problem is automatically localized and easy to revert. 2008: Eric Ries * blog post, September 2008
  21. 24.

    http://timothyfitz.com/2009/02/08/continuous-deployment/ Continuous Deployment is simple: just ship your code to

    customers as often as possible. 2009: Timothy Fitz * blog post, February 2009
  22. 28.

    2010: Facebook * Velocity Conference, June 2010 a culture of

    making very frequent small changes if you've only changed one thing at a time, it is really easy to figure out what broke the site https://www.youtube.com/watch?v=nEmJ_5UHs1g Robert Johnson
  23. 31.

    * book published August 2010 2010: Continuous Delivery Imagine that

    you released frequently, so the delta between what is currently in production and the new release is small. If that were true, the risk of release would be greatly diminished
  24. 32.

    * blog post, March 2011 2011: Wealthfront we ship to

    production more than 30 times a day http://eng.wealthfront.com/2011/03/12/lean-startup-stage-at-sxsw/
  25. 33.

    * book published September 2011 2011: Lean Startup Small Batches

    at IMVU IMVU makes about fifty changes to its product every single day. Just as with the Toyota Production System, the key to being able to operate this quickly is to check for defects immediately, thus preventing bigger problems later.
  26. 34.

    * Ruby Midwest, November 2011 2011: GitHub We deploy 10

    to 40 times a day. We like very short changes. If something breaks, you know exactly what breaks, really quickly. https://www.youtube.com/watch?v=qyz3jkOBbQY Zach Holman
  27. 36.

    Work in small batches. With a small batch size, you

    are reducing the amount of complexity that has to be dealt with at any one time by the people working on the batch. Break down large releases into small units of deployment 2012: Damon Edwards http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches-improve-flow/ * published March 2012
  28. 37.

    each release is a rollback point small incremental releases actually

    help reduce risk 2012: Sam Newman * GeeCon, May 2012
  29. 38.

    https://www.joelonsoftware.com/2012/07/09/software-inventory/ * published July 2012 2012: Joel Spolsky If you’re

    already shipping monthly, figure out how to ship weekly. Keep pushing the bar on more and more frequent deployment of smaller and smaller changes.
  30. 40.

    2012: Michael Nygard The longer you go between deployments, the

    longer you build up ‘inventory’, the more opportunities there are for defects. https://www.youtube.com/watch?v=Luskg9ES9qI * GOTO Aarhus conference, October 2012
  31. 42.

    How long would it take your organization to deploy a

    change that involved just one single line of code? Do you do this on a repeatable, reliable basis? * Berlin, October 2012 2012: Jez Humble https://www.youtube.com/watch?v=skLJuksCRTw
  32. 45.

    And deploying smaller sets of features more frequently reduces the

    number of changes per deployment, which is an inherent benefit of continuous delivery and helps mitigate risk by making it easier to identify and triage problems if things go south during a deployment. https://medium.com/netflix-techblog/deploying-the-netflix-api-79b6176cc3f0 2013: Netflix * published August 2013
  33. 46.

    2013: gilt.com continuous delivery is critical Every release is small.

    You know what's changed in production. If it doesn't work, rollback is simple * Surge conference, September 2013 https://www.youtube.com/watch?v=8-6azNVq2X0
  34. 48.

    2014: Limoncelli et al * book published September 2014 Small

    batches means doing a lot of small releases with a few features rather than a small number of large releases with lots of features. To lower overall risk, it’s better to do many small releases containing only a few features each.
  35. 49.

    2014: Limoncelli et al * book published September 2014 The

    small batches principle is counter-intuitive because there is a human tendency to avoid risky behavior. Deploying software in production involves risk; therefore businesses traditionally minimize the frequency of deployments.
  36. 50.

    2014: Limoncelli et al * book published September 2014 While

    this makes them feel better, they actually are shooting themselves in the foot because the deployments that are done are bigger and riskier, and the team doing them is out of practice by the time the next one rolls around.
  37. 51.

    We love to deploy new code incrementally hundreds of times

    a day. And there's good reason for that: it's safer overall. Incremental deploys are easier to understand and fix than one gigantic deploy once a year. https://zachholman.com/talk/move-fast-break-nothing/ 2014: Github * published October 2014
  38. 52.

    Break down your feature into commits that are safe to

    push to production https://www.infoq.com/presentations/cd-linkedin/ 2014: LinkedIn * QCon San Francisco, November 2014 Jason Toy
  39. 53.

    https://tech.ticketmaster.com/2014/11/14/fear-and-paranoia-in-a-devops-world/ 2014: Ticketmaster * blog post, November 2014 The premise

    of frequent small production deployments is that we minimize the risk in each deployment by keeping the change set small
  40. 54.

    https://codeascraft.com/2015/02/20/re-introducing-deployinator-now-as-a-gem/ 2015: Etsy We deploy code about 40 times per

    day. This allows us to push smaller changes we are confident about and experiment at a fast rate. Jayson Paul * published February 2015
  41. 55.

    in practice, large-impact, high- risk deployments end up happening infrequently

    due to understandable fear unfortunately, this means that our changes build up between releases 2015: Building Microservices * book published February 2015
  42. 56.

    the bigger the delta between releases, the higher the risk

    that we’ll get something wrong * book published February 2015 2015: Building Microservices
  43. 57.

    throughout the book, I promote the need to make small,

    incremental changes one of the key drivers is to understand the impact of each alteration * book published February 2015 2015: Building Microservices
  44. 59.

    2015: gilt.com * QCon London, July 2015 simpler and easier

    deployments and rollbacks Yoni Goldberg teams can decide when they want to deploy deploy multiple times per day
  45. 60.

    2015: gilt.com * QCon London, July 2015 database schema changes

    are required to be incremental Yoni Goldberg database schema changes are released separately from service code
  46. 61.

    2016: Zalando * published February 2016 Strive for very short

    release cycles, optimally deploying daily Small releases tend to have fewer bugs. https://github.com/zalando/engineering-principles/blob/master/README.md
  47. 67.

    It is much easier to measure and understand the impact

    of a single change rather than a batch of changes released simultaneously. 2016: Google * book published April 2016
  48. 68.

    If the release is performed in smaller batches, we can

    move faster with more confidence because each code change can be understood in isolation in the larger system. 2016: Google * book published April 2016
  49. 74.

    2016: Uber * published June 2016 https://eng.uber.com/micro-deploy/ Engineers write code,

    review it, test it, and put it into production the same day. Each week in 2016, thousands of engineers push to prod several thousand service builds
  50. 75.

    * book published June 2016 2016: Microservice Architecture reducing the

    size of the deployment unit a notion we call “batch size reduction” reducing the size or scope of the problem
  51. 76.

    emphasis on small batch sizes 2016: DevOps Handbook the principle

    of small batch sizes also applies to code reviews * book published October 2016
  52. 77.

    make regular, small, incremental changes 2016: AWS re:Invent * Las

    Vegas - December 1, 2016 Tim DiLauro AWS Solutions Architect https://www.youtube.com/watch?v=8e2zCHx-K3M which allows you to ultimately move faster
  53. 79.

    2017: Uber * SRECon, March 2017 Sebastian Yates https://www.youtube.com/watch?v=4A4SLk1PRvg [service

    owners are] able to deploy at any schedule they like 40 deploys a day
  54. 80.

    the heart of doing evolutionary architecture is to make small

    changes * book published September 2017 2017: Building Evolutionary Architectures
  55. 81.

    an architecture that allows small, incremental changes is easier to

    evolve because developers have a smaller scope of change 2017: Building Evolutionary Architectures * book published September 2017
  56. 83.

    Melanie Cebula 2017: Airbnb * FutureStack New York, September 2017

    https://www.youtube.com/watch?v=N1BWMW9NEQc
  57. 86.

    For years, IT teams in medium to large enterprises released

    updates to applications in large batches with the belief that this is more efficient, less impactful, and a better return on the investment. Because we believed this, we built processes, compliance checks, architecture guidelines, and measures to support the large sized, complex releases. But this belief is incorrect. 2017: Salesforce * blog post, December 2017 https://www.salesforce.com/blog/2017/12/smaller-batches-improves-application-roi.html
  58. 87.

    * second edition, published January 2018 designing for deployment gives

    you the ability to make large changes in small steps 2018: Release It!
  59. 88.

    * second edition, published January 2018 to be successful, your

    software will be deployed early and often smaller, easier deployments mean you can make big changes over a series of small steps 2018: Release It!
  60. 92.

    2018: Michele Titolo Microservice deployments should be small. If you

    have small services, you are probably making small changes. * QCon New York, June 2018
  61. 93.

    2018: Michele Titolo One of the benefits of small changes

    is that you can release frequently * QCon New York, June 2018
  62. 96.

    2018: Charity Majors https://charity.wtf/2018/08/19/shipping-software-should-not-be-scary/ * published August 2018 suffice it

    to say that we now know that smaller and more frequent changes are much safer than larger and less frequent changes ship early, ship often, ship smaller change sets
  63. 97.

    2018: Hulu * QCon San Francisco, November 2018 https://www.youtube.com/watch?v=3vyLcbI16P8 Andrew

    McVeigh Granular deployment At Hulu, it was not uncommon for us to deploy 100 times in a day.
  64. 104.

    2019: Charity Majors * published May 1, 2019 https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/ Fear

    of deploys is the ultimate technical debt Do not batch up multiple people’s commits into a deploy Good teams ship often
  65. 109.

    2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8 Number

    of meetings per deploy: drive to zero You don’t have time to have 10 meetings a day for the 10 times you deploy today.
  66. 110.

    2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8 If

    you have a competitor that is updating 10 times a day and you are updating every 2 weeks, then you will fall behind.
  67. 111.

    2019: Jez Humble * CraftConf, May 9 2019 https://www.youtube.com/watch?v=CN6uhzNM4eA Working

    in small batches. Taking big ideas and splitting them into small chunks so they can be delivered in less than a week. And released frequently.
  68. 113.

    2019: Dave Farley * JAX DevOps London, May 15 2019

    https://www.youtube.com/watch?v=NiTxzYA_qRQ
  69. 114.

    2019: Dave Farley * JAX DevOps London, May 15 2019

    https://www.youtube.com/watch?v=NiTxzYA_qRQ We could decide to write our software in big pieces and deploy infrequently. But that is a low quality solution. Each piece is complicated and risky.
  70. 115.

    2019: Dave Farley * JAX DevOps London, May 15 2019

    Or we could decide to break our problem down into small little pieces. Each piece is smaller, and simpler, and lower risk. https://www.youtube.com/watch?v=NiTxzYA_qRQ
  71. 117.