$30 off During Our Annual Pro Sale. View Details »

Small batch deployments - 2019-05-17

Small batch deployments - 2019-05-17

#ContinuousDelivery
#Deployments
#DevOps

sullis

May 17, 2019
Tweet

More Decks by sullis

Other Decks in Programming

Transcript

  1. Small Batch Deployments
    Sean Sullivan
    May 17, 2019

    View Slide

  2. Principles of Continuous Delivery
    continuousdelivery.com

    View Slide

  3. 1986: Fred Brooks
    “No Silver Bullet” paper
    Growing software organically,
    adding more and more
    function to systems
    as they are
    run, used, and tested. 


    View Slide

  4. 1999: Kent Beck
    * first edition, published October 1999
    We need to control the
    development of software by
    making many small adjustments,
    not by making a few
    large adjustments,
    kind of like driving a car.

    View Slide

  5. 1999: Kent Beck
    * first edition, published October 1999
    Big changes made all at once
    just don't work.
    Any problem is solved with a
    series of the smallest changes
    that make a difference.

    View Slide

  6. 2003: Poppendieck
    * book published May 2003
    Remember the rule of small batches:
    If you integrate changes in
    small batches,
    it will be infinitely easier to
    detect and fix problems.

    View Slide

  7. 2004: Kent Beck
    * second edition, published November 2004
    Software development has long
    delivered value in big chunks.
    Many teams make the problem worse by
    tending to respond to stress by making
    the chunks of value bigger,
    from deploying software less frequently
    to integrating less often.

    View Slide

  8. 2004: Kent Beck
    * second edition, published November 2004
    The more things are deferred,
    the larger the chunk, the higher the risk.
    In contrast, the principle of flow suggests
    that for improvement,
    deploy smaller increments of value
    ever more frequently.

    View Slide

  9. 2004: Kent Beck
    * second edition, published November 2004
    It's always tempting
    to make big changes in big steps.
    Momentous change taken
    all at once is dangerous.

    View Slide

  10. 2004: Kent Beck
    * second edition, published November 2004
    Baby steps acknowledge that the
    overhead of small steps is much less
    than when a team wastefully recoils
    from aborted big changes.

    View Slide

  11. 2004: Kent Beck
    * second edition, published November 2004
    Baby steps are expressed in practices
    like test-first programming, which
    proceeds one test at a time, and
    continuous integration, which
    integrates and tests a few hours'
    worth of changes at a time.

    View Slide

  12. 2004: Kent Beck
    * second edition, published November 2004
    Big deployments have a high risk and
    high human and economic costs.

    View Slide

  13. 2004: Kent Beck
    * second edition, published November 2004
    Daily Deployment
    Put new software into production
    every night. Any gap between what is
    on a programmer's desk and what is
    in production is a risk.

    View Slide

  14. 2004: Kent Beck
    * second edition, published November 2004
    The deployment tools must be
    automated, including the
    ability to roll out incrementally
    and roll back in case of failure.

    View Slide

  15. 2004: Kent Beck
    * second edition, published November 2004
    There are many barriers
    to deploying frequently.
    Some are psychological or social, like a
    deployment process so stressful that
    people don't want to go through it twice
    as often.

    View Slide

  16. 2004: Kent Beck
    * second edition, published November 2004
    Whatever the barrier, working to
    remove it and then letting
    more frequent deployment
    come as a natural consequence will
    help you improve development.

    View Slide

  17. 2006: Poppendieck
    * book published September 2006
    Work in small batches
    Reduce project size.
    Shorten release cycles.
    Stabilize.
    Repeat.

    View Slide

  18. 2007: Michael Nygard
    * first edition, published March 2007
    One solution — the easy but harmful one
    — is to slow down the release calendar.
    Like going to the dentist less frequently
    because it hurts, this response to the
    problem can only exacerbate the issue.

    View Slide

  19. 2007: Michael Nygard
    * first edition, published March 2007
    The right response is to
    reduce the effort needed,
    remove people from the process,
    and make the whole thing
    more automated and standardized.

    View Slide

  20. http://www.startuplessonslearned.com/2009/02/work-in-small-batches.html
    software should be designed, written,
    and deployed in small batches
    2009: Eric Ries
    * February 2009

    View Slide

  21. 2009: Flickr
    * Velocity Conference, June 2009
    https://www.youtube.com/watch?v=LdOe18KhtT4

    View Slide

  22. 2009: Flickr
    * Velocity Conference, June 2009
    https://www.youtube.com/watch?v=LdOe18KhtT4

    View Slide

  23. https://omniti.com/seeds/online-application-deployment-reducing-risk.html
    2010: Theo Schlossnagle
    * published March 2010
    micro-releases:
    releases must never get too large
    instead amortize risk
    with small, controlled releases

    View Slide

  24. 2010: Facebook
    * Velocity Conference, June 2010
    a culture of making very frequent small changes
    if you've only changed one thing at a time, it is
    really easy to figure out what broke the site
    https://www.youtube.com/watch?v=nEmJ_5UHs1g
    Robert Johnson

    View Slide

  25. 2010: Etsy
    * Velocity Conference, June 2010
    https://www.kitchensoap.com/2010/06/24/ops-meta-metrics-velocity-2010-slides/
    John Allspaw

    View Slide

  26. 2010: Etsy
    * Velocity Conference, June 2010
    https://www.kitchensoap.com/2010/06/24/ops-meta-metrics-velocity-2010-slides/
    John Allspaw

    View Slide

  27. * book published August 2010
    2010: Continuous Delivery
    Imagine that you released frequently,
    so the delta between what is
    currently in production
    and the new release is small.
    If that were true,
    the risk of release would be
    greatly diminished

    View Slide

  28. incremental
    deployment
    2012: gilt.com
    deploy early,
    deploy often
    * Portland Oregon, January 2012

    View Slide

  29. Work in small batches.
    With a small batch size, you are reducing the amount
    of complexity that has to be dealt with at any one time
    by the people working on the batch.
    Break down large releases into
    small units of deployment
    2012: Damon Edwards
    http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches-improve-flow/
    * published March 2012

    View Slide

  30. each release is a
    rollback point
    small incremental releases
    actually help
    reduce risk
    2012: Sam Newman
    * GeeCon, May 2012

    View Slide

  31. https://www.joelonsoftware.com/2012/07/09/software-inventory/
    * published July 2012
    2012: Joel Spolsky
    If you’re already shipping monthly,
    figure out how to ship weekly.
    Keep pushing the bar on more and
    more frequent deployment of
    smaller and smaller changes.

    View Slide

  32. 2012: PuppetConf
    https://twitter.com/RealGeneKim/status/251457175128530944
    * tweet, September 2012

    View Slide

  33. How long would it take
    your organization to
    deploy a change that involved
    just one single line of code?
    Do you do this on a repeatable,
    reliable basis?
    * October 2012
    2012: Jez Humble
    https://www.youtube.com/watch?v=skLJuksCRTw

    View Slide

  34. 2012: Etsy
    https://www.youtube.com/watch?v=JR-ccCTmMKY
    * GOTO Aarhus conference, October 2012

    View Slide

  35. 2012: Don Reinertsen
    https://www.youtube.com/watch?v=cAMyGVSj9m4
    * December 2012

    View Slide

  36. 2012: Don Reinertsen
    https://www.youtube.com/watch?v=cAMyGVSj9m4
    * December 2012

    View Slide

  37. And deploying smaller sets of features more frequently
    reduces the number of changes per deployment,
    which is an inherent benefit of continuous delivery and
    helps mitigate risk by making it easier to identify and triage
    problems if things go south during a deployment.
    https://medium.com/netflix-techblog/deploying-the-netflix-api-79b6176cc3f0
    2013: Netflix
    * published August 2013

    View Slide

  38. 2013: gilt.com
    continuous delivery is critical
    Every release is small.
    You know what's changed in
    production.
    If it doesn't work, rollback is simple
    * Surge conference, September 2013
    https://www.youtube.com/watch?v=8-6azNVq2X0

    View Slide

  39. We love to deploy new code
    incrementally hundreds of times a day.
    And there's good reason for that:
    it's safer overall.
    Incremental deploys are easier to understand and
    fix than one gigantic deploy once a year.
    https://zachholman.com/talk/move-fast-break-nothing/
    2014: Github
    * published October 2014

    View Slide

  40. https://codeascraft.com/2015/02/20/re-introducing-deployinator-now-as-a-gem/
    2015: Etsy
    We deploy code about 40 times per day.
    This allows us to push smaller changes
    we are confident about
    and experiment at a fast rate.
    Jayson Paul
    * published February 2015

    View Slide

  41. in practice, large-impact, high-
    risk deployments end up
    happening infrequently
    due to understandable fear
    unfortunately, this means that
    our changes build up
    between releases
    2015: Building Microservices
    * book published February 2015

    View Slide

  42. the bigger the delta
    between releases,
    the higher the risk
    that we’ll get something
    wrong!
    * book published February 2015
    2015: Building Microservices

    View Slide

  43. throughout the book, I
    promote the need to make
    small, incremental changes
    one of the key drivers is
    to understand the impact
    of each alteration
    * book published February 2015
    2015: Building Microservices

    View Slide

  44. 2015: Lyft
    * DockerCon, June 2015
    https://www.youtube.com/watch?v=iC2T3gJsB0g
    Matthew Leventi

    View Slide

  45. 2015: gilt.com
    * QCon London, July 2015
    simpler and easier
    deployments and rollbacks
    Yoni Goldberg
    teams can decide when
    they want to deploy
    deploy multiple times per day

    View Slide

  46. 2015: gilt.com
    * QCon London, July 2015
    database schema changes
    are required to be incremental
    Yoni Goldberg
    database schema changes
    are released separately
    from service code

    View Slide

  47. 2016: Instagram
    * SREcon, April 2016
    Michael Gorven
    https://www.youtube.com/watch?v=XNJ30Vy5xSg

    View Slide

  48. 2016: Instagram
    * SREcon, April 2016
    Michael Gorven
    https://www.youtube.com/watch?v=XNJ30Vy5xSg

    View Slide

  49. 2016: Instagram
    * SREcon, April 2016
    Michael Gorven
    https://www.youtube.com/watch?v=XNJ30Vy5xSg

    View Slide

  50. Simple releases are
    generally better than
    complicated releases.
    2016: Google
    * book published April 2016

    View Slide

  51. It is much easier to measure
    and understand the
    impact of a single change
    rather than a batch of changes
    released simultaneously.
    2016: Google
    * book published April 2016

    View Slide

  52. If the release is performed in
    smaller batches, we can move
    faster with more confidence
    because each code change can
    be understood in isolation in
    the larger system.
    2016: Google
    * book published April 2016

    View Slide

  53. 2016: Uber
    * published June 2016
    https://eng.uber.com/micro-deploy/

    View Slide

  54. 2016: Uber
    * published June 2016
    https://eng.uber.com/micro-deploy/
    Engineers write code,
    review it, test it, and
    put it into production
    the same day.
    Each week in 2016,
    thousands of engineers
    push to prod
    several thousand
    service builds

    View Slide

  55. * book published June 2016
    2016: Microservice Architecture
    reducing the
    size of the deployment unit
    a notion we call
    “batch size reduction”
    reducing the
    size or scope of the problem

    View Slide

  56. emphasis on small batch sizes
    2016: DevOps Handbook
    the principle of small batch sizes
    also applies to code reviews
    * book published October 2016

    View Slide

  57. make regular,
    small, incremental changes
    2016: AWS re:Invent
    * Las Vegas - December 1, 2016
    Tim DiLauro
    AWS Solutions Architect
    https://www.youtube.com/watch?v=8e2zCHx-K3M
    which allows you to ultimately
    move faster

    View Slide

  58. 2017: Uber
    * SRECon, March 2017
    Sebastian Yates
    https://www.youtube.com/watch?v=4A4SLk1PRvg

    View Slide

  59. 2017: Uber
    * SRECon, March 2017
    Sebastian Yates
    https://www.youtube.com/watch?v=4A4SLk1PRvg
    [service owners are]
    able to deploy
    at any schedule they like
    they can do 40 deploys a day

    View Slide

  60. the heart of doing
    evolutionary architecture
    is to
    make small changes
    * book published September 2017
    2017: Building Evolutionary Architectures

    View Slide

  61. an architecture that allows
    small, incremental changes
    is easier to evolve because
    developers have a smaller
    scope of change
    2017: Building Evolutionary Architectures
    * book published September 2017

    View Slide

  62. with improved
    engineering practices comes
    decreased risk
    2017: Building Evolutionary Architectures
    * book published September 2017

    View Slide

  63. Melanie Cebula
    2017: Airbnb
    * FutureStack New York, September 2017
    https://www.youtube.com/watch?v=N1BWMW9NEQc

    View Slide

  64. * second edition, published January 2018
    designing for deployment
    gives you the ability to
    make large changes
    in small steps
    2018: Release It!

    View Slide

  65. * second edition, published January 2018
    to be successful,
    your software will be
    deployed early and often
    smaller, easier deployments
    mean you can make
    big changes over a
    series of small steps
    2018: Release It!

    View Slide

  66. 2018: Accelerate
    * book published March 2018

    View Slide

  67. * tweet, April 2018
    https://twitter.com/bridgetkromhout/status/983739341074845697
    2018: Bridget Kromhout

    View Slide

  68. 2018: Michele Titolo
    Microservice deployments
    should be small.
    If you have small services, you are
    probably making small changes.
    * QCon New York, June 2018

    View Slide

  69. 2018: Michele Titolo
    One of the benefits
    of small changes
    is that you can
    release frequently
    * QCon New York, June 2018

    View Slide

  70. 2018: Flow Commerce
    * QCon New York, June 2018
    Mike Bryzek

    View Slide

  71. 2018: Flow Commerce
    * QCon New York, June 2018
    Mike Bryzek

    View Slide

  72. 2018: Charity Majors
    https://charity.wtf/2018/08/19/shipping-software-should-not-be-scary/
    * published August 2018
    suffice it to say that we now know that
    smaller and more frequent changes
    are much safer than larger and less frequent changes
    ship early, ship often, ship smaller change sets

    View Slide

  73. 2019: Financial Times
    * QCon London, March 2019
    https://www.infoq.com/presentations/microservices-financial-times
    Sarah Wells

    View Slide

  74. 2019: Jez Humble
    * tweet, March 2019
    https://twitter.com/jezhumble/status/1106297185119797248

    View Slide

  75. 2019: Starbucks
    * tweet, April 10 2019
    https://twitter.com/davidbrunelle/status/1116169369879363584

    View Slide

  76. 2019: Starbucks
    * tweet, April 10 2019
    https://twitter.com/davidbrunelle/status/1116169370911141888

    View Slide

  77. 2019: Charity Majors
    * tweet, April 15 2019
    https://twitter.com/mipsytipsy/status/1117858831294320640

    View Slide

  78. 2019: Charity Majors
    * published May 1, 2019
    https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/
    Fear of deploys is the ultimate technical debt
    Do not batch up multiple people’s commits
    into a deploy
    Good teams ship often

    View Slide

  79. 2019: Adrian Cockcroft
    * CraftConf, May 9 2019
    https://www.youtube.com/watch?v=rnygCAvVBj8

    View Slide

  80. 2019: Adrian Cockcroft
    * CraftConf, May 9 2019
    https://www.youtube.com/watch?v=rnygCAvVBj8

    View Slide

  81. 2019: Adrian Cockcroft
    * CraftConf, May 9 2019
    https://www.youtube.com/watch?v=rnygCAvVBj8

    View Slide

  82. 2019: Adrian Cockcroft
    * CraftConf, May 9 2019
    https://www.youtube.com/watch?v=rnygCAvVBj8

    View Slide

  83. 2019: Adrian Cockcroft
    * CraftConf, May 9 2019
    https://www.youtube.com/watch?v=rnygCAvVBj8
    Number of meetings per deploy:
    drive to zero
    You don’t have time to have 10 meetings
    a day for the 10 times you deploy today.

    View Slide

  84. 2019: Adrian Cockcroft
    * CraftConf, May 9 2019
    https://www.youtube.com/watch?v=rnygCAvVBj8
    If you have a competitor
    that is updating 10 times a day
    and you are updating
    every 2 weeks,
    then you will fall behind.

    View Slide

  85. 2019: Twilio
    * CraftConf, May 10 2019
    https://www.youtube.com/watch?v=BIkDeTEdbSo
    Justin Kitagawa

    View Slide

  86. Make It So
    small batches of work
    frequent deployments

    View Slide

  87. View Slide