Small Batch Deployments 2019-06-04

Slide 1

Slide 1 text

Small Batch Deployments Sean Sullivan June 4, 2019

Slide 2

Slide 2 text

Principles of Continuous Delivery continuousdelivery.com

Slide 3

Slide 3 text

1986: Fred Brooks “No Silver Bullet” paper Growing software organically, adding more and more function to systems as they are run, used, and tested.  

Slide 4

Slide 4 text

1999: Kent Beck * ﬁrst edition, published October 1999 We need to control the development of software by making many small adjustments, not by making a few large adjustments, kind of like driving a car.

Slide 5

Slide 5 text

1999: Kent Beck * ﬁrst edition, published October 1999 Big changes made all at once just don't work. Any problem is solved with a series of the smallest changes that make a difference.

Slide 6

Slide 6 text

2001: Agile Manifesto * published February 2001 Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. https://agilemanifesto.org/principles.html

Slide 7

Slide 7 text

2003: Poppendieck * book published May 2003 Remember the rule of small batches: If you integrate changes in small batches, it will be inﬁnitely easier to detect and ﬁx problems.

Slide 8

Slide 8 text

2004: Kent Beck * second edition, published November 2004 Software development has long delivered value in big chunks. Many teams make the problem worse by tending to respond to stress by making the chunks of value bigger, from deploying software less frequently to integrating less often.

Slide 9

Slide 9 text

2004: Kent Beck * second edition, published November 2004 The more things are deferred, the larger the chunk, the higher the risk. In contrast, the principle of ﬂow suggests that for improvement, deploy smaller increments of value ever more frequently.

Slide 10

Slide 10 text

2004: Kent Beck * second edition, published November 2004 It's always tempting to make big changes in big steps. Momentous change taken all at once is dangerous.

Slide 11

Slide 11 text

2004: Kent Beck * second edition, published November 2004 Baby steps acknowledge that the overhead of small steps is much less than when a team wastefully recoils from aborted big changes.

Slide 12

Slide 12 text

2004: Kent Beck * second edition, published November 2004 Baby steps are expressed in practices like test-ﬁrst programming, which proceeds one test at a time, and continuous integration, which integrates and tests a few hours' worth of changes at a time.

Slide 13

Slide 13 text

2004: Kent Beck * second edition, published November 2004 Big deployments have a high risk and high human and economic costs.

Slide 14

Slide 14 text

2004: Kent Beck * second edition, published November 2004 Daily Deployment Put new software into production every night. Any gap between what is on a programmer's desk and what is in production is a risk.

Slide 15

Slide 15 text

2004: Kent Beck * second edition, published November 2004 The deployment tools must be automated, including the ability to roll out incrementally and roll back in case of failure.

Slide 16

Slide 16 text

2004: Kent Beck * second edition, published November 2004 There are many barriers to deploying frequently. Some are psychological or social, like a deployment process so stressful that people don't want to go through it twice as often.

Slide 17

Slide 17 text

2004: Kent Beck * second edition, published November 2004 Whatever the barrier, working to remove it and then letting more frequent deployment come as a natural consequence will help you improve development.

Slide 18

Slide 18 text

2006: Poppendieck * book published September 2006 We have been conditioned to think that the large-batch, all-or-nothing approach to software development is good. It’s time to recondition ourselves to think that this is the worst possible approach to good software development.

Slide 19

Slide 19 text

2006: Poppendieck * book published September 2006 Work in small batches Reduce project size. Shorten release cycles. Stabilize. Repeat.

Slide 20

Slide 20 text

2007: Michael Nygard * ﬁrst edition, published March 2007 One solution — the easy but harmful one — is to slow down the release calendar. Like going to the dentist less frequently because it hurts, this response to the problem can only exacerbate the issue.

Slide 21

Slide 21 text

2007: Michael Nygard * ﬁrst edition, published March 2007 The right response is to reduce the effort needed, remove people from the process, and make the whole thing more automated and standardized.

Slide 22

Slide 22 text

http://www.startuplessonslearned.com/2008/09/new-version-of-joel-test-draft.html Do you work in small batches? Just like in lean manufacturing, it's generally more efﬁcient to drive down the batch size. I try to encourage engineers to check in anytime they have the software in a working state in their sandbox. And it's way easier to deploy small bits of code, since if something goes wrong, the problem is automatically localized and easy to revert. 2008: Eric Ries * blog post, September 2008

Slide 23

Slide 23 text

http://www.startuplessonslearned.com/2009/02/work-in-small-batches.html software should be designed, written, and deployed in small batches 2009: Eric Ries * blog post, February 2009

Slide 24

Slide 24 text

http://timothyﬁtz.com/2009/02/08/continuous-deployment/ Continuous Deployment is simple: just ship your code to customers as often as possible. 2009: Timothy Fitz * blog post, February 2009

Slide 25

Slide 25 text

2009: Flickr * Velocity Conference, June 2009 https://www.youtube.com/watch?v=LdOe18KhtT4

Slide 26

Slide 26 text

2009: Flickr * Velocity Conference, June 2009 https://www.youtube.com/watch?v=LdOe18KhtT4

Slide 27

Slide 27 text

https://omniti.com/seeds/online-application-deployment-reducing-risk.html 2010: Theo Schlossnagle * published March 2010 micro-releases: releases must never get too large instead amortize risk with small, controlled releases

Slide 28

Slide 28 text

2010: Facebook * Velocity Conference, June 2010 a culture of making very frequent small changes if you've only changed one thing at a time, it is really easy to ﬁgure out what broke the site https://www.youtube.com/watch?v=nEmJ_5UHs1g Robert Johnson

Slide 29

Slide 29 text

2010: Etsy * Velocity Conference, June 2010 https://www.kitchensoap.com/2010/06/24/ops-meta-metrics-velocity-2010-slides/ John Allspaw

Slide 30

Slide 30 text

2010: Etsy * Velocity Conference, June 2010 https://www.kitchensoap.com/2010/06/24/ops-meta-metrics-velocity-2010-slides/ John Allspaw

Slide 31

Slide 31 text

* book published August 2010 2010: Continuous Delivery Imagine that you released frequently, so the delta between what is currently in production and the new release is small. If that were true, the risk of release would be greatly diminished

Slide 32

Slide 32 text

* blog post, March 2011 2011: Wealthfront we ship to production more than 30 times a day http://eng.wealthfront.com/2011/03/12/lean-startup-stage-at-sxsw/

Slide 33

Slide 33 text

* book published September 2011 2011: Lean Startup Small Batches at IMVU IMVU makes about ﬁfty changes to its product every single day. Just as with the Toyota Production System, the key to being able to operate this quickly is to check for defects immediately, thus preventing bigger problems later.

Slide 34

Slide 34 text

* Ruby Midwest, November 2011 2011: GitHub We deploy 10 to 40 times a day. We like very short changes. If something breaks, you know exactly what breaks, really quickly. https://www.youtube.com/watch?v=qyz3jkOBbQY Zach Holman

Slide 35

Slide 35 text

incremental deployment 2012: gilt.com deploy early, deploy often * Portland Oregon, January 2012

Slide 36

Slide 36 text

Work in small batches. With a small batch size, you are reducing the amount of complexity that has to be dealt with at any one time by the people working on the batch. Break down large releases into small units of deployment 2012: Damon Edwards http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches-improve-ﬂow/ * published March 2012

Slide 37

Slide 37 text

each release is a rollback point small incremental releases actually help reduce risk 2012: Sam Newman * GeeCon, May 2012

Slide 38

Slide 38 text

https://www.joelonsoftware.com/2012/07/09/software-inventory/ * published July 2012 2012: Joel Spolsky If you’re already shipping monthly, ﬁgure out how to ship weekly. Keep pushing the bar on more and more frequent deployment of smaller and smaller changes.

Slide 39

Slide 39 text

2012: PuppetConf https://twitter.com/RealGeneKim/status/251457175128530944 * tweet, September 2012

Slide 40

Slide 40 text

2012: Michael Nygard The longer you go between deployments, the longer you build up ‘inventory’, the more opportunities there are for defects. https://www.youtube.com/watch?v=Luskg9ES9qI * GOTO Aarhus conference, October 2012

Slide 41

Slide 41 text

2012: Etsy https://www.youtube.com/watch?v=JR-ccCTmMKY * GOTO Aarhus conference, October 2012

Slide 42

Slide 42 text

How long would it take your organization to deploy a change that involved just one single line of code? Do you do this on a repeatable, reliable basis? * Berlin, October 2012 2012: Jez Humble https://www.youtube.com/watch?v=skLJuksCRTw

Slide 43

Slide 43 text

2012: Don Reinertsen https://www.youtube.com/watch?v=cAMyGVSj9m4 * December 2012

Slide 44

Slide 44 text

2012: Don Reinertsen https://www.youtube.com/watch?v=cAMyGVSj9m4 * December 2012

Slide 45

Slide 45 text

And deploying smaller sets of features more frequently reduces the number of changes per deployment, which is an inherent benefit of continuous delivery and helps mitigate risk by making it easier to identify and triage problems if things go south during a deployment. https://medium.com/netflix-techblog/deploying-the-netflix-api-79b6176cc3f0 2013: Netflix * published August 2013

Slide 46

Slide 46 text

2013: gilt.com continuous delivery is critical Every release is small. You know what's changed in production. If it doesn't work, rollback is simple * Surge conference, September 2013 https://www.youtube.com/watch?v=8-6azNVq2X0

Slide 47

Slide 47 text

https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/ 2014: Spotify * published March 2014

Slide 48

Slide 48 text

2014: Limoncelli et al * book published September 2014 Small batches means doing a lot of small releases with a few features rather than a small number of large releases with lots of features. To lower overall risk, it’s better to do many small releases containing only a few features each.

Slide 49

Slide 49 text

2014: Limoncelli et al * book published September 2014 The small batches principle is counter-intuitive because there is a human tendency to avoid risky behavior. Deploying software in production involves risk; therefore businesses traditionally minimize the frequency of deployments.

Slide 50

Slide 50 text

2014: Limoncelli et al * book published September 2014 While this makes them feel better, they actually are shooting themselves in the foot because the deployments that are done are bigger and riskier, and the team doing them is out of practice by the time the next one rolls around.

Slide 51

Slide 51 text

We love to deploy new code incrementally hundreds of times a day. And there's good reason for that: it's safer overall. Incremental deploys are easier to understand and ﬁx than one gigantic deploy once a year. https://zachholman.com/talk/move-fast-break-nothing/ 2014: Github * published October 2014

Slide 52

Slide 52 text

Break down your feature into commits that are safe to push to production https://www.infoq.com/presentations/cd-linkedin/ 2014: LinkedIn * QCon San Francisco, November 2014 Jason Toy

Slide 53

Slide 53 text

https://tech.ticketmaster.com/2014/11/14/fear-and-paranoia-in-a-devops-world/ 2014: Ticketmaster * blog post, November 2014 The premise of frequent small production deployments is that we minimize the risk in each deployment by keeping the change set small

Slide 54

Slide 54 text

https://codeascraft.com/2015/02/20/re-introducing-deployinator-now-as-a-gem/ 2015: Etsy We deploy code about 40 times per day. This allows us to push smaller changes we are conﬁdent about and experiment at a fast rate. Jayson Paul * published February 2015

Slide 55

Slide 55 text

in practice, large-impact, high- risk deployments end up happening infrequently due to understandable fear unfortunately, this means that our changes build up between releases 2015: Building Microservices * book published February 2015

Slide 56

Slide 56 text

the bigger the delta between releases, the higher the risk that we’ll get something wrong * book published February 2015 2015: Building Microservices

Slide 57

Slide 57 text

throughout the book, I promote the need to make small, incremental changes one of the key drivers is to understand the impact of each alteration * book published February 2015 2015: Building Microservices

Slide 58

Slide 58 text

2015: Lyft * DockerCon, June 2015 https://www.youtube.com/watch?v=iC2T3gJsB0g Matthew Leventi

Slide 59

Slide 59 text

2015: gilt.com * QCon London, July 2015 simpler and easier deployments and rollbacks Yoni Goldberg teams can decide when they want to deploy deploy multiple times per day

Slide 60

Slide 60 text

2015: gilt.com * QCon London, July 2015 database schema changes are required to be incremental Yoni Goldberg database schema changes are released separately from service code

Slide 61

Slide 61 text

2016: Zalando * published February 2016 Strive for very short release cycles, optimally deploying daily Small releases tend to have fewer bugs. https://github.com/zalando/engineering-principles/blob/master/README.md

Slide 62

Slide 62 text

2016: Timothy Fitz * Agile India, March 2016 https://www.youtube.com/watch?v=qu89JWEshlU

Slide 63

Slide 63 text

2016: Instagram * SREcon, April 2016 Michael Gorven https://www.youtube.com/watch?v=XNJ30Vy5xSg

Slide 64

Slide 64 text

2016: Instagram * SREcon, April 2016 Michael Gorven https://www.youtube.com/watch?v=XNJ30Vy5xSg

Slide 65

Slide 65 text

2016: Instagram * SREcon, April 2016 Michael Gorven https://www.youtube.com/watch?v=XNJ30Vy5xSg

Slide 66

Slide 66 text

Simple releases are generally better than complicated releases. 2016: Google * book published April 2016

Slide 67

Slide 67 text

It is much easier to measure and understand the impact of a single change rather than a batch of changes released simultaneously. 2016: Google * book published April 2016

Slide 68

Slide 68 text

If the release is performed in smaller batches, we can move faster with more conﬁdence because each code change can be understood in isolation in the larger system. 2016: Google * book published April 2016

Slide 69

Slide 69 text

2016: Netﬂix * DevOpsDays Rockies, April 2016 https://www.youtube.com/watch?v=HmM4V33ReCw Dave Hahn

Slide 70

Slide 70 text

2016: Shopify * RailsConf, May 2016 https://www.youtube.com/watch?v=HNH7El_BEsw Kat Drobnjakovic We deploy Shopify an average 30 times each day.

Slide 71

Slide 71 text

2016: Danilo Sato * DevOps Singapore, May 2016 https://www.youtube.com/watch?v=eBRspUcDCTc

Slide 72

Slide 72 text

2016: Danilo Sato * DevOps Singapore, May 2016 https://www.youtube.com/watch?v=eBRspUcDCTc

Slide 73

Slide 73 text

2016: Uber * published June 2016 https://eng.uber.com/micro-deploy/

Slide 74

Slide 74 text

2016: Uber * published June 2016 https://eng.uber.com/micro-deploy/ Engineers write code, review it, test it, and put it into production the same day. Each week in 2016, thousands of engineers push to prod several thousand service builds

Slide 75

Slide 75 text

* book published June 2016 2016: Microservice Architecture reducing the size of the deployment unit a notion we call “batch size reduction” reducing the size or scope of the problem

Slide 76

Slide 76 text

emphasis on small batch sizes 2016: DevOps Handbook the principle of small batch sizes also applies to code reviews * book published October 2016

Slide 77

Slide 77 text

make regular, small, incremental changes 2016: AWS re:Invent * Las Vegas - December 1, 2016 Tim DiLauro AWS Solutions Architect https://www.youtube.com/watch?v=8e2zCHx-K3M which allows you to ultimately move faster

Slide 78

Slide 78 text

2017: Uber * SRECon, March 2017 Sebastian Yates https://www.youtube.com/watch?v=4A4SLk1PRvg

Slide 79

Slide 79 text

2017: Uber * SRECon, March 2017 Sebastian Yates https://www.youtube.com/watch?v=4A4SLk1PRvg [service owners are] able to deploy at any schedule they like 40 deploys a day

Slide 80

Slide 80 text

the heart of doing evolutionary architecture is to make small changes * book published September 2017 2017: Building Evolutionary Architectures

Slide 81

Slide 81 text

an architecture that allows small, incremental changes is easier to evolve because developers have a smaller scope of change 2017: Building Evolutionary Architectures * book published September 2017

Slide 82

Slide 82 text

with improved engineering practices comes decreased risk 2017: Building Evolutionary Architectures * book published September 2017

Slide 83

Slide 83 text

Melanie Cebula 2017: Airbnb * FutureStack New York, September 2017 https://www.youtube.com/watch?v=N1BWMW9NEQc

Slide 84

Slide 84 text

2017: Randy Shoup * DevOpsDays Cuba, October 2017 https://www.slideshare.net/RandyShoup/evolving-architecture-and-organization-lessons-from-google-and-ebay

Slide 85

Slide 85 text

2017: Randy Shoup * DevOpsDays Cuba, October 2017 https://www.slideshare.net/RandyShoup/evolving-architecture-and-organization-lessons-from-google-and-ebay

Slide 86

Slide 86 text

For years, IT teams in medium to large enterprises released updates to applications in large batches with the belief that this is more efﬁcient, less impactful, and a better return on the investment. Because we believed this, we built processes, compliance checks, architecture guidelines, and measures to support the large sized, complex releases. But this belief is incorrect. 2017: Salesforce * blog post, December 2017 https://www.salesforce.com/blog/2017/12/smaller-batches-improves-application-roi.html

Slide 87

Slide 87 text

* second edition, published January 2018 designing for deployment gives you the ability to make large changes in small steps 2018: Release It!

Slide 88

Slide 88 text

* second edition, published January 2018 to be successful, your software will be deployed early and often smaller, easier deployments mean you can make big changes over a series of small steps 2018: Release It!

Slide 89

Slide 89 text

* DevOpsDays NYC, January 18 2018 2018: Thomas Limoncelli https://www.youtube.com/watch?v=3uVdRpjn8iA

Slide 90

Slide 90 text

2018: Accelerate * book published March 2018

Slide 91

Slide 91 text

* tweet, April 2018 https://twitter.com/bridgetkromhout/status/983739341074845697 2018: Bridget Kromhout

Slide 92

Slide 92 text

2018: Michele Titolo Microservice deployments should be small. If you have small services, you are probably making small changes. * QCon New York, June 2018

Slide 93

Slide 93 text

2018: Michele Titolo One of the beneﬁts of small changes is that you can release frequently * QCon New York, June 2018

Slide 94

Slide 94 text

2018: Flow Commerce * QCon New York, June 2018 Mike Bryzek

Slide 95

Slide 95 text

2018: Flow Commerce * QCon New York, June 2018 Mike Bryzek

Slide 96

Slide 96 text

2018: Charity Majors https://charity.wtf/2018/08/19/shipping-software-should-not-be-scary/ * published August 2018 sufﬁce it to say that we now know that smaller and more frequent changes are much safer than larger and less frequent changes ship early, ship often, ship smaller change sets

Slide 97

Slide 97 text

2018: Hulu * QCon San Francisco, November 2018 https://www.youtube.com/watch?v=3vyLcbI16P8 Andrew McVeigh Granular deployment At Hulu, it was not uncommon for us to deploy 100 times in a day.

Slide 98

Slide 98 text

2018: Slack * QCon San Francisco, November 2018 https://www.youtube.com/watch?v=mwsfNio2Dho Holly Allen

Slide 99

Slide 99 text

2019: Financial Times * QCon London, March 2019 https://www.infoq.com/presentations/microservices-ﬁnancial-times Sarah Wells

Slide 100

Slide 100 text

2019: Jez Humble * tweet, March 2019 https://twitter.com/jezhumble/status/1106297185119797248

Slide 101

Slide 101 text

2019: Starbucks * tweet, April 10 2019 https://twitter.com/davidbrunelle/status/1116169369879363584

Slide 102

Slide 102 text

2019: Starbucks * tweet, April 10 2019 https://twitter.com/davidbrunelle/status/1116169370911141888

Slide 103

Slide 103 text

2019: Charity Majors * tweet, April 15 2019 https://twitter.com/mipsytipsy/status/1117858831294320640

Slide 104

Slide 104 text

2019: Charity Majors * published May 1, 2019 https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/ Fear of deploys is the ultimate technical debt Do not batch up multiple people’s commits into a deploy Good teams ship often

Slide 105

Slide 105 text

2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8

Slide 106

Slide 106 text

2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8

Slide 107

Slide 107 text

2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8

Slide 108

Slide 108 text

2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8

Slide 109

Slide 109 text

2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8 Number of meetings per deploy: drive to zero You don’t have time to have 10 meetings a day for the 10 times you deploy today.

Slide 110

Slide 110 text

2019: Adrian Cockcroft * CraftConf, May 9 2019 https://www.youtube.com/watch?v=rnygCAvVBj8 If you have a competitor that is updating 10 times a day and you are updating every 2 weeks, then you will fall behind.

Slide 111

Slide 111 text

2019: Jez Humble * CraftConf, May 9 2019 https://www.youtube.com/watch?v=CN6uhzNM4eA Working in small batches. Taking big ideas and splitting them into small chunks so they can be delivered in less than a week. And released frequently.

Slide 112

Slide 112 text

2019: Twilio * CraftConf, May 10 2019 https://www.youtube.com/watch?v=BIkDeTEdbSo Justin Kitagawa

Slide 113

Slide 113 text

2019: Dave Farley * JAX DevOps London, May 15 2019 https://www.youtube.com/watch?v=NiTxzYA_qRQ

Slide 114

Slide 114 text

2019: Dave Farley * JAX DevOps London, May 15 2019 https://www.youtube.com/watch?v=NiTxzYA_qRQ We could decide to write our software in big pieces and deploy infrequently. But that is a low quality solution. Each piece is complicated and risky.

Slide 115

Slide 115 text

2019: Dave Farley * JAX DevOps London, May 15 2019 Or we could decide to break our problem down into small little pieces. Each piece is smaller, and simpler, and lower risk. https://www.youtube.com/watch?v=NiTxzYA_qRQ