Continuous Delivery: More Than a Pipeline

Slide 1

Slide 1 text

Eberhard Wolff Fellow INNOQ @ewolff http://ewolff.com

Slide 2

Slide 2 text

http://continuous-delivery-buch.de/ http://continuous-delivery-book.com/

Slide 3

Slide 3 text

http://microservices-buch.de/ http://microservices-book.com/

Slide 4

Slide 4 text

http://microservices-buch.de/ ueberblick.html http://microservices-book.com/ primer.html FREE!!!!

Slide 5

Slide 5 text

http://microservices-praxisbuch.de http://practical-microservices.com/

Slide 6

Slide 6 text

http://microservices-praxisbuch.de/ rezepte.html http://practical-microservices.com/ recipes.html FREE!!!!

Slide 7

Slide 7 text

Why Care About Continuous Delivery?

Slide 8

Slide 8 text

Continuous Delivery is just about time-to- market!

Slide 9

Slide 9 text

Continuous Delivery is just about time-to- market!

Slide 10

Slide 10 text

https://puppet.com/resources/whitepaper/state-of-devops-report

Slide 11

Slide 11 text

Deployment Frequency: Results • Elite Performers vs. Low Performers • Multiple times per day vs. once per week / month • 2.555x better lead time for change • 2.604x better time to restore service • 7x better change failure rate • 50% vs 30% time spent on new work

Slide 12

Slide 12 text

Higher Deployment Frequency Improves Lead Time Reliability New Work % dramatically

Slide 13

Slide 13 text

MAERSK COST OF DELAY POWER LAW CURVE 200K 10 30 60 70 80 20 50 40 Number of Requirements 1.2MM 2.2MM 400K 1.4MM 2.4MM 600K 1.6MM 2.6MM 1MM 2MM 1.8MM 2.8MM Cost of Delay/Week Source: State of DevOps Report, blackswarmfarming.com

Slide 14

Slide 14 text

Slide 15

Slide 15 text

No Silver Bullet Essence and Accident in Software Engineering Frederick P. Brooks, Jr. There is no single development... which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.

Slide 16

Slide 16 text

No Silver Bullet

Slide 17

Slide 17 text

No Silver Bullet except more frequent deployments?

Slide 18

Slide 18 text

Continuous Delivery: Goals • Deploy often • More than once a week / month …because that is still low performer • Multiple times a day! • How???

Slide 19

Slide 19 text

There many barriers to deploy multiple times a day!

Slide 20

Slide 20 text

Deployment Testing Architecture Release Branching Multiple Deploy- ments per Day Elite

Slide 21

Slide 21 text

Choose your speed - no need to break every barrier!

Slide 22

Slide 22 text

There is much room for improvement!

Slide 23

Slide 23 text

Deployment Testing Architecture Release Branching ? ? The next step is important!

Slide 24

Slide 24 text

Deployment

Slide 25

Slide 25 text

Manual Deployment • Manually deploy software to servers using scripts • Somehow check correctness • Takes hours or days

Slide 26

Slide 26 text

Problem: Slow deployment Max. Frequency: Week

Slide 27

Slide 27 text

Automated Deployment • Fully automated scripts …might be Dockerfiles • Faster • Automated deployment to deploy every few months? • Deployment: often focus but not enough

Slide 28

Slide 28 text

Customer Scenario • Quarterly releases • 10 weeks of testing • Release two days over the weekend • Goal: Several deployments a day • Automated deployment not enough Development Testing Release Testing + Release + Development Time

Slide 29

Slide 29 text

Testing

Slide 30

Slide 30 text

Separate Dev and QA • Hand over code • Hand over bug reports

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

Separate Dev and QA • Problem: Hand over of code • Hand over of bug reports • Takes time • Lean: Hand-over is waste • Does not enforce collaboration

Slide 39

Slide 39 text

Development Testing Release Time

Slide 40

Slide 40 text

Problem: Hand-over Max. Frequency: Month

Slide 41

Slide 41 text

Cross-functional Teams

Slide 42

Slide 42 text

Cross-functional Teams

Slide 43

Slide 43 text

Cross-functional Teams • Hand-over in team quicker • Might work very closely …pairing

Slide 44

Slide 44 text

Development Testing Release Time Development Testing Release Development Testing Release

Slide 45

Slide 45 text

Problem: Manual work Max. Frequency: Weeks

Slide 46

Slide 46 text

Only Automated Tests For Deployment • Fully automated test suite • …including capacity tests and acceptance tests • No manual sign-off • Really: no manual sign-off!

Slide 47

Slide 47 text

Commit Stage Automated Acceptance Testing Automated Capacity Testing Manual Explorative Testing Release Continuous Delivery Pipeline

Slide 48

Slide 48 text

Development Testing Release Time Development Testing Release Development Testing Release

Slide 49

Slide 49 text

Problem: Time for tests Max. Frequency: Days / hours

Slide 50

Slide 50 text

Going faster • Reduce risk not just with tests • …but with releasing • Smaller deployment unit • Full automation might not be feasible for large deployment units

Slide 51

Slide 51 text

Architecture

Slide 52

Slide 52 text

Monolith • Deployment monolith • Deploy the whole system at once • Internal structure: Might be great or not so great… • A valid architecture

Slide 53

Slide 53 text

Monolith: Deployment • Deploy all at once • It is hard to deploy a large system • Tests even harder • Can take loooooong – days, weeks, months • So large infrequent deployments • Risky

Slide 54

Slide 54 text

Problem: Deployment size Max. Frequency: Months / weeks

Slide 55

Slide 55 text

Microservices: Deployment • Deploy individually • Requires automated deployment • Commodity with Docker, Kubernetes… • Requires decoupled components & tests • Enables faster deployments & tests • Enables frequent deployments

Slide 56

Slide 56 text

Development Testing Release Time Development Testing Release Development Testing Release

Slide 57

Slide 57 text

Releases

Slide 58

Slide 58 text

Release Train • Release multiple dependent systems together • Extreme: Release all systems in the IT together • The reason why developers at one point in the sixties or so stopped deploying continuously?

Slide 59

Slide 59 text

Release Train • In total: a day or a few days Release System 1 Release dependent System 2 Release dependent System 3 …

Slide 60

Slide 60 text

Problem: Coordinate many changes, risk Max. Frequency: Months

Slide 61

Slide 61 text

Release per System • Much easier to coordinate • Backward compatible interfaces • Speed depends on deployment size …see architecture

Slide 62

Slide 62 text

Problem: Risk Max. Frequency: Weeks

Slide 63

Slide 63 text

Blue / Green Deployment Server Server Server Server Database Server Server Server Server Database

Slide 64

Slide 64 text

Blue / Green Deployment • Needs lots of resources • Might be unrealistic: another production environment? • Migrate database • ...and keep consistent for fallback • Easier with small microservices

Slide 65

Slide 65 text

Canary Release Server Server Server Server Database Server Server

Slide 66

Slide 66 text

Canary Release Server Server Server Server Database Server Server Server Server

Slide 67

Slide 67 text

Canary Releases • Less resources • Needs backwards compatibility of other components / database • Easier with small microservices

Slide 68

Slide 68 text

Problem: Risk (depends on size of deployment unit) Max. Frequency: Days / hours

Slide 69

Slide 69 text

Dark Launching • Install new version • Let it handle production traffic • ...but don’t let it change data. • Requires monitoring • Enables less focus on tests

Slide 70

Slide 70 text

Dark Launching Server Server Server Server Database Server

Slide 71

Slide 71 text

Branching

Slide 72

Slide 72 text

Feature Branches • Create a new branch to implement a feature • Integrate into master when done • Easy to parallelize development • Easy to add features to master at a specific point in time

Slide 73

Slide 73 text

Feature Branch Master Feature Registration Feature Prediction Commit Branch Commit Branch Commit Commit Commit Merge Merge Release with Features

Slide 74

Slide 74 text

Feature Branches • Feature branches’ goal: Delay integration • Might still compile all changes from all branches together to find issues early • True integration into master only after go / no go for features

Slide 75

Slide 75 text

Feature Branch Master Feature Registration Feature Prediction Commit Branch Commit Branch Commit Commit Commit Merge Merge Release with Features Integration Integration

Slide 76

Slide 76 text

Feature Branches vs. Continuous Integration • Continuous Integration: Continuously integrate all changes • Feature branches’ goal: Delay integration • Both approaches might work • See https://www.heise.de/ developer/artikel/Continuous-Integration- widerspricht-Feature-Branches-2736487.html • …or https://www.innoq.com/en/blog/continuous- integration-contradicts-feature-branches/

Slide 77

Slide 77 text

Problem: Integration takes time Max. Frequency: Days?

Slide 78

Slide 78 text

Trunk-based Development • Each commit goes to master • No more integration

Slide 79

Slide 79 text

Trunk-based Development • Could deploy any time • But: User must not use unfinished features Commit Registration Commit Prediction Commit Registration Commit Prediction Commit Prediction

Slide 80

Slide 80 text

Feature Toggles • Deployment should not activate new features • Disable feature in production • Problem: Must be cleaned up eventually • Problem: Harder to test

Slide 81

Slide 81 text

No content

Slide 82

Slide 82 text

Humble et al: Lean Enterprise, source: https://paulhammant.com/2013/03/13/facebook-tbd-take-2/

Slide 83

Slide 83 text

Conclusion • More frequent deployments have many advantages • Choose a desired speed • ...and break the next barrier! • It’s not just microservices or deployment automation!