Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Delivery: More Than a Pipeline

2350801025b8e8592dbaa8dd98a24cbb?s=47 Eberhard Wolff
November 14, 2018
150

Continuous Delivery: More Than a Pipeline

Continuous Delivery is a lot more than just a pipeline. This talks discusses reasons why software should be deployed more frequently and the barriers that keep teams from doing so.

2350801025b8e8592dbaa8dd98a24cbb?s=128

Eberhard Wolff

November 14, 2018
Tweet

Transcript

  1. Eberhard Wolff Fellow INNOQ @ewolff http://ewolff.com

  2. http://continuous-delivery-buch.de/ http://continuous-delivery-book.com/

  3. http://microservices-buch.de/ http://microservices-book.com/

  4. http://microservices-buch.de/ ueberblick.html http://microservices-book.com/ primer.html FREE!!!!

  5. http://microservices-praxisbuch.de http://practical-microservices.com/

  6. http://microservices-praxisbuch.de/ rezepte.html http://practical-microservices.com/ recipes.html FREE!!!!

  7. Why Care About Continuous Delivery?

  8. Continuous Delivery is just about time-to- market!

  9. Continuous Delivery is just about time-to- market!

  10. https://puppet.com/resources/whitepaper/state-of-devops-report

  11. Deployment Frequency: Results • Elite Performers vs. Low Performers •

    Multiple times per day vs. once per week / month • 2.555x better lead time for change • 2.604x better time to restore service • 7x better change failure rate • 50% vs 30% time spent on new work
  12. Higher Deployment Frequency Improves Lead Time Reliability New Work %

    dramatically
  13. MAERSK COST OF DELAY POWER LAW CURVE 200K 10 30

    60 70 80 20 50 40 Number of Requirements 1.2MM 2.2MM 400K 1.4MM 2.4MM 600K 1.6MM 2.6MM 1MM 2MM 1.8MM 2.8MM Cost of Delay/Week Source: State of DevOps Report, blackswarmfarming.com
  14. MAERSK COST OF DELAY POWER LAW CURVE 200K 10 30

    60 70 80 20 50 40 Number of Requirements 1.2MM 2.2MM 400K 1.4MM 2.4MM 600K 1.6MM 2.6MM 1MM 2MM 1.8MM 2.8MM Cost of Delay/Week Source: State of DevOps Report, blackswarmfarming.com Cost of delay might be much higher than cost reductions in operations!
  15. No Silver Bullet Essence and Accident in Software Engineering Frederick

    P. Brooks, Jr. There is no single development... which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.
  16. No Silver Bullet

  17. No Silver Bullet except more frequent deployments?

  18. Continuous Delivery: Goals • Deploy often • More than once

    a week / month …because that is still low performer • Multiple times a day! • How???
  19. There many barriers to deploy multiple times a day!

  20. Deployment Testing Architecture Release Branching Multiple Deploy- ments per Day

    Elite
  21. Choose your speed - no need to break every barrier!

  22. There is much room for improvement!

  23. Deployment Testing Architecture Release Branching ? ? The next step

    is important!
  24. Deployment

  25. Manual Deployment • Manually deploy software to servers using scripts

    • Somehow check correctness • Takes hours or days
  26. Problem: Slow deployment Max. Frequency: Week

  27. Automated Deployment • Fully automated scripts …might be Dockerfiles •

    Faster • Automated deployment to deploy every few months? • Deployment: often focus but not enough
  28. Customer Scenario • Quarterly releases • 10 weeks of testing

    • Release two days over the weekend • Goal: Several deployments a day • Automated deployment not enough Development Testing Release Testing + Release + Development Time
  29. Testing

  30. Separate Dev and QA • Hand over code • Hand

    over bug reports
  31. None
  32. None
  33. None
  34. None
  35. None
  36. None
  37. None
  38. Separate Dev and QA • Problem: Hand over of code

    • Hand over of bug reports • Takes time • Lean: Hand-over is waste • Does not enforce collaboration
  39. Development Testing Release Time

  40. Problem: Hand-over Max. Frequency: Month

  41. Cross-functional Teams

  42. Cross-functional Teams

  43. Cross-functional Teams • Hand-over in team quicker • Might work

    very closely …pairing
  44. Development Testing Release Time Development Testing Release Development Testing Release

  45. Problem: Manual work Max. Frequency: Weeks

  46. Only Automated Tests For Deployment • Fully automated test suite

    • …including capacity tests and acceptance tests • No manual sign-off • Really: no manual sign-off!
  47. Commit Stage Automated Acceptance Testing Automated Capacity Testing Manual Explorative

    Testing Release Continuous Delivery Pipeline
  48. Development Testing Release Time Development Testing Release Development Testing Release

  49. Problem: Time for tests Max. Frequency: Days / hours

  50. Going faster • Reduce risk not just with tests •

    …but with releasing • Smaller deployment unit • Full automation might not be feasible for large deployment units
  51. Architecture

  52. Monolith • Deployment monolith • Deploy the whole system at

    once • Internal structure: Might be great or not so great… • A valid architecture
  53. Monolith: Deployment • Deploy all at once • It is

    hard to deploy a large system • Tests even harder • Can take loooooong – days, weeks, months • So large infrequent deployments • Risky
  54. Problem: Deployment size Max. Frequency: Months / weeks

  55. Microservices: Deployment • Deploy individually • Requires automated deployment •

    Commodity with Docker, Kubernetes… • Requires decoupled components & tests • Enables faster deployments & tests • Enables frequent deployments
  56. Development Testing Release Time Development Testing Release Development Testing Release

  57. Releases

  58. Release Train • Release multiple dependent systems together • Extreme:

    Release all systems in the IT together • The reason why developers at one point in the sixties or so stopped deploying continuously?
  59. Release Train • In total: a day or a few

    days Release System 1 Release dependent System 2 Release dependent System 3 …
  60. Problem: Coordinate many changes, risk Max. Frequency: Months

  61. Release per System • Much easier to coordinate • Backward

    compatible interfaces • Speed depends on deployment size …see architecture
  62. Problem: Risk Max. Frequency: Weeks

  63. Blue / Green Deployment Server Server Server Server Database Server

    Server Server Server Database
  64. Blue / Green Deployment • Needs lots of resources •

    Might be unrealistic: another production environment? • Migrate database • ...and keep consistent for fallback • Easier with small microservices
  65. Canary Release Server Server Server Server Database Server Server

  66. Canary Release Server Server Server Server Database Server Server Server

    Server
  67. Canary Releases • Less resources • Needs backwards compatibility of

    other components / database • Easier with small microservices
  68. Problem: Risk (depends on size of deployment unit) Max. Frequency:

    Days / hours
  69. Dark Launching • Install new version • Let it handle

    production traffic • ...but don’t let it change data. • Requires monitoring • Enables less focus on tests
  70. Dark Launching Server Server Server Server Database Server

  71. Branching

  72. Feature Branches • Create a new branch to implement a

    feature • Integrate into master when done • Easy to parallelize development • Easy to add features to master at a specific point in time
  73. Feature Branch Master Feature Registration Feature Prediction Commit Branch Commit

    Branch Commit Commit Commit Merge Merge Release with Features
  74. Feature Branches • Feature branches’ goal: Delay integration • Might

    still compile all changes from all branches together to find issues early • True integration into master only after go / no go for features
  75. Feature Branch Master Feature Registration Feature Prediction Commit Branch Commit

    Branch Commit Commit Commit Merge Merge Release with Features Integration Integration
  76. Feature Branches vs. Continuous Integration • Continuous Integration: Continuously integrate

    all changes • Feature branches’ goal: Delay integration • Both approaches might work • See https://www.heise.de/ developer/artikel/Continuous-Integration- widerspricht-Feature-Branches-2736487.html • …or https://www.innoq.com/en/blog/continuous- integration-contradicts-feature-branches/
  77. Problem: Integration takes time Max. Frequency: Days?

  78. Trunk-based Development • Each commit goes to master • No

    more integration
  79. Trunk-based Development • Could deploy any time • But: User

    must not use unfinished features Commit Registration Commit Prediction Commit Registration Commit Prediction Commit Prediction
  80. Feature Toggles • Deployment should not activate new features •

    Disable feature in production • Problem: Must be cleaned up eventually • Problem: Harder to test
  81. None
  82. Humble et al: Lean Enterprise, source: https://paulhammant.com/2013/03/13/facebook-tbd-take-2/

  83. Conclusion • More frequent deployments have many advantages • Choose

    a desired speed • ...and break the next barrier! • It’s not just microservices or deployment automation!
  84. But there are sooo many problems to solve! L

  85. Actually there are sooo many solutions! J

  86. Identify barriers and eliminate them!