DevOpsDaysPortugal 2019 - Diogo Oliveira - Moving Mountains: The Real-World Enterprise Journey to Continuous Delivery

OutSystems set out on a journey to Continuous Delivery, dramatically changing the development processes, delivery, and quality, while at the same time, continuing to develop and deliver new product features.

A few years ago, OutSystems had an annual release cadence. The once-a-year release was blocking us from having customer feedback in the loop as early as we should and our lead time was huge.

Customers want value as fast as possible, and their feedback is of the utmost importance. We needed to be faster and release more often (while maintaining great quality). In the last few years, we have transformed how we deliver software, and we’d love to share our journey and experience.

Transcript

1. Moving Mountains: The Real-World Enterprise Journey to Continuous Delivery DevOpsDays Lisbon June 3, 2019
2. Setting the stage About me About OutSystems Continuous Delivery journey The beginning Efficiency phase Acceleration phase Autonomy phase Takeaways
3. Setting the stage About me About OutSystems Continuous Delivery journey The beginning Efficiency phase Acceleration phase Autonomy phase Takeaways
4. Diogo Oliveira Senior Software Engineer / Continuous Delivery Expert @ OutSystems R&D 4
5. Setting the stage About me About OutSystems Continuous Delivery journey The beginning Efficiency phase Acceleration phase Autonomy phase Takeaways
6. International company with its R&D based in Portugal (Lisbon + Braga) 6
7. Low-code rapid application development and delivery platform 7
8. Low-code rapid application development and delivery platform 8
9. Low-code rapid application development and delivery platform 9
11. Quality assurance is a big challenge
12. Multiple stacks Multiple browsers Multiple devices Quality assurance is a big challenge
13. Multiple stacks Multiple browsers Multiple devices Need to execute ~30.000 distinct fully automated tests * possible stack combinations (+300.000 test executions) Quality assurance is a big challenge
14. Setting the stage About me About OutSystems Continuous Delivery journey The beginning Efficiency phase Acceleration phase Autonomy phase Takeaways
15. Major release every 2 years
16. 16 Major release every 2 years Maintenance team
17. 17 Major release every 2 years Maintenance team Reintegrate hell
18. Major release every 2 years Maintenance team Reintegrate hell Monolithic product
19. 19 Major release every 2 years Maintenance team Reintegrate hell Monolithic product Very long feedback loops
20. 20 Major release every 2 years Maintenance team Reintegrate hell Monolithic product Very long feedback loops Painful product stabilization
21. 21 Major release every 2 years Maintenance team Reintegrate hell Monolithic product Very long feedback loops Painful product stabilization Lack of product / tests ownership
22. 22 Major release every 2 years Maintenance team Reintegrate hell Monolithic product Very long feedback loops Painful product stabilization Lack of product / tests ownership Lack of sustainable CI/CD practice
23. 23 Major release every 2 years Maintenance team Reintegrate hell Monolithic product Very long feedback loops Painful product stabilization Lack of product / tests ownership Lack of sustainable CI/CD practice Manual build and test infrastructure provisioning
24. This context was highly inefficient and would not enable us to scale. We need to be much more efficient! 24
25. Setting the stage About me About OutSystems Continuous Delivery journey The beginning Efficiency phase Acceleration phase Autonomy phase Takeaways
26. 26 New major with support for mobile apps
27. 27 Shared across many teams New major with support for mobile apps
28. 28 Shared across many teams Need for good collaboration New major with support for mobile apps
29. 29 Shared across many teams Need for continuous work integration Need for good collaboration New major with support for mobile apps
30. 30 Shared across many teams High risk project Need for continuous work integration Need for good collaboration New major with support for mobile apps
31. 31 Shared across many teams High risk project Need for frequent releases Need for continuous work integration Need for good collaboration New major with support for mobile apps
32. 32 Shared across many teams High risk project Need for continuous feedback on changes Need for frequent releases Need for continuous work integration Need for good collaboration New major with support for mobile apps
33. 33 We needed to do something...
34. 34 Continuous Integration/Delivery Development Model Organization Structure
35. 35 Continuous Integration/Delivery Development Model Organization Structure
38. 38 Lesson Learned People need to be aware of their challenges and the value of doing things differently
39. 39 CINTIA (Continuous INTegration and Intelligent Alert system)
40. 40 Lesson Learned Empowering development teams should be encouraged
41. 41 Lesson Learned Empowering development teams should be encouraged Lesson Learned Invest on infrastructure automation from day 0
42. 42 Lesson Learned Empowering development teams should be encouraged Lesson Learned Invest on infrastructure automation from day 0 Lesson Learned Elasticity and reliability are crucial for reliable and fast feedback loop
43. 43 GoCD to orchestrate the pipeline
44. 44 GoCD to orchestrate the pipeline Automatic failure assignment
45. 45 GoCD to orchestrate the pipeline Automatic failure assignment Runs all tests of main stack
46. 46 GoCD to orchestrate the pipeline Automatic failure assignment Runs all tests of main stack Fast feedback loop
47. 47 GoCD to orchestrate the pipeline Automatic failure assignment Runs all tests of main stack Fast feedback loop Heavy parallelization with discardable agents
48. 48 GoCD to orchestrate the pipeline Automatic failure assignment Runs all tests of main stack Fast feedback loop Heavy parallelization with discardable agents Make it visible (on TVs)
49. 49 Lesson Learned Constant visibility over the build pipeline is essential
50. 50 Lesson Learned Constant visibility over the build pipeline is essential Lesson Learned You need to treat your build pipeline as a product
51. 51 Continuous Integration/Delivery Development Model Organization Structure
52. 52 Strategic project called “Rethinking R&D Development Model”
53. 53 Development & QA best practices Strategic project called “Rethinking R&D Development Model”
54. 54 Development & QA best practices New release model Strategic project called “Rethinking R&D Development Model”
55. 55 Development & QA best practices New release model Decided to work in a single branch (per major)! b10_0_0 (development is done in b10_0_0 for major version 10.0) b9_1_0 (development is done in b9_1_0 for major version 9.1) Trunk (development is done in Trunk for the most recent major version, e.g.: 11.0) Strategic project called “Rethinking R&D Development Model”
56. 56 Lesson Learned Working in single branch might be hard at the beginning, but it is the way to go
58. 58 Continuous Integration/Delivery Development Model Organization Structure
59. Teams organization more aligned with the desired product architecture (Conway’s Law) 59
60. Teams organization more aligned with the desired product architecture (Conway’s Law) 60 Maintenance is now part of each team’s responsibilities
61. Teams organization more aligned with the desired product architecture (Conway’s Law) 61 Maintenance is now part of each team’s responsibilities Teams have more autonomy
62. Teams organization more aligned with the desired product architecture (Conway’s Law) 62 Maintenance is now part of each team’s responsibilities Teams have more autonomy Teams have clear ownership on product and tests
63. 63 Lesson Learned Ownership and autonomy are crucial to drive product and tests evolution
64. 64 But hey! Now that I have autonomy, why do I need to wait for other teams? Minion, a developer at OutSystems
66. 66 We needed to accelerate
67. Setting the stage About me About OutSystems Continuous Delivery journey The beginning Efficiency phase Acceleration phase Autonomy phase Takeaways
68. 68 What was blocking our acceleration?
71. 71 • Low change risk • Smooth update (easy to revert) • Potential for incremental value
72. 72 • Low change risk • Smooth update (easy to revert) • Potential for incremental value • High change risk • Effortful update (not easy to revert) • More disruptive / large scale work
73. 73 • Low change risk • Smooth update (easy to revert) • Potential for incremental value • High change risk • Effortful update (not easy to revert) • More disruptive / large scale work We’re releasing all together Aren’t we wasting opportunities?
74. 74 What if we split the validations and release independently?
75. 75 Compress all feedback loops
76. 76 Compress all feedback loops Build Pipeline < 1h all validations (~30.000 tests)
77. 77 Compress all feedback loops Build Pipeline < 1h all validations (~30.000 tests) Dogfooding Get real feedback usage fast (in one of our largest factories)
78. 78 Compress all feedback loops Build Pipeline < 1h all validations (~30.000 tests) Dogfooding Get real feedback usage fast (in one of our largest factories) Telemetry Instrument the code to understand product usage and issues
79. 79 Lesson Learned Having fast feedback is not a choice, it needs to be fast
80. 80 Lesson Learned Having fast feedback is not a choice, it needs to be fast Lesson Learned We need to know how our users use the product, as soon as possible
81. 81 Value at the customers hands now
82. 82 Value at the customers hands now Auto update Automatic update for our IDE
83. 83 Value at the customers hands now Auto update Automatic update for our IDE Automatic Cloud Rollout For LifeTime
84. 84 Value at the customers hands now Auto update Automatic update for our IDE Beta version Get feedback as soon as possible for our IDE Automatic Cloud Rollout For LifeTime
85. 85 Value at the customers hands now Auto update Automatic update for our IDE Beta version Get feedback as soon as possible for our IDE Feature Toggling Code can be delivered at any time Automatic Cloud Rollout For LifeTime
86. 86 Lesson Learned If we want our customers to use the latest product version we need to control the product distribution until it is in the hands of users and being used
88. Things are going pretty fast We can’t depend on some specific teams to have the product out there 88
89. Things are going pretty fast We can’t depend on some specific teams to have the product out there Everyone needs autonomy! 89
90. Setting the stage About me About OutSystems Continuous Delivery journey The beginning Efficiency phase Acceleration phase Autonomy phase Takeaways
91. Knowledge about release and build pipeline ecosystem in a specific group of people 91
92. Knowledge about release and build pipeline ecosystem in a specific group of people 92 Engineering Productivity Group
93. Knowledge about release and build pipeline ecosystem in a specific group of people 93 Making a new release Engineering Productivity Group
94. Knowledge about release and build pipeline ecosystem in a specific group of people 94 Making a new release Maintaining and evolving the build pipeline (build system, CI/CD, etc.) Engineering Productivity Group
95. 95 Enable autonomous release process
96. 96 Enable autonomous release process Automate the release process 1-click release of new product versions
97. 97 Enable autonomous release process Enable autonomous change of build pipeline ecosystem and infra Automate the release process 1-click release of new product versions
98. 98 Enable autonomous release process Enable autonomous change of build pipeline ecosystem and infra Automate the release process 1-click release of new product versions Leverage Pipeline as Code to change the build pipeline Implement a modern and easily changed build system (using Cake)
99. 99 Enable autonomous release process Enable autonomous change of build pipeline ecosystem and infra Automate the release process 1-click release of new product versions Leverage Pipeline as Code to change the build pipeline Implement a modern and easily changed build system (using Cake) Ongoing effort...
100. 10 0 Lesson Learned In order to grow and scale, teams should have full control on how they validate and deliver the product areas they own
101. Setting the stage About me About OutSystems Continuous Delivery journey The beginning Efficiency phase Acceleration phase Autonomy phase Takeaways
102. 10 2
103. Autonomy 10 3
104. Autonomy Teams need autonomy on their product area 10 4
105. Autonomy Teams need autonomy on their product area Teams need autonomy and full control on their delivery process 10 5
106. Autonomy Teams need autonomy on their product area Teams need autonomy and full control on their delivery process Infrastructure 10 6
107. Autonomy Teams need autonomy on their product area Teams need autonomy and full control on their delivery process Infrastructure Infrastructure should be a concern from day 0 10 7
108. Autonomy Teams need autonomy on their product area Teams need autonomy and full control on their delivery process Infrastructure Infrastructure should be a concern from day 0 Elasticity and reliability are crucial to scale CI/CD 10 8
109. Autonomy Teams need autonomy on their product area Teams need autonomy and full control on their delivery process Infrastructure Infrastructure should be a concern from day 0 Elasticity and reliability are crucial to scale CI/CD Fast Feedback 10 9
110. Autonomy Teams need autonomy on their product area Teams need autonomy and full control on their delivery process Infrastructure Infrastructure should be a concern from day 0 Elasticity and reliability are crucial to scale CI/CD Fast Feedback Developers should have very fast feedback on their product changes 11 0
111. Autonomy Teams need autonomy on their product area Teams need autonomy and full control on their delivery process Infrastructure Infrastructure should be a concern from day 0 Elasticity and reliability are crucial to scale CI/CD Fast Feedback Developers should have very fast feedback on their product changes Everyone needs to know how users use the product as soon as possible 11 1
112. Autonomy Teams need autonomy on their product area Teams need autonomy and full control on their delivery process Infrastructure Infrastructure should be a concern from day 0 Elasticity and reliability are crucial to scale CI/CD Fast Feedback Developers should have very fast feedback on their product changes Everyone needs to know how users use the product as soon as possible Treat your Build Pipeline as a Product 11 2
113. Autonomy Teams need autonomy on their product area Teams need autonomy and full control on their delivery process Infrastructure Infrastructure should be a concern from day 0 Elasticity and reliability are crucial to scale CI/CD Fast Feedback Developers should have very fast feedback on their product changes Everyone needs to know how users use the product as soon as possible Treat your Build Pipeline as a Product Build pipeline is not a “side project”, it is an important product 11 3
114. Autonomy Teams need autonomy on their product area Teams need autonomy and full control on their delivery process Infrastructure Infrastructure should be a concern from day 0 Elasticity and reliability are crucial to scale CI/CD Fast Feedback Developers should have very fast feedback on their product changes Everyone needs to know how users use the product as soon as possible Treat your Build Pipeline as a Product Build pipeline is not a “side project”, it is an important product Build pipeline needs constant love (e.g., watch out for feedback loops) 11 4
115. Thank You Q&A :-) Diogo Oliveira diogo.oliveira@outsystems.com