Orchestration in meatspace

Traditional enterprises typically rely on a lot of orchestration to deploy their software, which makes the whole process slow and error prone. How can we move them to a more autonomous approach, in order to speed up the feedback cycle from idea to production?

This slideshow was presented at ChefConf 2014 and DevOps Summit Amsterdam 2014.

A recorded video of the presentation at ChefConf can be found at https://www.youtube.com/watch?v=lj8TwsM8xl0

For more details on this topic see my blog http://niek.bartholomeus.be

Transcript

1. in meatspace Orchestration
2. Niek Bartholomeus @niekbartho DevOps consultant with a dev background Me http://niek.bartholomeus.be
3. Two parallel universes http://music.cbc.ca/#/blogs/2013/11/Remembrance-Day-playlist- songs-from-the-Great-War … in a traditional enterprise: … in a modern company: Software delivery
4. Two parallel universes http://music.cbc.ca/#/blogs/2013/11/Remembrance-Day-playlist- songs-from-the-Great-War … in a traditional enterprise: … in a modern company: Software delivery Orchestration Autonomy Releasing software is DIFFICULT Releasing software is EASY
5. ! Traditional enterprises: - company-wide infrequent orchestrated releases - whole infrastructure is brought down - lots of teams involved, lots of manual work - high ceremony, brittle, error prone ! Modern companies: - Autonomous self-empowered teams - ops embedded in dev teams - loosely coupled architectures - continuously deploying new features Notes
6. Foundations of configuration management Mark Burgess Move from centrally managed, tightly coupled systems to autonomous systems making promises to each other to improve the reliability of our infrastructure
7. This need for autonomy is also found in the book “In Search of Certainty” of Mark Burgess. This mindset is also the foundation of the modern configuration management tools which prefer a loose coupling with the systems they manage (i.e. the nodes decide when and how they commit to their promise) Therefore these tools thrive in the autonomous “universe” Notes
8. How can we move traditional enterprises to more autonomy? Our mission:
9. It seems that if we want to improve software delivery in enterprises we have to move them towards more autonomy so our mission of today will be to find ways to do that. Notes
10. 1. Problem space 2. Scientific foundation 3. Solutions Agenda:
11. Study of the problem space Step 1:
12. Problem space 1. Hierarchical organization chart T o p ! d o w n n
13. Decisions making is centralized at the top of the hierarchy and is imposed to the lower layers who simply execute the instructions Issues: 1 Technical decisions are made far away from where the knowledge sits 2 Not motivating for knowledge workers at the bottom of the hierarchy (results in lack of good people and poorly designed systems) => needs a huge communication channel top-down Notes
14. Problem space 2. Standardization
15. False assumption that the environment is stable and that the users’ needs and the way the world behaves (and therefore the automation work that needs to be done) can all be exactly planned upfront (reductionism, determinism, “follow the plan”) Improve efficiency by standardizing the work flows and high job specialization (functional silos) “Optimization of resources over speed of delivery” Notes
16. Problem space 3. Tightly integrated application landscape
17. Many heterogeneous and tightly integrated applications and consequently many multi-application feature requests This and standardization are the main cause for company-wide releases: just follow the plan and make sure your part is ready in time The smallest deviation from the plan within one team can cause major chain reactions in the other applications Notes
18. Problem space 4. Manual work
19. Software configuration management like - keeping track of which versions of the software components belong together - What is deployed in which environment Deployment requests and follow ups (Word, Excel and email) Release orchestration (planning, approvals, deployments, …) Notes
20. My personal theory Automating business processes is complex creative dynamic
21. My personal theory Automating business processes is complex creative dynamic Therefore we need a quick feedback cycle from idea to production
22. My personal theory Automating business processes is complex creative dynamic Can only work when releasing software is easy Therefore we need a quick feedback cycle from idea to production
23. My personal theory Automating business processes is complex creative dynamic Can only work when releasing software is easy Therefore we need a quick feedback cycle from idea to production For this we need autonomy
24. requires My personal theory Automating business processes is complex creative dynamic Can only work when releasing software is easy Therefore we need a quick feedback cycle from idea to production For this we need autonomy
25. Complex: both on business (inherent complexity of business and unpredictable nature of human behavior) and technical (lots of technologies, huge infrastructures) side Creative: every problem is a new problem, it’s always “the first time”, no recurrence (all recurrent processes should be automated) Dynamic: both business needs as technical capacities change at a rapid pace ! In such an environment we need specialists of all domains (business and technical) using all their creativity, closely collaborating, continuously validating their ideas with the reality => quick feedback cycle needed => performant release process needed => as we just saw, this requires autonomy ! Notes
26. What is autonomy? Autonomous unit = a unit(1) that has the freedom to do its work(2) independently(3) (1) unit = the development team Definition: In our context: (2) work = building new features (3) independent = be self-sufficient for all services or technologies that are needed to do the work
27. What is autonomy? Possible dependencies: • On infrastructure Very tight - keep them within the team => Build cross-functional teams
28. What is autonomy? Possible dependencies: • On infrastructure • On other applications Very tight - keep them within the team => Build cross-functional teams Avoid them where possible, limit them otherwise => Support decoupled architectures, backward compatibility
29. What is autonomy? Possible dependencies: • On infrastructure • On other applications • On management Very tight - keep them within the team => Build cross-functional teams Avoid them where possible, limit them otherwise => Support decoupled architectures, backward compatibility Technical decisions must be made inside the team => Create self-empowered teams
30. Building our solution on a scientific foundation Step 2:
31. A theory of organization structuring Henry Mintzberg Published in 1979, “The Structuring of Organizations” is an empirical study of how organizations structure themselves to consistently bring value based on the environment they work in
32. The Structuring of Organizations Organized human activity = division of labor + coordination of the tasks Definition:
33. The Structuring of Organizations Coordination mechanisms: Mutual adjustment Direct supervision Standardization
34. The Structuring of Organizations Design parameters: ! • Job specialization • Behavior formalization • Unit grouping • Unit size • Decentralization of decision making • …
35. The Structuring of Organizations Situational factors: ! • Complexity of the environment • Stability of the environment • Diversity of the environment • Age and size of the organization • …
36. The Structuring of Organizations Common configurations: ! • Administrative bureaucracy • Professional bureaucracy • Adhocracy • …
37. Applying the theory to our environment 1/2 • Job specialization: specialists are needed but they should have general knowledge ! • Behavior formalization: limited - mostly through training and knowledge-sharing (work shops, communities, …)
38. Applying the theory to our environment 2/2 • Unit grouping: by end-to-end service, not by specialization ! • Unit size: small because we must rely on mutual adjustment as the coordination mechanism ! • Decentralization of decision making: yes - technical decisions must be made where the knowledge is: on the work floor
39. Mintzberg’s findings Environment is complex creative dynamic requires Small units grouped by service coordinating through mutual adjustment and empowered to take technical decisions
40. Conclusion requires requires ≈ Small units grouped by service coordinating through mutual adjustment and empowered to take technical decisions cross-functional teams self-empowered teams Environment is complex creative dynamic
41. Choice of organization structure complex creative dynamic requires adhocracy simple stable requires administrative bureaucracy
42. Choice of organization structure complex creative dynamic requires adhocracy simple stable requires administrative bureaucracy Traditional enterprises
43. An environment that is complex, creative and dynamic requires an adhocracy An environment that is simple and stable require an administrative bureacracy Traditional enterprises are organized like an administrative bureaucracy It seems that traditional enterprises have chosen the wrong organization structure! Notes
44. How can we move traditional enterprises to more autonomy? Our mission:
45. How can we move traditional enterprises to more autonomy? Our mission: from an administrative bureaucracy to an adhocracy? Revised
46. Put the theory into practice! Step 3:
47. Our action plan 3.1. Enhance the existing communication flows between the teams ! ! 3.2. Increase autonomy a. between the different development teams and their applications ! ! b. between the components of each application Tactical solution Strategi c Unicorn solution
48. Enhance the existing communication flows Step 3.1: Tactical solution
49. Enhance the communication flows Feature tracking
50. Enhance the communication flows Version control Feature tracking
51. Enhance the communication flows Version control Continuous integration Feature tracking
52. Enhance the communication flows Version control Software repository Continuous integration Feature tracking
53. Define a simple and consistent process of software delivery and automate it where possible Each tool should be integrated with the other tools so that the users can see all the relevant information for their domain ! Steps - integrate feature tracking with version control - set up continuous integration and a software repository for all built components ! ! ! Notes
54. Visualize the information
55. The resulting build artifacts must be well visualized with all the necessary context so each team can quickly see what they are interested in
56. Enhance the communication flows Version control Software repository Continuous integration Feature tracking
57. Enhance the communication flows Version control Software repository Continuous integration Feature tracking Deployment automation ! Infrastructure
58. Enhance the communication flows Release orchestration Version control Software repository Continuous integration Feature tracking Deployment automation ! Infrastructure
59. set up deployment automation facilitate release orchestration with tooling and integrate it with the existing tools This is the least common area of automation in the schema so let us have a closer look at what I mean exactly ! ! Notes
60. Multi-application features Application 1 Application 2 Application 50 … Feature A Feature B Feature C Organization XYZ - Release 2014.1 X X X X X X
61. Because of the tightly integrated application landscape a lot of features span multiple applications. ! Notes
62. dev + unit test Release orchestration Application 1 Application 2 Application 50 … Organization XYZ - Release 2014.1 dev + unit test system test UI test dev + unit test API test integration test acceptance test release release orchestration Development Integration Acceptance Prod Environment
63. Because of a false belief in planning (see earlier) traditional enterprises think that introducing company-wide releases is the easiest way to tackle integration issues As soon as more than one application is under test, we must move into shared environments for all apps The tactical solution is to industrialize the orchestration that is required to make this work: move from Word, Excel and mail to a tool that allows for - creation of deployment requests by devs - approval of deployment requests by release management, based on the status of the implemented features - groups the deployment request in one big run for deployment - coordinate with the ops teams who will do the deployment - manual steps and manual interventions if an automation fails (happens frequently in traditional enterprises) Notes
64. Visualize the information
65. The developer selects an application and a release. Automatically a list of the applicable change requests is shown, including their statuses. This made it easier for the release coordinator to validate the deployment requests For each deployment of a component he appends a step and selects the applicable version number. Automatically the relevant deployment related information is shown. ! Notes
66. Dependencies on the infrastructure Application Packages Packages Packages Packages Packages Config files Packages Packages User acounts Packages Packages Dirs Files Packages Packages Services Packages Packages Cron schedules Server Model Packages … … Configs … … { Development Operations maintain consult sync
67. Besides the dependencies between the dev teams there was also the more tight dependency on the infrastructure side This can be tackled by using a declarative model to communicate the system configurations of the infrastructure to Development - Improves the visibility of the dependencies (devs can easily see the current state of the infrastructure, no more false assumptions) - no more need to wade through historic deployment instructions to find out what exactly and how it was installed on a particular server - Makes it possible to easily provision test systems that are similar to production (infra-as-code/infra-as-model) Notes
68. Increase the autonomy between the development teams Step 3.2: Strategic solution
69. dev + unit test Orchestrated releases Application 1 Application 2 Application 50 … Organization XYZ - Release 2014.1 dev + unit test system test UI test dev + unit test API test integration test acceptance test release orchestration Development Integration Acceptance Prod autonomy Environment
70. The tactical solution was to streamline and automate the existing communication flows of release orchestration in order to have a better visibility on all the dependencies that exist between the different teams. But if one team fails to keep its commitment to the plan it will still cause a chain reaction in the other teams. The strategic solution is to get rid of this orchestration, to move towards autonomy. Notes
71. Feature C Wait time Application 1 Feature A Feature X Feature Y Feature Z multi-app feature single-app feature Orchestrated Release wait time
72. Orchestrated releases can also cause considerable wait time causing a missed opportunity to bring value quickly the the end users. Notes
73. Single-application releases Application 1 Application 2 Application 50 … Feature A Feature B Feature C Organization XYZ - Release 2014.1 X X X X X X
74. Single-application releases Application 1 Feature A Feature C App 1 - Release 2014.1.3 Requires backward compatibility Application 2 Application 50 … Feature A Feature B Feature C Organization XYZ - Release 2014.1 X X X X X X
75. We can move away from company-wide orchestrated releases by making our application backward compatible with the other applications Notes
76. Backward compatibility App 1 App 2 App 3 App 4
77. Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1
78. Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1
79. Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1
80. Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1
81. Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1 Breaking change
82. Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1 2 Breaking change
83. Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1 2 Breaking change
84. Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1 2 3 Breaking change
85. Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1 2 3 Breaking change
86. Backward compatibility App 2 App 3 App 4 App 1
87. Backward compatibility App 2 App 3 App 4 Addition e.g. add firstName 1 App 1 Deprecation e.g. keep fName expand
88. Backward compatibility App 2 App 3 App 4 Addition e.g. add firstName 1 App 1 Backward compatible Deprecation e.g. keep fName expand
89. Backward compatibility App 2 App 3 App 4 Addition e.g. add firstName 1 2 App 1 Backward compatible Deprecation e.g. keep fName expand
90. Backward compatibility App 2 App 3 App 4 Addition e.g. add firstName 1 2 3 App 1 Backward compatible Deprecation e.g. keep fName expand
91. Backward compatibility App 2 App 3 App 4 Addition e.g. add firstName 1 2 3 App 1 Backward compatible 4 expand contract
92. Backward compatibility follows the expand/contract principle (add/deprecate) which allows for a grace period for the depending apps to upgrade As soon as we support backward compatibility we become independent from the organisation- wide release: whatever delays occur there we are not impacted With this approach there is still quite some orchestration happening (4 steps), but the difference is that the decision making power has moved down from the organization to the application. Cf our scientific section we have changed the coordination mechanism from direct supervision (central orchestration) to mutual adjustment between the autonomous applications. They only have to take into account one dependency: make sure that we deliver our multi-app feature BEFORE the organization-wide release date. Supporting backward compatibility requires applications to have a well-structured/modular code base and a decoupled architecture There are several scales on which to apply the backward compatibility (two apps side by side, a v2 path in a rest resource, adding a field in It also requires that the depending apps are released frequently, to avoid the need to support too many deprecated features in parallel Notes
93. Feature C Reduced wait time Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z Orchestrated Release App Release wait time
94. Another advantage of becoming independent from the company-wide release is that we can start releasing on our own tempo. As a result the wait time decreases and the feedback cycle increases so the users will get added value sooner. Notes
95. Increase the autonomy between the components of the application Step 3.2.b: Unicorn solution
96. Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
97. Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
98. Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
99. Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
100. Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
101. Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
102. If we want an even faster feedback cycle we can use feature flags for not-yet-finished features to reduce the wait time to zero for each feature. This causes an increase in the number of releases and consequently a faster feedback cycle, which is a good thing. It may also have a negative side effect if the application requires down time during the deployment because of a temporary inconsistency period that may arise when the components of the application are deployed, e.g. the new version of the application code requires the new version of the db schema and vice versa. This is a similar problem as with the company-wide release, only on a lower level: the level of the application. The solution is also similar: we have to make the components backward compatible (and in general more autonomous) to arrive at zero-downtime deployments Because the communications on this component level is much more granular supporting backward compatibility on this level is much more difficult. Notes
103. Database fName Application read fName write fName Backward compatibility
104. Database fName fName firstName Add e.g. add firstName 1 Deprecate e.g. keep fName expand Application read fName write fName Backward compatibility
105. Database fName fName firstName Add e.g. add firstName 1 Deprecate e.g. keep fName expand Application read fName write fName Backward compatibility 2 write firstName Duplicate write
106. Database fName fName firstName Add e.g. add firstName 1 Deprecate e.g. keep fName expand Application read fName write fName Backward compatibility 2 write firstName Migrate data 3 Duplicate write
107. Database fName fName firstName Add e.g. add firstName 1 Deprecate e.g. keep fName expand Application read fName write fName Backward compatibility 2 write firstName Migrate data 3 read firstName 4 Duplicate write Modify read Deduplicate write
108. Database firstName Add e.g. add firstName 1 expand Application read fName write fName Backward compatibility 2 write firstName Migrate data 3 read firstName 4 contract 5 Duplicate write Modify read Deduplicate write
109. Making database changes backward compatible, e.g. changing the name of a field, requires a lot of orchestration Notes
110. Unicorns
111. This solution is typically applied at some of the most successful digital companies that exist because they apply extremely fast feedback cycles (the reason of their success) and cannot afford any downtime. Having a centralized orchestrator is not interesting for them because they work in huge infrastructures with high scalability needs and high numbers of applications and components that continuously need changing. ! ! Fully cross-functional and self-empowered teams Huge infra with high scalability needs Fully automated release pipeline (deployments and testing) No-downtime deployments Fully decoupled architectures with autonomous applications and components (e.g. app code and db changes, communication with load balancers and monitoring systems) Backward compatibility, feature flagging, dark launching Notes
112. Traditional separation Application Packages Packages Packages Packages Packages Config files Packages Packages User acounts Packages Packages Dirs Files Packages Packages Services Packages Packages Cron schedules Server Model Packages … … Configs … … { Development Operations maintain consult sync
113. Modern separation Application Packages Packages Packages Packages Packages Config files Packages Packages User acounts Packages Packages Dirs Files Packages Packages Services Packages Packages Cron schedules Server Model Application … … Packages … … Configs … … sync Creative Automated Devs & infra specialists maintain
114. We have to move from the traditional separation between dev and ops to a separation between creative and automated/standardized Functional and operational evolutions of the app are done in the Creative side Repetitive work (i.e. traditional day-to-day ops work) is automated in the Automated side Include the application in the model: - one declarative model for the application with the system configurations it needs - one tool for app deployment and CM (makes sense because we now have only one team anyway) - drift management for the application - orchestration needs can be covered by the components through mutual coordination using the information from the model (or by tooling if it gets too complex: it must calculate the exact deployment process from the model/ desired state and the current state) - provisioning and upgrading becomes possible for less-technical people Notes
115. Summary: Orchestration problems Enhance the existing flows Increase autonomy
116. Niek Bartholomeus @niekbartho Contact me! http://niek.bartholomeus.be niek.bartholomeus@gmail.com