Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Orchestration in meatspace

Orchestration in meatspace

Traditional enterprises typically rely on a lot of orchestration to deploy their software, which makes the whole process slow and error prone. How can we move them to a more autonomous approach, in order to speed up the feedback cycle from idea to production?

This slideshow was presented at ChefConf 2014 and DevOps Summit Amsterdam 2014.

A recorded video of the presentation at ChefConf can be found at https://www.youtube.com/watch?v=lj8TwsM8xl0

For more details on this topic see my blog http://niek.bartholomeus.be

29df1d6ab49d2bb0a1eb296d259121d8?s=128

Niek Bartholomeus

April 17, 2014
Tweet

Transcript

  1. in meatspace Orchestration

  2. Niek Bartholomeus @niekbartho DevOps consultant with a dev background Me

    http://niek.bartholomeus.be
  3. Two parallel universes http://music.cbc.ca/#/blogs/2013/11/Remembrance-Day-playlist- songs-from-the-Great-War … in a traditional enterprise:

    … in a modern company: Software delivery
  4. Two parallel universes http://music.cbc.ca/#/blogs/2013/11/Remembrance-Day-playlist- songs-from-the-Great-War … in a traditional enterprise:

    … in a modern company: Software delivery Orchestration Autonomy Releasing software is DIFFICULT Releasing software is EASY
  5. ! Traditional enterprises: - company-wide infrequent orchestrated releases - whole

    infrastructure is brought down - lots of teams involved, lots of manual work - high ceremony, brittle, error prone ! Modern companies: - Autonomous self-empowered teams - ops embedded in dev teams - loosely coupled architectures - continuously deploying new features Notes
  6. Foundations of configuration management Mark Burgess Move from centrally managed,

    tightly coupled systems to autonomous systems making promises to each other to improve the reliability of our infrastructure
  7. This need for autonomy is also found in the book

    “In Search of Certainty” of Mark Burgess. This mindset is also the foundation of the modern configuration management tools which prefer a loose coupling with the systems they manage (i.e. the nodes decide when and how they commit to their promise) Therefore these tools thrive in the autonomous “universe” Notes
  8. How can we move traditional enterprises to more autonomy? Our

    mission:
  9. It seems that if we want to improve software delivery

    in enterprises we have to move them towards more autonomy so our mission of today will be to find ways to do that. Notes
  10. 1. Problem space 2. Scientific foundation 3. Solutions Agenda:

  11. Study of the problem space Step 1:

  12. Problem space 1. Hierarchical organization chart T o p !

    d o w n n
  13. Decisions making is centralized at the top of the hierarchy

    and is imposed to the lower layers who simply execute the instructions Issues: 1 Technical decisions are made far away from where the knowledge sits 2 Not motivating for knowledge workers at the bottom of the hierarchy (results in lack of good people and poorly designed systems) => needs a huge communication channel top-down Notes
  14. Problem space 2. Standardization

  15. False assumption that the environment is stable and that the

    users’ needs and the way the world behaves (and therefore the automation work that needs to be done) can all be exactly planned upfront (reductionism, determinism, “follow the plan”) Improve efficiency by standardizing the work flows and high job specialization (functional silos) “Optimization of resources over speed of delivery” Notes
  16. Problem space 3. Tightly integrated application landscape

  17. Many heterogeneous and tightly integrated applications and consequently many multi-application

    feature requests This and standardization are the main cause for company-wide releases: just follow the plan and make sure your part is ready in time The smallest deviation from the plan within one team can cause major chain reactions in the other applications Notes
  18. Problem space 4. Manual work

  19. Software configuration management like - keeping track of which versions

    of the software components belong together - What is deployed in which environment Deployment requests and follow ups (Word, Excel and email) Release orchestration (planning, approvals, deployments, …) Notes
  20. My personal theory Automating business processes is complex creative dynamic

  21. My personal theory Automating business processes is complex creative dynamic

    Therefore we need a quick feedback cycle from idea to production
  22. My personal theory Automating business processes is complex creative dynamic

    Can only work when releasing software is easy Therefore we need a quick feedback cycle from idea to production
  23. My personal theory Automating business processes is complex creative dynamic

    Can only work when releasing software is easy Therefore we need a quick feedback cycle from idea to production For this we need autonomy
  24. requires My personal theory Automating business processes is complex creative

    dynamic Can only work when releasing software is easy Therefore we need a quick feedback cycle from idea to production For this we need autonomy
  25. Complex: both on business (inherent complexity of business and unpredictable

    nature of human behavior) and technical (lots of technologies, huge infrastructures) side Creative: every problem is a new problem, it’s always “the first time”, no recurrence (all recurrent processes should be automated) Dynamic: both business needs as technical capacities change at a rapid pace ! In such an environment we need specialists of all domains (business and technical) using all their creativity, closely collaborating, continuously validating their ideas with the reality => quick feedback cycle needed => performant release process needed => as we just saw, this requires autonomy ! Notes
  26. What is autonomy? Autonomous unit = a unit(1) that has

    the freedom to do its work(2) independently(3) (1) unit = the development team Definition: In our context: (2) work = building new features (3) independent = be self-sufficient for all services or technologies that are needed to do the work
  27. What is autonomy? Possible dependencies: • On infrastructure Very tight

    - keep them within the team => Build cross-functional teams
  28. What is autonomy? Possible dependencies: • On infrastructure • On

    other applications Very tight - keep them within the team => Build cross-functional teams Avoid them where possible, limit them otherwise => Support decoupled architectures, backward compatibility
  29. What is autonomy? Possible dependencies: • On infrastructure • On

    other applications • On management Very tight - keep them within the team => Build cross-functional teams Avoid them where possible, limit them otherwise => Support decoupled architectures, backward compatibility Technical decisions must be made inside the team => Create self-empowered teams
  30. Building our solution on a scientific foundation Step 2:

  31. A theory of organization structuring Henry Mintzberg Published in 1979,

    “The Structuring of Organizations” is an empirical study of how organizations structure themselves to consistently bring value based on the environment they work in
  32. The Structuring of Organizations Organized human activity = division of

    labor + coordination of the tasks Definition:
  33. The Structuring of Organizations Coordination mechanisms: Mutual adjustment Direct supervision

    Standardization
  34. The Structuring of Organizations Design parameters: ! • Job specialization

    • Behavior formalization • Unit grouping • Unit size • Decentralization of decision making • …
  35. The Structuring of Organizations Situational factors: ! • Complexity of

    the environment • Stability of the environment • Diversity of the environment • Age and size of the organization • …
  36. The Structuring of Organizations Common configurations: ! • Administrative bureaucracy

    • Professional bureaucracy • Adhocracy • …
  37. Applying the theory to our environment 1/2 • Job specialization:

    specialists are needed but they should have general knowledge ! • Behavior formalization: limited - mostly through training and knowledge-sharing (work shops, communities, …)
  38. Applying the theory to our environment 2/2 • Unit grouping:

    by end-to-end service, not by specialization ! • Unit size: small because we must rely on mutual adjustment as the coordination mechanism ! • Decentralization of decision making: yes - technical decisions must be made where the knowledge is: on the work floor
  39. Mintzberg’s findings Environment is complex creative dynamic requires Small units

    grouped by service coordinating through mutual adjustment and empowered to take technical decisions
  40. Conclusion requires requires ≈ Small units grouped by service coordinating

    through mutual adjustment and empowered to take technical decisions cross-functional teams self-empowered teams Environment is complex creative dynamic
  41. Choice of organization structure complex creative dynamic requires adhocracy simple

    stable requires administrative bureaucracy
  42. Choice of organization structure complex creative dynamic requires adhocracy simple

    stable requires administrative bureaucracy Traditional enterprises
  43. An environment that is complex, creative and dynamic requires an

    adhocracy An environment that is simple and stable require an administrative bureacracy Traditional enterprises are organized like an administrative bureaucracy It seems that traditional enterprises have chosen the wrong organization structure! Notes
  44. How can we move traditional enterprises to more autonomy? Our

    mission:
  45. How can we move traditional enterprises to more autonomy? Our

    mission: from an administrative bureaucracy to an adhocracy? Revised
  46. Put the theory into practice! Step 3:

  47. Our action plan 3.1. Enhance the existing communication flows between

    the teams ! ! 3.2. Increase autonomy a. between the different development teams and their applications ! ! b. between the components of each application Tactical solution Strategi c Unicorn solution
  48. Enhance the existing communication flows Step 3.1: Tactical solution

  49. Enhance the communication flows Feature tracking

  50. Enhance the communication flows Version control Feature tracking

  51. Enhance the communication flows Version control Continuous integration Feature tracking

  52. Enhance the communication flows Version control Software repository Continuous integration

    Feature tracking
  53. Define a simple and consistent process of software delivery and

    automate it where possible Each tool should be integrated with the other tools so that the users can see all the relevant information for their domain ! Steps - integrate feature tracking with version control - set up continuous integration and a software repository for all built components ! ! ! Notes
  54. Visualize the information

  55. The resulting build artifacts must be well visualized with all

    the necessary context so each team can quickly see what they are interested in
  56. Enhance the communication flows Version control Software repository Continuous integration

    Feature tracking
  57. Enhance the communication flows Version control Software repository Continuous integration

    Feature tracking Deployment automation ! Infrastructure
  58. Enhance the communication flows Release orchestration Version control Software repository

    Continuous integration Feature tracking Deployment automation ! Infrastructure
  59. set up deployment automation facilitate release orchestration with tooling and

    integrate it with the existing tools This is the least common area of automation in the schema so let us have a closer look at what I mean exactly ! ! Notes
  60. Multi-application features Application 1 Application 2 Application 50 … Feature

    A Feature B Feature C Organization XYZ - Release 2014.1 X X X X X X
  61. Because of the tightly integrated application landscape a lot of

    features span multiple applications. ! Notes
  62. dev + unit test Release orchestration Application 1 Application 2

    Application 50 … Organization XYZ - Release 2014.1 dev + unit test system test UI test dev + unit test API test integration test acceptance test release release orchestration Development Integration Acceptance Prod Environment
  63. Because of a false belief in planning (see earlier) traditional

    enterprises think that introducing company-wide releases is the easiest way to tackle integration issues As soon as more than one application is under test, we must move into shared environments for all apps The tactical solution is to industrialize the orchestration that is required to make this work: move from Word, Excel and mail to a tool that allows for - creation of deployment requests by devs - approval of deployment requests by release management, based on the status of the implemented features - groups the deployment request in one big run for deployment - coordinate with the ops teams who will do the deployment - manual steps and manual interventions if an automation fails (happens frequently in traditional enterprises) Notes
  64. Visualize the information

  65. The developer selects an application and a release. Automatically a

    list of the applicable change requests is shown, including their statuses. This made it easier for the release coordinator to validate the deployment requests For each deployment of a component he appends a step and selects the applicable version number. Automatically the relevant deployment related information is shown. ! Notes
  66. Dependencies on the infrastructure Application Packages Packages Packages Packages Packages

    Config files Packages Packages User acounts Packages Packages Dirs Files Packages Packages Services Packages Packages Cron schedules Server Model Packages … … Configs … … { Development Operations maintain consult sync
  67. Besides the dependencies between the dev teams there was also

    the more tight dependency on the infrastructure side This can be tackled by using a declarative model to communicate the system configurations of the infrastructure to Development - Improves the visibility of the dependencies (devs can easily see the current state of the infrastructure, no more false assumptions) - no more need to wade through historic deployment instructions to find out what exactly and how it was installed on a particular server - Makes it possible to easily provision test systems that are similar to production (infra-as-code/infra-as-model) Notes
  68. Increase the autonomy between the development teams Step 3.2: Strategic

    solution
  69. dev + unit test Orchestrated releases Application 1 Application 2

    Application 50 … Organization XYZ - Release 2014.1 dev + unit test system test UI test dev + unit test API test integration test acceptance test release orchestration Development Integration Acceptance Prod autonomy Environment
  70. The tactical solution was to streamline and automate the existing

    communication flows of release orchestration in order to have a better visibility on all the dependencies that exist between the different teams. But if one team fails to keep its commitment to the plan it will still cause a chain reaction in the other teams. The strategic solution is to get rid of this orchestration, to move towards autonomy. Notes
  71. Feature C Wait time Application 1 Feature A Feature X

    Feature Y Feature Z multi-app feature single-app feature Orchestrated Release wait time
  72. Orchestrated releases can also cause considerable wait time causing a

    missed opportunity to bring value quickly the the end users. Notes
  73. Single-application releases Application 1 Application 2 Application 50 … Feature

    A Feature B Feature C Organization XYZ - Release 2014.1 X X X X X X
  74. Single-application releases Application 1 Feature A Feature C App 1

    - Release 2014.1.3 Requires backward compatibility Application 2 Application 50 … Feature A Feature B Feature C Organization XYZ - Release 2014.1 X X X X X X
  75. We can move away from company-wide orchestrated releases by making

    our application backward compatible with the other applications Notes
  76. Backward compatibility App 1 App 2 App 3 App 4

  77. Backward compatibility App 1 App 2 App 3 App 4

    Modification e.g. change fName to firstName 1
  78. Backward compatibility App 1 App 2 App 3 App 4

    Modification e.g. change fName to firstName 1
  79. Backward compatibility App 1 App 2 App 3 App 4

    Modification e.g. change fName to firstName 1
  80. Backward compatibility App 1 App 2 App 3 App 4

    Modification e.g. change fName to firstName 1
  81. Backward compatibility App 1 App 2 App 3 App 4

    Modification e.g. change fName to firstName 1 Breaking change
  82. Backward compatibility App 1 App 2 App 3 App 4

    Modification e.g. change fName to firstName 1 2 Breaking change
  83. Backward compatibility App 1 App 2 App 3 App 4

    Modification e.g. change fName to firstName 1 2 Breaking change
  84. Backward compatibility App 1 App 2 App 3 App 4

    Modification e.g. change fName to firstName 1 2 3 Breaking change
  85. Backward compatibility App 1 App 2 App 3 App 4

    Modification e.g. change fName to firstName 1 2 3 Breaking change
  86. Backward compatibility App 2 App 3 App 4 App 1

  87. Backward compatibility App 2 App 3 App 4 Addition e.g.

    add firstName 1 App 1 Deprecation e.g. keep fName expand
  88. Backward compatibility App 2 App 3 App 4 Addition e.g.

    add firstName 1 App 1 Backward compatible Deprecation e.g. keep fName expand
  89. Backward compatibility App 2 App 3 App 4 Addition e.g.

    add firstName 1 2 App 1 Backward compatible Deprecation e.g. keep fName expand
  90. Backward compatibility App 2 App 3 App 4 Addition e.g.

    add firstName 1 2 3 App 1 Backward compatible Deprecation e.g. keep fName expand
  91. Backward compatibility App 2 App 3 App 4 Addition e.g.

    add firstName 1 2 3 App 1 Backward compatible 4 expand contract
  92. Backward compatibility follows the expand/contract principle (add/deprecate) which allows for

    a grace period for the depending apps to upgrade As soon as we support backward compatibility we become independent from the organisation- wide release: whatever delays occur there we are not impacted With this approach there is still quite some orchestration happening (4 steps), but the difference is that the decision making power has moved down from the organization to the application. Cf our scientific section we have changed the coordination mechanism from direct supervision (central orchestration) to mutual adjustment between the autonomous applications. They only have to take into account one dependency: make sure that we deliver our multi-app feature BEFORE the organization-wide release date. Supporting backward compatibility requires applications to have a well-structured/modular code base and a decoupled architecture There are several scales on which to apply the backward compatibility (two apps side by side, a v2 path in a rest resource, adding a field in It also requires that the depending apps are released frequently, to avoid the need to support too many deprecated features in parallel Notes
  93. Feature C Reduced wait time Application 1 Feature A Feature

    X Feature Y multi-app feature single-app feature Feature Z Orchestrated Release App Release wait time
  94. Another advantage of becoming independent from the company-wide release is

    that we can start releasing on our own tempo. As a result the wait time decreases and the feedback cycle increases so the users will get added value sooner. Notes
  95. Increase the autonomy between the components of the application Step

    3.2.b: Unicorn solution
  96. Feature C Increase the feedback cycle Application 1 Feature A

    Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
  97. Feature C Increase the feedback cycle Application 1 Feature A

    Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
  98. Feature C Increase the feedback cycle Application 1 Feature A

    Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
  99. Feature C Increase the feedback cycle Application 1 Feature A

    Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
  100. Feature C Increase the feedback cycle Application 1 Feature A

    Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
  101. Feature C Increase the feedback cycle Application 1 Feature A

    Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time
  102. If we want an even faster feedback cycle we can

    use feature flags for not-yet-finished features to reduce the wait time to zero for each feature. This causes an increase in the number of releases and consequently a faster feedback cycle, which is a good thing. It may also have a negative side effect if the application requires down time during the deployment because of a temporary inconsistency period that may arise when the components of the application are deployed, e.g. the new version of the application code requires the new version of the db schema and vice versa. This is a similar problem as with the company-wide release, only on a lower level: the level of the application. The solution is also similar: we have to make the components backward compatible (and in general more autonomous) to arrive at zero-downtime deployments Because the communications on this component level is much more granular supporting backward compatibility on this level is much more difficult. Notes
  103. Database fName Application read fName write fName Backward compatibility

  104. Database fName fName firstName Add e.g. add firstName 1 Deprecate

    e.g. keep fName expand Application read fName write fName Backward compatibility
  105. Database fName fName firstName Add e.g. add firstName 1 Deprecate

    e.g. keep fName expand Application read fName write fName Backward compatibility 2 write firstName Duplicate write
  106. Database fName fName firstName Add e.g. add firstName 1 Deprecate

    e.g. keep fName expand Application read fName write fName Backward compatibility 2 write firstName Migrate data 3 Duplicate write
  107. Database fName fName firstName Add e.g. add firstName 1 Deprecate

    e.g. keep fName expand Application read fName write fName Backward compatibility 2 write firstName Migrate data 3 read firstName 4 Duplicate write Modify read Deduplicate write
  108. Database firstName Add e.g. add firstName 1 expand Application read

    fName write fName Backward compatibility 2 write firstName Migrate data 3 read firstName 4 contract 5 Duplicate write Modify read Deduplicate write
  109. Making database changes backward compatible, e.g. changing the name of

    a field, requires a lot of orchestration Notes
  110. Unicorns

  111. This solution is typically applied at some of the most

    successful digital companies that exist because they apply extremely fast feedback cycles (the reason of their success) and cannot afford any downtime. Having a centralized orchestrator is not interesting for them because they work in huge infrastructures with high scalability needs and high numbers of applications and components that continuously need changing. ! ! Fully cross-functional and self-empowered teams Huge infra with high scalability needs Fully automated release pipeline (deployments and testing) No-downtime deployments Fully decoupled architectures with autonomous applications and components (e.g. app code and db changes, communication with load balancers and monitoring systems) Backward compatibility, feature flagging, dark launching Notes
  112. Traditional separation Application Packages Packages Packages Packages Packages Config files

    Packages Packages User acounts Packages Packages Dirs Files Packages Packages Services Packages Packages Cron schedules Server Model Packages … … Configs … … { Development Operations maintain consult sync
  113. Modern separation Application Packages Packages Packages Packages Packages Config files

    Packages Packages User acounts Packages Packages Dirs Files Packages Packages Services Packages Packages Cron schedules Server Model Application … … Packages … … Configs … … sync Creative Automated Devs & infra specialists maintain
  114. We have to move from the traditional separation between dev

    and ops to a separation between creative and automated/standardized Functional and operational evolutions of the app are done in the Creative side Repetitive work (i.e. traditional day-to-day ops work) is automated in the Automated side Include the application in the model: - one declarative model for the application with the system configurations it needs - one tool for app deployment and CM (makes sense because we now have only one team anyway) - drift management for the application - orchestration needs can be covered by the components through mutual coordination using the information from the model (or by tooling if it gets too complex: it must calculate the exact deployment process from the model/ desired state and the current state) - provisioning and upgrading becomes possible for less-technical people Notes
  115. Summary: Orchestration problems Enhance the existing flows Increase autonomy

  116. Niek Bartholomeus @niekbartho Contact me! http://niek.bartholomeus.be niek.bartholomeus@gmail.com