Tweet
Tweet

Slide 1

Slide 1 text

in meatspace Orchestration

Slide 2

Slide 2 text

Niek Bartholomeus @niekbartho DevOps consultant with a dev background Me http://niek.bartholomeus.be

Slide 3

Slide 3 text

Two parallel universes http://music.cbc.ca/#/blogs/2013/11/Remembrance-Day-playlist- songs-from-the-Great-War … in a traditional enterprise: … in a modern company: Software delivery

Slide 4

Slide 4 text

Two parallel universes http://music.cbc.ca/#/blogs/2013/11/Remembrance-Day-playlist- songs-from-the-Great-War … in a traditional enterprise: … in a modern company: Software delivery Orchestration Autonomy Releasing software is DIFFICULT Releasing software is EASY

Slide 5

Slide 5 text

! Traditional enterprises: - company-wide infrequent orchestrated releases - whole infrastructure is brought down - lots of teams involved, lots of manual work - high ceremony, brittle, error prone ! Modern companies: - Autonomous self-empowered teams - ops embedded in dev teams - loosely coupled architectures - continuously deploying new features Notes

Slide 6

Slide 6 text

Foundations of configuration management Mark Burgess Move from centrally managed, tightly coupled systems to autonomous systems making promises to each other to improve the reliability of our infrastructure

Slide 7

Slide 7 text

This need for autonomy is also found in the book “In Search of Certainty” of Mark Burgess. This mindset is also the foundation of the modern configuration management tools which prefer a loose coupling with the systems they manage (i.e. the nodes decide when and how they commit to their promise) Therefore these tools thrive in the autonomous “universe” Notes

Slide 8

Slide 8 text

How can we move traditional enterprises to more autonomy? Our mission:

Slide 9

Slide 9 text

It seems that if we want to improve software delivery in enterprises we have to move them towards more autonomy so our mission of today will be to find ways to do that. Notes

Slide 10

Slide 10 text

1. Problem space 2. Scientific foundation 3. Solutions Agenda:

Slide 11

Slide 11 text

Study of the problem space Step 1:

Slide 12

Slide 12 text

Problem space 1. Hierarchical organization chart T o p ! d o w n n

Slide 13

Slide 13 text

Decisions making is centralized at the top of the hierarchy and is imposed to the lower layers who simply execute the instructions Issues: 1 Technical decisions are made far away from where the knowledge sits 2 Not motivating for knowledge workers at the bottom of the hierarchy (results in lack of good people and poorly designed systems) => needs a huge communication channel top-down Notes

Slide 14

Slide 14 text

Problem space 2. Standardization

Slide 15

Slide 15 text

False assumption that the environment is stable and that the users’ needs and the way the world behaves (and therefore the automation work that needs to be done) can all be exactly planned upfront (reductionism, determinism, “follow the plan”) Improve efficiency by standardizing the work flows and high job specialization (functional silos) “Optimization of resources over speed of delivery” Notes

Slide 16

Slide 16 text

Problem space 3. Tightly integrated application landscape

Slide 17

Slide 17 text

Many heterogeneous and tightly integrated applications and consequently many multi-application feature requests This and standardization are the main cause for company-wide releases: just follow the plan and make sure your part is ready in time The smallest deviation from the plan within one team can cause major chain reactions in the other applications Notes

Slide 18

Slide 18 text

Problem space 4. Manual work

Slide 19

Slide 19 text

Software configuration management like - keeping track of which versions of the software components belong together - What is deployed in which environment Deployment requests and follow ups (Word, Excel and email) Release orchestration (planning, approvals, deployments, …) Notes

Slide 20

Slide 20 text

My personal theory Automating business processes is complex creative dynamic

Slide 21

Slide 21 text

My personal theory Automating business processes is complex creative dynamic Therefore we need a quick feedback cycle from idea to production

Slide 22

Slide 22 text

My personal theory Automating business processes is complex creative dynamic Can only work when releasing software is easy Therefore we need a quick feedback cycle from idea to production

Slide 23

Slide 23 text

My personal theory Automating business processes is complex creative dynamic Can only work when releasing software is easy Therefore we need a quick feedback cycle from idea to production For this we need autonomy

Slide 24

Slide 24 text

requires My personal theory Automating business processes is complex creative dynamic Can only work when releasing software is easy Therefore we need a quick feedback cycle from idea to production For this we need autonomy

Slide 25

Slide 25 text

Complex: both on business (inherent complexity of business and unpredictable nature of human behavior) and technical (lots of technologies, huge infrastructures) side Creative: every problem is a new problem, it’s always “the first time”, no recurrence (all recurrent processes should be automated) Dynamic: both business needs as technical capacities change at a rapid pace ! In such an environment we need specialists of all domains (business and technical) using all their creativity, closely collaborating, continuously validating their ideas with the reality => quick feedback cycle needed => performant release process needed => as we just saw, this requires autonomy ! Notes

Slide 26

Slide 26 text

What is autonomy? Autonomous unit = a unit(1) that has the freedom to do its work(2) independently(3) (1) unit = the development team Definition: In our context: (2) work = building new features (3) independent = be self-sufficient for all services or technologies that are needed to do the work

Slide 27

Slide 27 text

What is autonomy? Possible dependencies: • On infrastructure Very tight - keep them within the team => Build cross-functional teams

Slide 28

Slide 28 text

What is autonomy? Possible dependencies: • On infrastructure • On other applications Very tight - keep them within the team => Build cross-functional teams Avoid them where possible, limit them otherwise => Support decoupled architectures, backward compatibility

Slide 29

Slide 29 text

What is autonomy? Possible dependencies: • On infrastructure • On other applications • On management Very tight - keep them within the team => Build cross-functional teams Avoid them where possible, limit them otherwise => Support decoupled architectures, backward compatibility Technical decisions must be made inside the team => Create self-empowered teams

Slide 30

Slide 30 text

Building our solution on a scientific foundation Step 2:

Slide 31

Slide 31 text

A theory of organization structuring Henry Mintzberg Published in 1979, “The Structuring of Organizations” is an empirical study of how organizations structure themselves to consistently bring value based on the environment they work in

Slide 32

Slide 32 text

The Structuring of Organizations Organized human activity = division of labor + coordination of the tasks Definition:

Slide 33

Slide 33 text

The Structuring of Organizations Coordination mechanisms: Mutual adjustment Direct supervision Standardization

Slide 34

Slide 34 text

The Structuring of Organizations Design parameters: ! • Job specialization • Behavior formalization • Unit grouping • Unit size • Decentralization of decision making • …

Slide 35

Slide 35 text

The Structuring of Organizations Situational factors: ! • Complexity of the environment • Stability of the environment • Diversity of the environment • Age and size of the organization • …

Slide 36

Slide 36 text

The Structuring of Organizations Common configurations: ! • Administrative bureaucracy • Professional bureaucracy • Adhocracy • …

Slide 37

Slide 37 text

Applying the theory to our environment 1/2 • Job specialization: specialists are needed but they should have general knowledge ! • Behavior formalization: limited - mostly through training and knowledge-sharing (work shops, communities, …)

Slide 38

Slide 38 text

Applying the theory to our environment 2/2 • Unit grouping: by end-to-end service, not by specialization ! • Unit size: small because we must rely on mutual adjustment as the coordination mechanism ! • Decentralization of decision making: yes - technical decisions must be made where the knowledge is: on the work floor

Slide 39

Slide 39 text

Mintzberg’s findings Environment is complex creative dynamic requires Small units grouped by service coordinating through mutual adjustment and empowered to take technical decisions

Slide 40

Slide 40 text

Conclusion requires requires ≈ Small units grouped by service coordinating through mutual adjustment and empowered to take technical decisions cross-functional teams self-empowered teams Environment is complex creative dynamic

Slide 41

Slide 41 text

Choice of organization structure complex creative dynamic requires adhocracy simple stable requires administrative bureaucracy

Slide 42

Slide 42 text

Choice of organization structure complex creative dynamic requires adhocracy simple stable requires administrative bureaucracy Traditional enterprises

Slide 43

Slide 43 text

An environment that is complex, creative and dynamic requires an adhocracy An environment that is simple and stable require an administrative bureacracy Traditional enterprises are organized like an administrative bureaucracy It seems that traditional enterprises have chosen the wrong organization structure! Notes

Slide 44

Slide 44 text

How can we move traditional enterprises to more autonomy? Our mission:

Slide 45

Slide 45 text

How can we move traditional enterprises to more autonomy? Our mission: from an administrative bureaucracy to an adhocracy? Revised

Slide 46

Slide 46 text

Put the theory into practice! Step 3:

Slide 47

Slide 47 text

Our action plan 3.1. Enhance the existing communication flows between the teams ! ! 3.2. Increase autonomy a. between the different development teams and their applications ! ! b. between the components of each application Tactical solution Strategi c Unicorn solution

Slide 48

Slide 48 text

Enhance the existing communication flows Step 3.1: Tactical solution

Slide 49

Slide 49 text

Enhance the communication flows Feature tracking

Slide 50

Slide 50 text

Enhance the communication flows Version control Feature tracking

Slide 51

Slide 51 text

Enhance the communication flows Version control Continuous integration Feature tracking

Slide 52

Slide 52 text

Enhance the communication flows Version control Software repository Continuous integration Feature tracking

Slide 53

Slide 53 text

Define a simple and consistent process of software delivery and automate it where possible Each tool should be integrated with the other tools so that the users can see all the relevant information for their domain ! Steps - integrate feature tracking with version control - set up continuous integration and a software repository for all built components ! ! ! Notes

Slide 54

Slide 54 text

Visualize the information

Slide 55

Slide 55 text

The resulting build artifacts must be well visualized with all the necessary context so each team can quickly see what they are interested in

Slide 56

Slide 56 text

Enhance the communication flows Version control Software repository Continuous integration Feature tracking

Slide 57

Slide 57 text

Enhance the communication flows Version control Software repository Continuous integration Feature tracking Deployment automation ! Infrastructure

Slide 58

Slide 58 text

Enhance the communication flows Release orchestration Version control Software repository Continuous integration Feature tracking Deployment automation ! Infrastructure

Slide 59

Slide 59 text

set up deployment automation facilitate release orchestration with tooling and integrate it with the existing tools This is the least common area of automation in the schema so let us have a closer look at what I mean exactly ! ! Notes

Slide 60

Slide 60 text

Multi-application features Application 1 Application 2 Application 50 … Feature A Feature B Feature C Organization XYZ - Release 2014.1 X X X X X X

Slide 61

Slide 61 text

Because of the tightly integrated application landscape a lot of features span multiple applications. ! Notes

Slide 62

Slide 62 text

dev + unit test Release orchestration Application 1 Application 2 Application 50 … Organization XYZ - Release 2014.1 dev + unit test system test UI test dev + unit test API test integration test acceptance test release release orchestration Development Integration Acceptance Prod Environment

Slide 63

Slide 63 text

Because of a false belief in planning (see earlier) traditional enterprises think that introducing company-wide releases is the easiest way to tackle integration issues As soon as more than one application is under test, we must move into shared environments for all apps The tactical solution is to industrialize the orchestration that is required to make this work: move from Word, Excel and mail to a tool that allows for - creation of deployment requests by devs - approval of deployment requests by release management, based on the status of the implemented features - groups the deployment request in one big run for deployment - coordinate with the ops teams who will do the deployment - manual steps and manual interventions if an automation fails (happens frequently in traditional enterprises) Notes

Slide 64

Slide 64 text

Visualize the information

Slide 65

Slide 65 text

The developer selects an application and a release. Automatically a list of the applicable change requests is shown, including their statuses. This made it easier for the release coordinator to validate the deployment requests For each deployment of a component he appends a step and selects the applicable version number. Automatically the relevant deployment related information is shown. ! Notes

Slide 66

Slide 66 text

Dependencies on the infrastructure Application Packages Packages Packages Packages Packages Config files Packages Packages User acounts Packages Packages Dirs Files Packages Packages Services Packages Packages Cron schedules Server Model Packages … … Configs … … { Development Operations maintain consult sync

Slide 67

Slide 67 text

Besides the dependencies between the dev teams there was also the more tight dependency on the infrastructure side This can be tackled by using a declarative model to communicate the system configurations of the infrastructure to Development - Improves the visibility of the dependencies (devs can easily see the current state of the infrastructure, no more false assumptions) - no more need to wade through historic deployment instructions to find out what exactly and how it was installed on a particular server - Makes it possible to easily provision test systems that are similar to production (infra-as-code/infra-as-model) Notes

Slide 68

Slide 68 text

Increase the autonomy between the development teams Step 3.2: Strategic solution

Slide 69

Slide 69 text

dev + unit test Orchestrated releases Application 1 Application 2 Application 50 … Organization XYZ - Release 2014.1 dev + unit test system test UI test dev + unit test API test integration test acceptance test release orchestration Development Integration Acceptance Prod autonomy Environment

Slide 70

Slide 70 text

The tactical solution was to streamline and automate the existing communication flows of release orchestration in order to have a better visibility on all the dependencies that exist between the different teams. But if one team fails to keep its commitment to the plan it will still cause a chain reaction in the other teams. The strategic solution is to get rid of this orchestration, to move towards autonomy. Notes

Slide 71

Slide 71 text

Feature C Wait time Application 1 Feature A Feature X Feature Y Feature Z multi-app feature single-app feature Orchestrated Release wait time

Slide 72

Slide 72 text

Orchestrated releases can also cause considerable wait time causing a missed opportunity to bring value quickly the the end users. Notes

Slide 73

Slide 73 text

Single-application releases Application 1 Application 2 Application 50 … Feature A Feature B Feature C Organization XYZ - Release 2014.1 X X X X X X

Slide 74

Slide 74 text

Single-application releases Application 1 Feature A Feature C App 1 - Release 2014.1.3 Requires backward compatibility Application 2 Application 50 … Feature A Feature B Feature C Organization XYZ - Release 2014.1 X X X X X X

Slide 75

Slide 75 text

We can move away from company-wide orchestrated releases by making our application backward compatible with the other applications Notes

Slide 76

Slide 76 text

Backward compatibility App 1 App 2 App 3 App 4

Slide 77

Slide 77 text

Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1

Slide 78

Slide 78 text

Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1

Slide 79

Slide 79 text

Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1

Slide 80

Slide 80 text

Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1

Slide 81

Slide 81 text

Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1 Breaking change

Slide 82

Slide 82 text

Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1 2 Breaking change

Slide 83

Slide 83 text

Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1 2 Breaking change

Slide 84

Slide 84 text

Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1 2 3 Breaking change

Slide 85

Slide 85 text

Backward compatibility App 1 App 2 App 3 App 4 Modification e.g. change fName to firstName 1 2 3 Breaking change

Slide 86

Slide 86 text

Backward compatibility App 2 App 3 App 4 App 1

Slide 87

Slide 87 text

Backward compatibility App 2 App 3 App 4 Addition e.g. add firstName 1 App 1 Deprecation e.g. keep fName expand

Slide 88

Slide 88 text

Backward compatibility App 2 App 3 App 4 Addition e.g. add firstName 1 App 1 Backward compatible Deprecation e.g. keep fName expand

Slide 89

Slide 89 text

Backward compatibility App 2 App 3 App 4 Addition e.g. add firstName 1 2 App 1 Backward compatible Deprecation e.g. keep fName expand

Slide 90

Slide 90 text

Backward compatibility App 2 App 3 App 4 Addition e.g. add firstName 1 2 3 App 1 Backward compatible Deprecation e.g. keep fName expand

Slide 91

Slide 91 text

Backward compatibility App 2 App 3 App 4 Addition e.g. add firstName 1 2 3 App 1 Backward compatible 4 expand contract

Slide 92

Slide 92 text

Backward compatibility follows the expand/contract principle (add/deprecate) which allows for a grace period for the depending apps to upgrade As soon as we support backward compatibility we become independent from the organisation- wide release: whatever delays occur there we are not impacted With this approach there is still quite some orchestration happening (4 steps), but the difference is that the decision making power has moved down from the organization to the application. Cf our scientific section we have changed the coordination mechanism from direct supervision (central orchestration) to mutual adjustment between the autonomous applications. They only have to take into account one dependency: make sure that we deliver our multi-app feature BEFORE the organization-wide release date. Supporting backward compatibility requires applications to have a well-structured/modular code base and a decoupled architecture There are several scales on which to apply the backward compatibility (two apps side by side, a v2 path in a rest resource, adding a field in It also requires that the depending apps are released frequently, to avoid the need to support too many deprecated features in parallel Notes

Slide 93

Slide 93 text

Feature C Reduced wait time Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z Orchestrated Release App Release wait time

Slide 94

Slide 94 text

Another advantage of becoming independent from the company-wide release is that we can start releasing on our own tempo. As a result the wait time decreases and the feedback cycle increases so the users will get added value sooner. Notes

Slide 95

Slide 95 text

Increase the autonomy between the components of the application Step 3.2.b: Unicorn solution

Slide 96

Slide 96 text

Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time

Slide 97

Slide 97 text

Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time

Slide 98

Slide 98 text

Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time

Slide 99

Slide 99 text

Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time

Slide 100

Slide 100 text

Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time

Slide 101

Slide 101 text

Feature C Increase the feedback cycle Application 1 Feature A Feature X Feature Y multi-app feature single-app feature Feature Z feature flag wait time

Slide 102

Slide 102 text

If we want an even faster feedback cycle we can use feature flags for not-yet-finished features to reduce the wait time to zero for each feature. This causes an increase in the number of releases and consequently a faster feedback cycle, which is a good thing. It may also have a negative side effect if the application requires down time during the deployment because of a temporary inconsistency period that may arise when the components of the application are deployed, e.g. the new version of the application code requires the new version of the db schema and vice versa. This is a similar problem as with the company-wide release, only on a lower level: the level of the application. The solution is also similar: we have to make the components backward compatible (and in general more autonomous) to arrive at zero-downtime deployments Because the communications on this component level is much more granular supporting backward compatibility on this level is much more difficult. Notes

Slide 103

Slide 103 text

Database fName Application read fName write fName Backward compatibility

Slide 104

Slide 104 text

Database fName fName firstName Add e.g. add firstName 1 Deprecate e.g. keep fName expand Application read fName write fName Backward compatibility

Slide 105

Slide 105 text

Database fName fName firstName Add e.g. add firstName 1 Deprecate e.g. keep fName expand Application read fName write fName Backward compatibility 2 write firstName Duplicate write

Slide 106

Slide 106 text

Database fName fName firstName Add e.g. add firstName 1 Deprecate e.g. keep fName expand Application read fName write fName Backward compatibility 2 write firstName Migrate data 3 Duplicate write

Slide 107

Slide 107 text

Database fName fName firstName Add e.g. add firstName 1 Deprecate e.g. keep fName expand Application read fName write fName Backward compatibility 2 write firstName Migrate data 3 read firstName 4 Duplicate write Modify read Deduplicate write

Slide 108

Slide 108 text

Database firstName Add e.g. add firstName 1 expand Application read fName write fName Backward compatibility 2 write firstName Migrate data 3 read firstName 4 contract 5 Duplicate write Modify read Deduplicate write

Slide 109

Slide 109 text

Making database changes backward compatible, e.g. changing the name of a field, requires a lot of orchestration Notes

Slide 110

Slide 110 text

Unicorns

Slide 111

Slide 111 text

This solution is typically applied at some of the most successful digital companies that exist because they apply extremely fast feedback cycles (the reason of their success) and cannot afford any downtime. Having a centralized orchestrator is not interesting for them because they work in huge infrastructures with high scalability needs and high numbers of applications and components that continuously need changing. ! ! Fully cross-functional and self-empowered teams Huge infra with high scalability needs Fully automated release pipeline (deployments and testing) No-downtime deployments Fully decoupled architectures with autonomous applications and components (e.g. app code and db changes, communication with load balancers and monitoring systems) Backward compatibility, feature flagging, dark launching Notes

Slide 112

Slide 112 text

Traditional separation Application Packages Packages Packages Packages Packages Config files Packages Packages User acounts Packages Packages Dirs Files Packages Packages Services Packages Packages Cron schedules Server Model Packages … … Configs … … { Development Operations maintain consult sync

Slide 113

Slide 113 text

Modern separation Application Packages Packages Packages Packages Packages Config files Packages Packages User acounts Packages Packages Dirs Files Packages Packages Services Packages Packages Cron schedules Server Model Application … … Packages … … Configs … … sync Creative Automated Devs & infra specialists maintain

Slide 114

Slide 114 text

We have to move from the traditional separation between dev and ops to a separation between creative and automated/standardized Functional and operational evolutions of the app are done in the Creative side Repetitive work (i.e. traditional day-to-day ops work) is automated in the Automated side Include the application in the model: - one declarative model for the application with the system configurations it needs - one tool for app deployment and CM (makes sense because we now have only one team anyway) - drift management for the application - orchestration needs can be covered by the components through mutual coordination using the information from the model (or by tooling if it gets too complex: it must calculate the exact deployment process from the model/ desired state and the current state) - provisioning and upgrading becomes possible for less-technical people Notes

Slide 115

Slide 115 text

Summary: Orchestration problems Enhance the existing flows Increase autonomy

Slide 116

Slide 116 text

Niek Bartholomeus @niekbartho Contact me! http://niek.bartholomeus.be niek.bartholomeus@gmail.com