Modern Enterprise Architecture: Architecting for Outcomes

A presentation given at Agile Meets Architecture 2023 on a modern approach to Enterprise Architecture following the paradigm shift in software development driven by, amongst other things, the Agile Manifesto, DevOps, cloud, and the flip from project-to-product.

Transcript

1. Modern Enterprise Architecture: Architecting for Outcomes Simon Rohrer
2. My journey Developer Enterprise architect Agile transformation lead Head of tech arch & WoW Agile adoption lead Ways of working lead
3. Modern Enterprise Architecture Teams-of- Teams-of- Teams: > 150 Employees Interact with Your Customers Digitally Change is a Constant Heterogeneous Environment: Old/New, Big/Small, Slow/Fast, Monolith/Distributed Vendors/Internal Systems of Systems of Systems
4. A B C D E Aligning Value, People and Technology Architecting for Outcomes: Better Value, Sooner, Safer, Happier Governing Continuously: Conversational and Automated Practicing DevOps at Enterprise Scale Curating the Evolution of Enterprise Architecture and Organisation • Conway’s law ++ and reverse Conway manoeuvre • Removing complexity and increasing autonomy • Enterprise architecture as politics and diplomacy • Outcomes of quality, fl ow, safety to balance pure “value” • Happier customers, colleagues, citizens & climate • Continuous improvement - be the best at being better • A stream of continuous architecture decisions • Moving decisions down into the right level of the org • Automating common top-level system policies • Learning by observing: intention & re fl ection • Humility in architecture: all decisions are contingent • Fitness functions for socio-technical architecture at scale • Evolving fast and slow Modern Enterprise Architecture
5. Growth vs Fixed de fi nitions
6. Uncovering better ways of developing software by doing it and helping others do it Early and continuous delivery
 of valuable software Continuous attention to technical excellence and good design “Agile” Processes & tools Sprints, standups, user stories, story points Scrum Fixed mindset Growth mindset Frameworks & certi fi cations & coaches Individuals and interactions over processes and tools
7. You build it, you run it (/ you deploy it / you test it / you design it / etc) Culture, Automation, Lean, Metrics, Sharing The 3 ways: Systems Thinking, Amplify Feedback Loops, Culture of Continuous Experimentation & Learning Business = Dev, Customer = Ops “DevOps” A DevOps department / team / engineer Kubernetes, cloud, automating release pipelines CI/CD Fixed mindset Growth mindset One more hando ff in tech: Dev -> DevOps -> Ops/SRE
8. Where are we? How did we get here?
9. 1990s 1995 1999 1994 1999 1994
10. Software ate the world
11. A paradigm shift happened in software development Plan • Requirements • High level design months Analysis & design team Code • Detailed design • Development months Development team Test months • Test plan • Test execution Test team Release • Change request • Deployment weeks Release team Operate • Incidents • Problem fixes forever Operations team Development/ operation team Days or hours
12. Not everyone noticed
13. Complexity in the existing (“legacy”) system landscape started to dominate the pace of change Business service Technical service Database
14. 1. Autonomy 2. Explicit boundaries 3. Partitioning of functionality 4. Dependencies de fi ned by functionality 5. Asynchronicity [by default] 6. Partitioning of data 7. No cross-fortress transactions 8. Single point security 9. Inside trust 10. Keep it simple Patterns were documented to manage complexity but not everyone used them. They still don’t. Autonomous business capability 2008
15. Our problems are similar but different to those in the 1990s The enterprise software landscape is so complex we need to use project management techniques to coordinate multiple teams for simple features Civil engineering processes are the best way to deliver any software change 1990s 2020s Plan • Requirements • High level design months Analysis & design team Code • Detailed design • Development months Development team Test months • Test plan • Test execution Test team Release • Change request • Deployment weeks Release team Operate • Incidents • Problem fixes forever Operations team
16. An organization that lacks a viable program in enterprise architecture will pay a severe cost in IT complexity. Complexity is the enemy. Enterprise Architecture is the solution. The only solution. Roger Sessions, 2008 https://web.archive.org/web/20211021220626/https://simplearchitectures.blogspot.com/2008/11/job-of-enterprise-architect.html
17. Help the organisation view itself as a complex sociotechnical system(-of- systems-of-systems) & reduce that complexity over time based on outcomes and learning Enterprise architecture Tell the organization what to do Fixed mindset Growth mindset
18. A Aligning Value, People and Technology
19. Organisation architecture System architecture Drives Conway’s law (very roughly)
20. If the architecture of the system and the architecture of the organization are at odds, the architecture of the organization wins. Ruth Malan, 2008 https://web.archive.org/web/20221205155201/https://ruthmalan.com/traceinthesand/conwayslaw.htm
21. BUT!
22. Where all hell really breaks loose is when management decides to reorganise things. … if you try to change the organisation, the software won’t let it happen. Allan Kelly, 2018 https://www.youtube.com/watch?v=Cu0AU8vw3xw
23. Organisation architecture System architecture Drives System architecture Organisation architecture Constrains
24. ???
25. We shape our buildings; thereafter they shape us Winston Churchill, 1943
26. We shape our architecture and then the architecture shapes us Gene Kim, 2022 https://web.archive.org/web/20231015100627/https://twitter.com/lbmkrishna/status/1547447927567953920?s=20
27. Organisation architecture System architecture Constrains Constrains
28. System architects (who we call architects) and business/organization architects (who we call managers) should not work as if one has no impact on the other. Ruth Malan, 2008 https://web.archive.org/web/20221205155201/https://ruthmalan.com/traceinthesand/conwayslaw.htm
29. Organization architects (“managers”) System architects (“architects”) Organisation architecture System architecture Constrains Constrains
31. Business architecture (“value streams”) Organisation architecture Drives Organization architects (“managers”)
32. Organisation architecture System architecture Constrains Constrains Business architecture (“value streams”) Constrains Constrains Constrains Constrains Organization architects (“managers”) System architects (“architects”)
33. Value Organization Technology & processes People, teams, teams-of-teams, departments Business services & business processes, IT services & IT components OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Work Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents Aligning all 3 for sustainable fl ow
34. Level 0: an ideal 2- pizza stream- aligned team Value 2 pizza team - <?20 people Software that fits inside the team’s head Full stack, full lifecycle, full burrito, T-shaped people - YBIYRI Self-organizing DDD Eventually consistent Independently deployable & testable “Monolith vs Microservice is missing the point” Emergent design & evolutionary* architecture OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents Value 2 pizza team - <?20 people Software that fits inside the team’s head Full stack, full lifecycle, full burrito, T-shaped people - YBIYRI Self-organizing DDD Eventually consistent Independently deployable & testable “Monolith vs Microservice is missing the point” Emergent design & evolutionary* architecture OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents Value 2 pizza team - <?20 people Software that fits inside the team’s head Full stack, full lifecycle, full burrito, T-shaped people - YBIYRI Self-organizing DDD Eventually consistent Independently deployable & testable “Monolith vs Microservice is missing the point” Emergent design & evolutionary* architecture OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents Value 2 pizza team - <?20 people Software that fits inside the team’s head Full stack, full lifecycle, full burrito, T-shaped people - YBIYRI Self-organizing DDD Eventually consistent Independently deployable & testable “Monolith vs Microservice is missing the point” Emergent design & evolutionary* architecture OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents
37. Value pizza team <?20 people Software that fits inside the team’s head Full stack, full ecycle, full burrito, -shaped people - YBIYRI Self-organizing DDD Eventually consistent Independently deployable & testable “Monolith vs Microservice is missing the point” Emergent design & evolutionary* architecture OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents
38. Level 1: an ideal team- of-teams Value Team-of-teams Dunbar number - <150 System-of-systems & domain that fits inside the team’s head “Tribe” Nested value Self-organizing? Nested DDD Shared data model? Eventually consistent Each component independently deployable & testable OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents Value Team-of-teams Dunbar number - <150 System-of-systems & domain that fits inside the team’s head “Tribe” Nested value Self-organizing? Nested DDD Shared data model? Eventually consistent Each component independently deployable & testable OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents Value Team-of-teams Dunbar number - <150 System-of-systems & domain that fits inside the team’s head “Tribe” Nested value Self-organizing? Nested DDD Shared data model? Eventually consistent Each component independently deployable & testable OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents Value Team-of-teams Dunbar number - <150 System-of-systems & domain that fits inside the team’s head “Tribe” Nested value Self-organizing? Nested DDD Shared data model? Eventually consistent Each component independently deployable & testable OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents
39. Level 2: an ideal business line OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Value Team-of-team-of-teams 1000–2000 System-of-system-of-systems Does this fit in anyone’s head? Department / business line Nested value Self-organizing??? Further nested DDD Minimally shared data model - just keys? Eventually consistent between subdomains Key paths and failure modes should be known OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Value Team-of-team-of-teams 1000–2000 System-of-system-of-systems Does this fit in anyone’s head? Department / business line Nested value Self-organizing??? Further nested DDD Minimally shared data model - just keys? Eventually consistent between subdomains Key paths and failure modes should be known OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Value Team-of-team-of-teams 1000–2000 System-of-system-of-systems Does this fit in anyone’s head? Department / business line Nested value Self-organizing??? Further nested DDD Minimally shared data model - just keys? Eventually consistent between subdomains Key paths and failure modes should be known OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Teams-of- Teams-of- Teams: > 150 Employees OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Value Team-of-team-of-teams 1000–2000 System-of-system-of-systems Does this fit in anyone’s head? Department / business line Nested value Self-organizing??? Further nested DDD Minimally shared data model - just keys? Eventually consistent between subdomains Key paths and failure modes should be known OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Systems of Systems of Systems
41. The best architectures, requirements, and designs emerge from self-organizing teams. agilemanifesto.org/principles.html, 2001
42. Value zza team 20 people Software that fits inside the team’s head stack, full le, full burrito, ped people - YBIYRI -organizing DDD Eventually consistent Independently deployable & testable “Monolith vs Microservice is missing the point” Emergent design & evolutionary* architecture OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents Value Team-of-teams Dunbar number - <150 System-of-systems & domain that fits inside the team’s head “Tribe” Nested value Self-organizing? Nested DDD Shared data model? Eventually consistent Each component independently deployable & testable OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Outcomes, hypotheses, bets User stories, tasks, changes, problems, incidents OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Value Team-of-team-of-teams 1000–2000 System-of-sys Does this fit in Department / business line Nested value Self-organizing??? Further n Minimally shared da Eventually consistent Key paths and failure m OKRs looking forward (Dev) KPIs & OLIs right now & looking back (Ops) Teams-of- Teams-of- Teams: > 150 Employees Self-organizing teams?
43. B Architecting for Outcomes: Better Value, Sooner, Safer, Happier
44. Standardisation Consistency Predictive planning Cost reduction Antipattern: traditional EA outcomes These are not the outcomes you’re looking for
45. Cloud Kubernetes Microservices Antipattern: “newer” EA outcomes These are not the outcomes you’re looking for
46. Customers Colleagues Citizens Climate Security Privacy Minimum viable compliance Quality Flow Pattern: align to business agility
47. We shape our architecture and then the architecture shapes us Gene Kim As the systems we build become larger, the coordination increases… …it can grow so so large that all our time and energy is spent coordinating - it is at the expense of our value creating activities https://web.archive.org/web/20231015100627/https://twitter.com/lbmkrishna/status/1547447927567953920?s=20
48. “Hello world” with tightly coupled complex architecture ¥$€ SOLUTION ARCHITECTS TESTING TEAM UI API LAYER GREETINGS SERVICE PLANET SERVICE “Hello world!” “!” /api Hello World T_HW “Hello wrld!”
50. Release trains are an iterative and incremental lifecycle, but they are not agile. https://www.jrothman.com/articles/2011/01/not-ready-for-agile-start-your-journey-with-release-trains/ Johanna Rothman, 2011
51. Autonomous team with their own valuable business capability Refactored Restructured organization and architecture ¥$€ EARTH GREETER MICROFRONTEND EARTH GREETER MICROSERVICE DB EARTH GREETER MICROSERVICE “Hello world!” ] “Hello world!”
53. C Governing Continuously: Conversational and Automated
54. Creating and reducing complexity may sound like perfect opposites. But in fact fundamental asymmetries exist between the two. https://hbr.org/2020/01/taming-complexity
55. Architecture governance to the rescue?
56. Antipattern: Architecture Review Boards
57. Initiate Plan Execute Close Analysis Design Build/Test Release Architecture board Review Architecture board Review How we used to govern architecture A project: waterfall / wagile / water-scrum-fall / “hybrid”
58. How does this work with modern delivery? Initiate Plan Execute Close Analysis Test/design/build Release Release Release Release Release Release Release Architecture board Review An agile or lean project
59. How does this work with modern delivery? Architecture board A product
60. Pattern: Black box architecture governance
61. Govern realised architectures, not (just) designs on paper “I think I need an new service” “I think my service needs to access data or functionality from another service” EA approve new services, new namespaces in Kubernetes (etc.) EA approve: • Service-to-service IAM • Firewalls “I want an exception to automated governance - e.g. synchronous request/response” EA approve: • Exceptions • Tech debt Enterprise Architecture as GitOps
62. Pattern: Continuous conversational “governance”
63. Continuous conversational governance Initiate Plan Execute Close Analysis Test / Design / Build Release Release Release Release Release Release Release Architecture C.o.P. Continuous conversation Agile/lean/DevOps/Continuous Delivery driven project
64. Continuous conversational governance A single product Architecture C.o.P. Continuous conversation
65. Continuous conversational governance across value streams Value stream B Value stream C Value stream A Value stream E Value stream F Value stream D Value stream M Value stream N Business unit Architecture C.o.P. Architecture centre of excellence
66. D Practicing DevOps at Enterprise Scale
67. Continuous conversational governance Architecture C.o.P. Architecture centre of excellence Enterprise scope Longer term timeframe Service management C.o.E.
68. Intention Reflection Working code deployed as running system Model storming Emergent design Discoverability & observability tooling Test-driven design Infrastructure-as- code Architecture decision records Continuous refactoring Whiteboard sessions Continuous conversational governance (after Ruth Malan)
70. E Curating the Evolution of Enterprise Architecture and Organisation
71. Continuous attention to technical excellence and good design enhances agility.
72. Evolutionary architecture fitness functions at enterprise level (“macro architecture”) •Number of 2-pizza teams required to implement a typical business feature - sociotechnical coupling & cohesion •Asynchronous connections ÷ synchronous connections
73. Theory #1 : gradual evolution Time Evolution
74. Theory #2 : punctuated equilibria Time Evolution
75. Reconciled: punctuated gradualism Continuous attention to complexity debt pay down and continuous improvement. Continuous funding. A step change: a business case explicitly to rewrite (or buy, or outsource, or write something new). Standalone funding. Time Evolution
76. Strangler fi g pattern
78. A B C D E Aligning Value, People and Technology Architecting for Outcomes: Better Value, Sooner, Safer, Happier Governing Continuously: Conversational and Automated Practicing DevOps at Enterprise Scale Curating the Evolution of Enterprise Architecture and Organisation • Conway’s law ++ and reverse Conway manoeuvre • Removing complexity and increasing autonomy • Enterprise architecture as politics and diplomacy • Outcomes of quality, fl ow, safety to balance pure “value” • Happier customers, colleagues, citizens & climate • Continuous improvement - be the best at being better • A stream of continuous architecture decisions • Moving decisions down into the right level of the org • Automating common top-level system policies • Learning by observing: intention & re fl ection • Humility in architecture: all decisions are contingent • Fitness functions for socio-technical architecture at scale • Evolving fast and slow Modern Enterprise Architecture
79. Thank you! @sirohrer