[KanDDDinsky 24] Navigating sociotechnical complexity with DDD & friends

Our work with software is a system of systems, comprising technical systems (software, hardware...) and social systems (teams, knowledge networks...). For decades, our primary focus has been on the outcome and output of technical systems. Social capacity (Agile, Kanban, process models) is often perceived as a support for producing software, and often practiced as if human organizations were a software factory, manageable by divide-and-conquer, cascading goals and aggregating results.

Software professionals and Agile practitioners often feel powerless when things get too complex and too political. We are frustrated that well-meaning attempts at fixing human systems with processes, governance structure, architecture and scaled agile frameworks don't seem to work.

This talk aims at cultivating a deeper understanding of sociotechnical complexity among software engineers, architects, Agile practitioners, product professionals and leaders at all levels in large organizations. You will be introduced to a multi-dimensional sociotechnical design toolbox with practices from Domain-Driven Design (DDD), sociotechnical architecture, Complexity Theory, Systems Thinking and Constraints Framework.

You will hear concrete examples from the speaker's personal journey from being a developer, then a software architect then sociotechnical architect and a change practitioner. You will also be introduced to a number of thinking, modeling and conversational tools that you can experiment with in your own context.

Join me to reflect together on, how DDD and adjacent practices can be leveraged as powerful enabling constraints to help us see the system, share the system, and change the system in a reality of vast sociotechnical complexity and constant change.

Transcript

1. Navigating sociotechnical complexity with DDD & friends Xin Yao, KanDDDinsky 2024, Berlin
2. DDD & architecture consultancy DDD Output + Reflective conversations + Collective change capacity + [A little about me] Change Smuggler, through DDD and architecture Xin Yao @settling_mud xin.yao.dk@gmail.com @xinyao@mathstodon.xyz /in/xinxin/ delay + delay Independent consultant delay Software quality & changeability + delay + Technical improvement Social improvement DDD +
3. Sociotechnical complexity Sociotechnical patterns Today's path Navigating sociotechnical complexity with DDD & friends Sociotechnical change practice
4. Sociotechnical complexity Sociotechnical patterns Today's path Navigating sociotechnical complexity with DDD & friends Sociotechnical change practice
5. A complex reality permeated by software Software industry matures Every business is a software business Aging companies with aging software Compounding social and technical complexity
6. Decouple and Connect the paradox in large-​ scale software development
7. Who work and create value together? Who else do we need to align understanding with? DDD DDD was born sociotechnical Credit: Paul Rayner, Eric Evans What software are we building? Why are we building it? How do we build and connect software - for long-​ term changeability? Strategic Design Tactical design Visual Collaborative Modeling (models) (models) Ubiquitous (attention to) Language ... to decouple and connect business complexity technical complexity social complexity
8. Many shades of complexity in software work How do we talk about things being complex in different ways?
9. Complex is not the same as Complicated A B C A B C D System properties predictable The whole is greater than the sum of its parts Knowable through analysis ("Sense, Analyze") Many moving parts sum up to the whole Plans & Policies ("Respond") Knowable through interaction ("Probe") System properties emergent, unpredictable Experiments & Patterns ("Sense, Respond") Repeatable, Consistent ("Scale best practice") Unrepeatable ("Hindsight does not lead to foresight") A B C D E x emergent new element disappeared relation emergent relation Cause -> Effect A B C Complicated Complex Emergence unintended change in relation Credit: Cynefin, Dave Snowden new element new relation unintended change in element
10. A B C A B C D System properties predictable The whole is greater than the sum of its parts Knowable through analysis ("Sense, Analyze") Many moving parts sum up to the whole Plans & Policies ("Respond") Knowable through interaction ("Probe") System properties emergent, unpredictable Experiments & Patterns ("Sense, Respond") Repeatable, Consistent ("Scale best practice") Unrepeatable ("Hindsight does not lead to foresight") A B C D E x emergent new element disappeared relation emergent relation Cause -> Effect A B C Complicated Complex Emergence unintended change in relation Credit: Cynefin, Dave Snowden new element new relation unintended change in element Ferrari vs. Brazilian rain forest
11. Are organizations working with software more like complex or complicated systems?
12. Are organizations working with software often managed as complex or complicated systems?
13. Is a software system more of a complex or complicated system?
14. Work is a social system with technical subsystems as parts Social systems (Complex) Technical systems (Complicated/Liminal*) [software] [organization] Business environment (Complex) *liminal: in between complex and complicated (credit: Dave Snowden) [Open Sociotechnical System]
15. Fluency Stage Technical practice vs. social/sociotechnical practice Evolution Genesis Custom Product Commodity (+rental) (+utility) 1 2 3 4 Technical innovation Mature technical practice Everything evolves with time Social practice? Sociotech practice? ? software design Novel Emerging Good Best
16. If the ancient Greeks could come back now and walk among us, they would not understand much of our technology and our science. It would be very foreign. But they would be quite at home in our social problems - wars, politics, economics, various kinds of difficulties. ~ Jay Forrester We haven't evolved much in our understanding of social systems
17. Scientific management A most impactful industrial age legacy 1856-1915 Frederick Taylor (1856-1915) In the past the man has been first; in the future the system must be first. Management is a true science, with laws as exact and as clearly defined as the fundamental principles of engineering.
18. Professional planners provide a grand design of work Material, capital & people with skills as production factors Design of work Production factor
19. Assign skills to work posts to optimize labor productivity and economic efficiency
20. Scientific management has been rebranded a lot of times, and the most modern version is ...
21. The Inverse Conway Maneuver Target architecture boundaries Team boundaries The new black in organization design Sociotechnical mirroring
22. (Example: governance process in a corporate change initiative) Our default social intervention mode is analytical divide-​ and-​ conquer
23. Our management language is full of engineering metaphors Fix problem Drive change Control process Align thinking Scale practice Measure performance Run a company Produce outcome
24. Most of us acknowledge that social systems are more complex, uncertain and unpredictable ... Complex is not the same as Complicated
25. The traits of complexity are anxiety inducing. Our human brain is hard-​ wired to fear the uncertain, unpredictable, uncontrollable, ambiguous. But we don't want to think of our work as a Brazilian rain forest a randomly evolving mess of emergent relations that keep changing
26. We would rather treat organizations as complicated systems - many moving parts and relations, but predictable, plannable, controllable. We are drawn to solutions that promise to divide-​ and-​ conquer human communication as decomposable software APIs. We wish our organizations were complicated Ferraris
27. Planning and process - a collective defense against our existential anxiety about uncertainty No one loves big, hairy plans. But, reality gets more complex & uncertain - inducing stress We need plans to break down big problems into manageable parts We cannot step into the future without a plan ... and processes to align the many moving parts, people and tasks Plans & processes make us feel safer, in control, on the same page Credit: Steve Hearsum
28. We identify with our "role" and feel alienated playing our "part" at the same time The planning roles The executing roles (Example: governance process in a corporate change initiative)
29. Downgrade social complexity to technical complication? Large-​ scale Agile practice Sociotechnical mirroring Agile frameworks Inverse Conway Spotify model Learned helplessness Skepticism Unknowability A big divide Pessimism Dehumanizing Lack of agency Reorg Transformation initiative
30. Sociotechnical complexity Sociotechnical patterns Today's path Navigating sociotechnical complexity with DDD & friends Sociotechnical change practice
31. Collaborative Modeling (COMO) [Sociotechnical pattern from DDD] See the forest and the trees
32. We cannot find optimal software boundaries with DDD Confession of a Change Smuggler in a complex domain with ever-​ changing business logic just
33. Everybody wants this! Bounded Context Bounded Context Bounded Context Bounded Context Bounded Context Bounded Context
34. In reality, you often get this
35. Premature convergence Bounded Context Bounded Context Bounded Context Bounded Context Bounded Context Bounded Context Bounded Context Bounded Context Bounded Context Bounded Context Bounded Context
36. In the context of architecture modernization (aka. splitting the monolith)
37. Component A Component B DoThis (a, b, c, d) Process 1 Service A Process 1 Process 2 Service B DoThis (a, b, c, d) Monolith (Micro)Services Decoupling by microservices?
38. Component A Component B DoThis (a, b, c, d) Process 1 Service A Process 1 Process 2 Service B ThisRequested (a, b, c, d) ThisCompleted (a, b, c, d) Monolith Event-​ Driven Decoupling by messaging?
39. Component A Component B DoThis (a, b, c, d) Process 1 Next tech trend du jour ? Cloud repatriation - back to modular monolith? Monolith MFE? BFF? Actor?
40. It's so easy to go for superficial boundaries and drop the deep conversations about logical coupling ... and the underlying social boundaries and power distribution COMO workshops for quality conversations not fixed outcomes
41. Let's edit out the uncomfortable elements [Sociotechnical pattern inspired by DDD] Make the implicit explicit [technical design pattern] Make the undiscussable discussable [social design pattern] Collaborative Modeling --> Reflective Conversation Shame Guilt Anxiety Fear Resentment Blame
42. Component A Component B DoThis (a, b, c, d) Process 1 Component A Process 1 Process 2 Component B DoThis (a, b, c, d) Monolith (Micro)Services Social relational muscles to find software boundaries Have we discussed the social and technical implications of passing data around? that give us the true freedom of change
43. Which conversations are we missing? The 3 P's Conversation about Purpose What are we committed to creating together, rather than making less bad? Conversation about Paradoxes How can we call out the organizational politics and constraints, and still make difficult decisions? Conversation about Power How can we name our discomfort and feel powerful at the same time? The social, the relational, the contextual
44. Sociotechnical complexity Sociotechnical patterns Today's path Navigating sociotechnical complexity with DDD & friends Sociotechnical change practice
45. Sometimes the social+technical is just too big of an "official" scope of work DDD "process consultant" "process architect"
46. DDD Process DDD as a process to start & anchor conversations Methods Principles, Patterns, Practices Facilitate change & communication (design discipline) Domain-​ driven discovery is sometimes employed as a process model in change initiatives
47. Example: Architecture modernization work at Milestone Systems
48. Smuggle social sense-​ making in technical design processes (e.g. collaborative modeling sessions) [Sociotechnical change pattern]
49. * Action learning DDD concepts while building a domain language * Hands-​ on Strategic DDD in subdomain analysis exercise. * Hands-​ on Tactical DDD in real use cases - aggregate extraction, domain models, state machines and ports & adapters ... DDD Learning Outcome * Sense making (bridge pent-​ house and engine room) * Visual negotiation * Collaborative moves * My/our point of influence (agency) Social Outcome * Convergence on subdomain boundaries in the M&MS area * (Almost) Convergence on teams' subdomain ownership - as modernization baseline * Identified "don't belong here" subdomains Material Outcome Spread awareness Experience reports as lightning talks at All-​ Hands Example: Architecture modernization work at Milestone Systems
50. Safe-​ to-​ fail change probes as Trojan Mice in tech-​ fueled social systems "Trojan Mice" https://whatsthepont.blog/2020/02/09/trojan-​mice-​in-900-​seconds/ [Sociotechnical change pattern]
51. Smuggle small Trojan Mice inside a big Trojan Horse Large change program Small social change experiments [Sociotechnical change pattern] The original model The new model
52. Real collaboration - we build on each other's moves [Sociotechnical change pattern] David Kantor: Four Player Model Oppose Observe Move Follow Challenge Surface differences Devil's advocate Detached Perspectives Reflect Analyze Add Appreciate Support Expand Implement Set direction Break ice Propose idea Initiate action
53. In this talk, we muddled through these sociotechnical topics Complexity appreciation complex vs. complicated embedded sociotechnical model technical & social practices reversibility of Conway's Law Sociotechnical patterns decouple & connect collaborative modeling logical coupling undiscussables reflective conversations purpose, paradoxes & power (3Ps) Sociotechnical change practice probe, sense, respond safe-​ to-​ fail social experiments change smuggling Trojan Mice mediocre vs. real collaboration
54. Distributed, domain-​ driven, and event-​ driven software and human systems A globally choreographed dance where small dancers orchestrate solo steps in their local workflows, and in concert with each other. Inspiration: Udi Dahan We know when to move and when to hold our poses, in service to the WHOLE system.
55. Thank you xin.yao.dk@gmail.com @xinyao@mathstodon.xyz /in/xinxin/ @settling_mud