Share
Scaling & Evolving Tech-driven Orgs Using sociotechnical Architecture & Systems Thinking
Product Thinking and DevOps are key to enable (Tech) Orgs increasing their ability to build better products at higher velocity. However, in order to continuously maximize the value exchange with the customer, we must consider all the socio-technical systems in the product development activities. All of these are continuously changing: customers, markets, organizations and teams evolve and products technical architecture will follow. Change is unavoidable, which means organizations must strive for operating models and systems that embrace it.
In this talk I will discuss several interesting traits to consider on the different (socio-technical) systems in order to maximize the org's ability to cope with such changes and evolution. For each trait I will share patterns of evolution and use cases, e.g.: evolving Data Science & ML operating models as function of org size and complexity; evolving Platform Teams from operations to product mindset; evolving technical leadership from pure informal networks to the addition of structural enabling teams, etc. These are patterns and use cases I observed over the last four years while working in different initiatives to "scale and evolve" bol.com.
Eduardo da Silva PRO
Video
Transcript
36 slides
-
@emgsilva esilva.net Scaling & Evolving Tech-driven Orgs Using sociotechnical Architecture
& Systems Thinking Eduardo da Silva (Principal Tech Lead @ bol.com) @emgsilva | [email protected] | esilva.net -
@emgsilva esilva.net @emgsilva What are important things to improve (scale,
evolve) our (Tech) Org (& Why)? How can we do it effectively? -
@emgsilva esilva.net @emgsilva ...the journey from the problem space of
the users to the solution space of the business. The goal of this journey is to reduce the gap between users and the business. -Naren Katakam credits: https://uxplanet.org/product-thinking-101-1d71a0784f60, Naren Katakam -
@emgsilva esilva.net @emgsilva Continuous Learning Loop Product-led Org (Service) Product
Product Thinking: customer-centric product discovery & delivery -
@emgsilva esilva.net @emgsilva "Our highest priority is to satisfy the
customer through early and continuous delivery of valuable software." -
@emgsilva esilva.net @emgsilva Product Thinking: customer-centric product discovery & delivery
=>End Goal: To Maximize the Value Exchange with Customer Value Exchange Value $$$ Product-led Org (Service) Product -
@emgsilva esilva.net @emgsilva Now… How can we “maximize the value
exchange with customer” (continuously + high velocity + ...) Value Exchange Value $$$ Product-led Org (Service) Product -
@emgsilva esilva.net @emgsilva Is DevOps or “BizDevOps” the answer? Stopping
with “silos” between “The Business” (Product) and “IT” (Tech) can definitely help improving things... ...but there are even more things to consider in this continuously changing equation (of maximizing the customer value exchange) >>> This is where we will focus today! <<< -
@emgsilva esilva.net @emgsilva LEt’s talk about Restaurants... ...some traits to
make “Great Restaurants” (products) -
@emgsilva esilva.net @emgsilva Trait 1: Your staff defines The Restaurant
cooking and experiences If you want an amazing French food experience you need to have French chefs and French wine experts; and they have to work together to come up with great french food experiences. Your teams shape your product -
@emgsilva esilva.net @emgsilva Trait 2: Chefs cannot cook, serve the
food and do the dishes at the same time... If you want to run your restaurant smoothly and have a strong team for a long time, you need to have enough people and skill to support the different tasks without having people “burning out” (or leaving) Your teams must have the right conditions to build the product for your customer -
@emgsilva esilva.net @emgsilva Trait 3: Kitchen + Serving + Wine
+ ... = Restaurant Your restaurant has different parts and people working on them (in parallel); however, you want to understand and optimize how it all comes together in order to maximize your “product” (value to customer) Your product is the combination of different value streams, owned by different teams in the organization but aligned to maximize value for customer -
@emgsilva esilva.net @emgsilva Trait 4: don’t hand recipes to your
chefs… Empower them to discover new recipes Do you want to achieve unique experiences (and value exchange) for your customers? Then enable your crew to "experiment & discover" how to maximize that! (continuously) Your teams discovers (with the customer) what are the best things to build in the product to maximize the value exchange (continuously) -
@emgsilva esilva.net @emgsilva Recapping: How can we “maximize the value
exchange with customer” (continuously + high velocity + ...) 1: Your teams shape your product 2: Your teams must have the right conditions to build the product for your customer 3: Your product is the combination of different value streams, owned by different teams in the organization but aligned to maximize value for customer 4: Your teams discovers (with the customer) what are the best things to build in the product to maximize the value exchange (continuously) -
@emgsilva esilva.net @emgsilva We must take an holistic view to
the “What” + “Who” + “How” systems… Isolated local optimizations will not do it! (Service) Product What enables maximize Value Exchange (Product) Value $$$ Who is doing this? (Teams) How are we doing it? (Tech Arch) -
@emgsilva esilva.net @emgsilva This is Systems Thinking... To manage a
system effectively, you might focus on the interactions of the parts rather than their behavior taken separately ...Understanding* proceeds from the whole to its parts, not from the parts to the whole as knowledge does. Russell Ackoff * Understanding => answering the “Why” question ...identify the containing whole. For example, to understand a car you identify the transportation system ... -
@emgsilva esilva.net @emgsilva to make a great “dish"... ...understand the
whole restaurant (system) context! -
@emgsilva esilva.net @emgsilva “You (should) never improve the quality of
the dishes if it does not improve the quality of the restaurant simultaneously” => Let the parts of the whole improve the quality of the whole! -
@emgsilva esilva.net @emgsilva (Whole) Systems Understanding = { Understanding “basic
Context” { Structure x dynamics } + Understanding of the “Emerging Context” { properties that rise from the interactions the structure and dynamics in the context } } reference: https://twitter.com/ruthmalan/status/1093294478641823744 -
@emgsilva esilva.net @emgsilva So, what about our world of building
Software Products? How can we use these ideas? -
@emgsilva esilva.net @emgsilva Definition: Sociotechnical architecture “Sociotechnical Architecture is about
taking an holistic co-design approach to technical and organizational systems, given the inherent impact they have on each other.” --Eduardo da Silva Source: Sociotechnical Architecture Definition thread “When we are architecting a software system, we must consider the impact on the teams in the organisation and vice-versa.” --Nick Tune -
@emgsilva esilva.net @emgsilva How to Approach Sociotechnical Architecture? while(true) {
// understand based on the whole & feedback loops data, then build hypothesis for direction of actions on parts (or new parts) hypothesis = synthesis(data); // action and analysis on the parts (collect feedback data to “learn”) data = actionAndAnalysis(hypothesis); } reference: Analysis, synthesis, systems thinking and the scientific method: Rediscovering the importance of open systems -
@emgsilva esilva.net @emgsilva Bol.com {prod/org/arch & DS} Evolution => Learning
from patterns of scaling & evolving, through a Sociotechnical Architecture & Systems thinking lens -
@emgsilva esilva.net @emgsilva * Active customers purchased at least once
at bol.com in the last 365 days. Active sellers offered at least one item via bol.com in the last 365 days or currently still offer items via bol.com -
@emgsilva esilva.net @emgsilva * Active customers purchased at least once
at bol.com in the last 365 days. Active sellers offered at least one item via bol.com in the last 365 days or currently still offer items via bol.com Monoliths Products Tech Arch Org (& Way of Work) Waterfall Agile Product Teams DevOps Teams Bol.com Sociotechnical Arch (prod/org/arch) evolution... Prod (Micro)services ~30 ~100 ~10 ~600 ~50 -
@emgsilva esilva.net @emgsilva Tech (arch) context Action on tech arch
Org context Action on org Data Science (DS) Evolution @ bol.com while(true) { // understand based on the whole & feedback loops data, then build hypothesis for direction of actions on parts (or new parts) hypothesis = synthesis(data); // action and analysis on the parts (collect feedback data to “learn”) data = actionAndAnalysis(hypothesis); } Sociotech Architecture State (Parts) Problem (Feedback loop & Observation) (Whole) Synthesis & Hypothesis Generation (Whole) Tech Org Prod Product context Problem (observed from the “Whole”) Synthesis (understanding) and definition of hypothesis Action on product time Actions on Parts year -
@emgsilva esilva.net @emgsilva ~20 DevOps Teams (mostly Engineering & Product),
No DS present on teams Bring “DS Pioneers” to Teams ~Service-Oriented Architecture (groups of teams own “big services”) Start developing DS Tech (e.g.: Create Hadoop Cluster) 1: Data Science evolution / “DS Projects” adoption Some context: • Data Science has very particular ways of working (i.e.: embedding it on a team is not trivial) • “DS Projects” means “DS Pioneers” come to a team (who has a certain scope), implement DS project and go to next “gig” Sociotech Architecture State (Parts) Problem (Feedback loop & Observation) (Whole) Synthesis & Hypothesis Generation (Whole) Tech Org Prod Online shop, moving to become also Retailer Platform (growing a lot!) Need DS to improve product relevant + address complexity, but no DS people on teams/org yet * Get DS “pioneers” in * Start DS Projects in strategic topics * Start building “DS Tech” DS projects on strategic parts of the product (e.g.: Recommendations; sales forecasts) time Actions on Parts ~2012 -
@emgsilva esilva.net @emgsilva ~70 DevOps Teams (mostly Engineering & Product,
~10 Data Scientists) ~10 DS Teams (the teams who had strong DS component or potential); explicit introduction of DS Leadership team ~20 DevOps Teams (mostly Engineering & Product), No DS present on teams ~ Microservices Architecture; Hadoop cluster for DS “jobs” Explore how to bring DS tech support to next level (including leverage GCP components ~Service-Oriented Architecture (groups of teams own “big services”) 1: Data Science evolution / From “DS Projects” to “DS Teams”* Sociotech Architecture State (Parts) Problem (Feedback loop & Observation) (Whole) Synthesis & Hypothesis Generation (Whole) Tech Org Prod Big focus on becoming a “Retailer Platform”; own shop becomes second prio. Growing even more! “DS projects” don’t scale or match philosophy for DevOps Teams (You Build It, You Run You It, You Love It) + low maturity of Data Science in org * create “DS Teams & Department”, working on high potential DS products * Mature DS Discipline DS is part of products with high potential time Actions on Parts ~2012 Online shop, moving to become also Retailer Platform (growing a lot!) Need DS to improve product relevant + address complexity, but no DS people on teams ~2017 * some are “complicated sub-systems” serving multiple “stream aligned teams”... but not all of them fit that team topology -
@emgsilva esilva.net @emgsilva Actions on Parts ~ Microservices Architecture moving
to “Product Architecture”; most DS developments runs on GCP Make Tech required to do DS part of the Platform (particularly the “consolidated” elements) ~ Microservices Architecture; Hadoop cluster for DS “jobs” ~110 DevOps Teams ~50 Data Scientists/ML Engineers + DS Leadership Team); move to product-led organization DS becomes part of the Product Teams who need it (not just complicated subsystem teams); DS Leadership work on further establishing the discipline ~70 DevOps Teams (mostly Engineering & Product, ~10 Data Scientists) 1: Data Science evolution / From “DS Teams” to “DS Discipline” Sociotech Architecture State (Parts) Problem (Feedback loop & Observation) (Whole) Synthesis & Hypothesis Generation (Whole) Tech Org Prod Further growth; becoming product-led org and need for DS “everywhere” (complexity requires DS/AI/ML) “DS Teams” and central DS Department doesn’t scale or match “architecture” & philosophy of our “product teams” (x-functional teams that together improve the product - “no DS islands”!) * stop central DS Department & teams and move into establishing DS as a discipline that products can have (like engineering disciplines) DS is used in products that need it, becomes a strategic/investment decision of the product team time ~2017 ~2020 Big focus on becoming a “Retailer Platform”; own shop becomes second prio. Growing even more! “DS projects” don’t scale or match philosophy for DevOps Teams (You Build It, You Run You It, You Love It) + low maturity of Data Science in org -
@emgsilva esilva.net @emgsilva ~110 DevOps Teams ~50 Data Scientists/ML Engineers
+ DS Leadership Team); move to product-led organization ~70 DevOps Teams (mostly Engineering & Product, ~15 Data Scientists) ~20 DevOps Teams (mostly Engineering & Product), No DS present on teams ~ Microservices Architecture moving to “Product Architecture”; most DS developments runs on GCP ~ Microservices Architecture; Hadoop cluster for DS “jobs” ~Service-Oriented Architecture (groups of teams own “big services”) 1: Data Science evolution: from project, to teams to discipline Sociotech Architecture State (Parts) Problem (Feedback loop & Observation) (Whole) Synthesis & Hypothesis Generation (Whole) Tech Org Prod Further growth; becoming product-led org and need for DS “everywhere” (complexity requires DS/AI/ML) “DS Teams” and central DS Department doesn’t scale or match “architecture” & philosophy of our “product teams” (x-functional teams that together improve the product - “no DS islands”!) time ~2017 ~2020 Big focus on becoming a “Retailer Platform”; own shop becomes second prio. Growing even more! “DS projects” don’t scale or match philosophy for DevOps Teams (You Build It, You Run You It, You Love It) + low maturity of Data Science in org Online shop, moving to become also Retailer Platform (growing a lot!) Need DS to improve product relevant + address complexity, but no DS people on teams/org yet ~2012 * Get DS “pioneers” in * Start DS Projects in strategic topics * Start building “DS Tech” * create “DS Teams & Department”, working on high potential DS products * Mature DS Discipline * stop central DS Department & teams and move into establishing DS as a discipline that products can have (like engineering disciplines) (...) (...) (...) (...) -
@emgsilva esilva.net @emgsilva Change happens to “balance or maximize” properties
of our sociotechnical system Sociotech Architecture State (Parts) Problem (Feedback loop & Observation) (Whole) Synthesis & Hypothesis Generation (Whole) Tech Org Prod time (...) (...) (...) (...) -
@emgsilva esilva.net @emgsilva What are important things to improve (scale,
evolve, etc.) our (Tech) Org (& Why)? How can we do it effectively? -
@emgsilva esilva.net @emgsilva Zoom-out and holistically see “Why & What”;
then Zoom-in to the parts on “How” Sociotech Architecture State (Parts) Problem (Feedback loop & Observation) (Whole) Synthesis & Hypothesis Generation (Whole) Tech Org Prod time (...) (...) (...) (...) What are important things to improve (scale, evolve) our (Tech) Org (& Why)? How can we do this effectively? -
@emgsilva esilva.net @emgsilva Thank you! More on this in: esilva.net/sociotechnical
github.com/ddd-crew/ddd-starter-modelling-process @emgsilva | esilva.net | [email protected] (ping me for advising/consulting/talks on this topic)