Scaling & Evolving Tech-driven Orgs Using sociotechnical Architecture & Systems Thinking

@emgsilva
esilva.net
Scaling & Evolving Tech-driven Orgs
Using sociotechnical Architecture &
Systems Thinking
Eduardo da Silva
(Principal Tech Lead @ bol.com)
@emgsilva | [email protected] | esilva.net

View Slide

@emgsilva
esilva.net @emgsilva
> WhoamI
https://careers.bol.com

View Slide

@emgsilva
What are important
things to improve
(scale, evolve) our
(Tech) Org (& Why)?
How can we do it
effectively?

View Slide

@emgsilva
...the journey from the problem space of
the users to the solution space of the
business. The goal of this journey is to
reduce the gap between users and the
business. -Naren Katakam
credits: https://uxplanet.org/product-thinking-101-1d71a0784f60, Naren Katakam

View Slide

@emgsilva
Continuous
Learning
Loop
Product-led Org
(Service)
Product
Product Thinking: customer-centric product discovery & delivery

View Slide

@emgsilva
"Our highest priority is to
satisfy the customer through
early and continuous delivery
of valuable software."

View Slide

@emgsilva
Product Thinking: customer-centric product discovery & delivery
=>End Goal: To Maximize the Value Exchange with Customer
Value
Exchange
Value
$$$
Product-led Org
(Service)
Product

View Slide

@emgsilva
Now… How can we “maximize the value exchange with
customer” (continuously + high velocity + ...)
Value
Exchange
Value
$$$
Product-led Org
(Service)
Product

View Slide

@emgsilva
Is DevOps or “BizDevOps” the answer?
Stopping with “silos” between “The Business”
(Product) and “IT” (Tech) can definitely help
improving things...
...but there are even more things to consider in
this continuously changing equation (of maximizing
the customer value exchange)
>>> This is where we will focus today! <<<

View Slide

@emgsilva
LEt’s talk about Restaurants...
...some traits to make “Great Restaurants” (products)

View Slide

@emgsilva
Trait 1: Your staff deﬁnes The Restaurant cooking and
experiences
If you want an amazing French food
experience you need to have French
chefs and French wine experts; and they
have to work together to come up with
great french food experiences.
Your teams shape your product

View Slide

@emgsilva
Trait 2: Chefs cannot cook, serve the food and do the
dishes at the same time...
If you want to run your restaurant
smoothly and have a strong team for
a long time, you need to have
enough people and skill to support
the different tasks without having
people “burning out” (or leaving)
Your teams must have the right conditions to build
the product for your customer

View Slide

@emgsilva
Trait 3: Kitchen + Serving + Wine + ... = Restaurant
Your restaurant has different
parts and people working on them
(in parallel); however, you want
to understand and optimize how it
all comes together in order to
maximize your “product” (value to
customer)
Your product is the combination of different value
streams, owned by different teams in the organization
but aligned to maximize value for customer

View Slide

@emgsilva
Trait 4: don’t hand recipes to your chefs… Empower
them to discover new recipes
Do you want to achieve unique
experiences (and value exchange) for
your customers? Then enable your crew
to "experiment & discover" how to
maximize that! (continuously)
Your teams discovers (with the customer) what are
the best things to build in the product to maximize
the value exchange (continuously)

View Slide

@emgsilva
Recapping: How can we “maximize the value exchange with
customer” (continuously + high velocity + ...)
1: Your teams shape your product
2: Your teams must have the right conditions to build the
product for your customer
3: Your product is the combination of different value streams,
owned by different teams in the organization but aligned to
maximize value for customer
4: Your teams discovers (with the customer) what are the
best things to build in the product to maximize the value
exchange (continuously)

View Slide

@emgsilva
We must take an holistic view to the “What” + “Who” +
“How” systems… Isolated local optimizations will not do it!
(Service)
Product
What
enables
maximize
Value
Exchange
(Product)
Value
$$$
Who is
doing this?
(Teams)
How are we
doing it?
(Tech Arch)

View Slide

@emgsilva
This is Systems Thinking... To manage a system
effectively, you might
focus on the
interactions of the
parts rather than their
behavior taken
separately
...Understanding*
proceeds from the
whole to its parts,
not from the parts
to the whole as
knowledge does.
Russell Ackoff
* Understanding => answering the “Why” question
...identify the
containing whole.
For example, to
understand a car
you identify the
transportation
system ...

View Slide

@emgsilva
to make a great “dish"...
...understand the whole restaurant (system) context!

View Slide

@emgsilva
“You (should) never improve the quality of the dishes if
it does not improve the quality of the restaurant
simultaneously”
=> Let the parts of the whole improve the quality of the whole!

View Slide

@emgsilva
(Whole) Systems Understanding = {
Understanding “basic Context” {
Structure x dynamics
}
+
Understanding of the “Emerging Context” {
properties that rise from the interactions the structure and
dynamics in the context
}
}
reference: https://twitter.com/ruthmalan/status/1093294478641823744

View Slide

@emgsilva
So, what about our world of building Software Products?
How can we use these ideas?

View Slide

@emgsilva
The “Whole” as A Sociotechnical Systems Model

View Slide

@emgsilva
Deﬁnition: Sociotechnical architecture
“Sociotechnical Architecture is
about taking an holistic co-design
approach to technical and
organizational systems, given the
inherent impact they have on each
other.” --Eduardo da Silva
Source: Sociotechnical Architecture Definition thread
“When we are architecting a software
system, we must consider the impact
on the teams in the organisation and
vice-versa.” --Nick Tune

View Slide

@emgsilva
How to Approach Sociotechnical Architecture?
while(true) {
// understand based on the whole &
feedback loops data, then build hypothesis
for direction of actions on parts (or new
parts)
hypothesis = synthesis(data);
// action and analysis on the parts
(collect feedback data to “learn”)
data = actionAndAnalysis(hypothesis);
}
reference: Analysis, synthesis, systems thinking and the scientific method: Rediscovering the importance of open systems

View Slide

@emgsilva
Bol.com {prod/org/arch & DS} Evolution
=> Learning from patterns of scaling & evolving,
through a Sociotechnical Architecture & Systems
thinking lens

View Slide

@emgsilva
* Active customers purchased at least once at
bol.com in the last 365 days. Active sellers
offered at least one item via bol.com in the last
365 days or currently still offer items via bol.com

View Slide

@emgsilva
* Active customers purchased at least once at
bol.com in the last 365 days. Active sellers
offered at least one item via bol.com in the last
365 days or currently still offer items via bol.com
Monoliths
Products
Tech
Arch
Org
(&
Way
of
Work)
Waterfall Agile
Product Teams
DevOps Teams
Bol.com Sociotechnical Arch (prod/org/arch) evolution...
Prod
(Micro)services
~30
~100
~10 ~600 ~50

View Slide

@emgsilva
Tech (arch) context Action on tech arch
Org context Action on org
Data Science (DS) Evolution @ bol.com
while(true) {
// understand based on the whole &
feedback loops data, then build
hypothesis for direction of actions
on parts (or new parts)
hypothesis = synthesis(data);
// action and analysis on the parts
(collect feedback data to “learn”)
data =
actionAndAnalysis(hypothesis);
}
Sociotech
Architecture
State
(Parts)
Problem
(Feedback
loop &
Observation)
(Whole)
Synthesis &
Hypothesis
Generation
(Whole)
Tech
Org
Prod
Product context
Problem (observed from the
“Whole”)
Synthesis (understanding)
and definition of hypothesis
Action on product
time
Actions
on Parts
year

View Slide

@emgsilva
~20 DevOps Teams (mostly
Engineering & Product), No
DS present on teams
Bring “DS Pioneers” to Teams
~Service-Oriented
Architecture (groups of
teams own “big services”)
Start developing DS Tech
(e.g.: Create Hadoop
Cluster)
1: Data Science evolution / “DS Projects” adoption
Some context:
● Data Science has very
particular ways of
working (i.e.:
embedding it on a
team is not trivial)
● “DS Projects” means
“DS Pioneers” come to
a team (who has a
certain scope),
implement DS project
and go to next “gig”
Sociotech
Architecture
State
(Parts)
Problem
(Feedback
loop &
Observation)
(Whole)
Synthesis &
Hypothesis
Generation
(Whole)
Tech
Org
Prod
Online shop, moving to
become also Retailer
Platform (growing a lot!)
Need DS to improve product
relevant + address
complexity, but no DS people
on teams/org yet
* Get DS “pioneers” in
* Start DS Projects in
strategic topics
* Start building “DS Tech”
DS projects on strategic
parts of the product (e.g.:
Recommendations; sales
forecasts)
time
Actions
on Parts
~2012

View Slide

@emgsilva
Engineering & Product, ~10
Data Scientists)
~10 DS Teams (the teams who had strong
DS component or potential); explicit
introduction of DS Leadership team
DS present on teams
~ Microservices
Architecture; Hadoop cluster
for DS “jobs”
Explore how to bring DS tech support
to next level (including leverage GCP
components
~Service-Oriented
1: Data Science evolution / From “DS Projects” to “DS Teams”*
Sociotech
Architecture
State
(Parts)
Problem
(Feedback
loop &
Observation)
(Whole)
Synthesis &
Hypothesis
Generation
(Whole)
Tech
Org
Prod Big focus on becoming a
“Retailer Platform”; own
shop becomes second prio.
Growing even more!
“DS projects” don’t scale or
match philosophy for DevOps
Teams (You Build It, You Run
You It, You Love It) + low
maturity of Data Science in
org
* create “DS Teams & Department”,
working on high potential DS products
* Mature DS Discipline
DS is part of products with high
potential
time
Actions
on Parts
~2012
relevant + address
on teams
~2017
* some are “complicated sub-systems” serving multiple “stream aligned teams”... but not all of them fit that team topology

View Slide

@emgsilva
Actions
on Parts
~ Microservices Architecture
moving to “Product
Architecture”; most DS
developments runs on GCP
Make Tech required to do DS part of
the Platform (particularly the
“consolidated” elements)
~ Microservices
for DS “jobs”
~110 DevOps Teams ~50 Data
Scientists/ML Engineers + DS
Leadership Team); move to
product-led organization
DS becomes part of the Product Teams
who need it (not just complicated
subsystem teams); DS Leadership work
on further establishing the discipline
Data Scientists)
1: Data Science evolution / From “DS Teams” to “DS Discipline”
Sociotech
Architecture
State
(Parts)
Problem
(Feedback
loop &
Observation)
(Whole)
Synthesis &
Hypothesis
Generation
(Whole)
Tech
Org
Prod Further growth; becoming
product-led org and need for
DS “everywhere” (complexity
requires DS/AI/ML)
“DS Teams” and central DS
Department doesn’t scale or
match “architecture” &
philosophy of our “product
teams” (x-functional teams
that together improve the
product - “no DS islands”!)
* stop central DS Department & teams
and move into establishing DS as a
discipline that products can have
(like engineering disciplines)
DS is used in products that need it,
becomes a strategic/investment
decision of the product team
time
~2017 ~2020
Big focus on becoming a
Growing even more!
org

View Slide

@emgsilva
~110 DevOps Teams ~50 Data
Scientists/ML Engineers + DS
Leadership Team); move to
product-led organization
Data Scientists)
DS present on teams
~ Microservices Architecture
moving to “Product
Architecture”; most DS
developments runs on GCP
~ Microservices
for DS “jobs”
~Service-Oriented
1: Data Science evolution: from project, to teams to discipline
Sociotech
Architecture
State
(Parts)
Problem
(Feedback
loop &
Observation)
(Whole)
Synthesis &
Hypothesis
Generation
(Whole)
Tech
Org
Prod Further growth; becoming
product-led org and need for
DS “everywhere” (complexity
requires DS/AI/ML)
“DS Teams” and central DS
Department doesn’t scale or
match “architecture” &
philosophy of our “product
teams” (x-functional teams
that together improve the
product - “no DS islands”!)
time
~2017 ~2020
Big focus on becoming a
Growing even more!
org
relevant + address
on teams/org yet
~2012
* Get DS “pioneers” in
* Start DS Projects in
strategic topics
* Start building “DS Tech”
* create “DS Teams & Department”,
working on high potential DS products
* Mature DS Discipline
* stop central DS Department & teams
and move into establishing DS as a
discipline that products can have
(like engineering disciplines)
(...)
(...)
(...)
(...)

View Slide

@emgsilva
Change happens to “balance or maximize” properties of our sociotechnical system
Sociotech
Architecture
State
(Parts)
Problem
(Feedback
loop &
Observation)
(Whole)
Synthesis &
Hypothesis
Generation
(Whole)
Tech
Org
Prod
time
(...)
(...)
(...)
(...)

View Slide

@emgsilva
What are important
things to improve
(scale, evolve, etc.)
our (Tech) Org (&
Why)?
How can we do it
effectively?

View Slide

@emgsilva
Zoom-out and holistically see “Why & What”; then Zoom-in to the parts on “How”
Sociotech
Architecture
State
(Parts)
Problem
(Feedback
loop &
Observation)
(Whole)
Synthesis &
Hypothesis
Generation
(Whole)
Tech
Org
Prod
time
(...)
(...)
(...)
(...)
What are important things
to improve (scale, evolve)
our (Tech) Org (& Why)?
How can we do this
effectively?

View Slide

@emgsilva
Thank you!
More on this in:
esilva.net/sociotechnical
github.com/ddd-crew/ddd-starter-modelling-process
@emgsilva | esilva.net | [email protected]
(ping me for advising/consulting/talks on this topic)

View Slide

Scaling & Evolving Tech-driven Orgs Using sociotechnical Architecture & Systems Thinking

Scaling & Evolving Tech-driven Orgs Using sociotechnical Architecture & Systems Thinking

Eduardo da Silva

More Decks by Eduardo da Silva

Other Decks in Technology

Featured

Transcript