PROGRAMMING
in the
LARGE
Ģ
B
ģ
z
@markhibberd
NICTA
Slide 2
Slide 2 text
Ģ
B
ģ
z
SYSTEM
FORMULATION
Slide 3
Slide 3 text
How do we arrive at
Systems?
Slide 4
Slide 4 text
Modernization
Old New
Slide 5
Slide 5 text
Productionization
Prototype Production
Slide 6
Slide 6 text
Consolidation
Bigger
Small
Small
Slide 7
Slide 7 text
Modularization
Big
Smaller
Smaller
Slide 8
Slide 8 text
Greenfield
Legacy
Slide 9
Slide 9 text
Incremental
Base
More More
Slide 10
Slide 10 text
THESE ARE NOT
SYSTEMS
Slide 11
Slide 11 text
Any organization that designs a
system will produce a design whose
structure is a copy of the organization's
communication structure
Conway’s Law
Slide 12
Slide 12 text
The rules and boundaries
of our systems should
stem from architecture
not organization
Slide 13
Slide 13 text
HOW DO SYSTEMS THINK
HOW DO SYSTEMS COMMUNICATE
HOW DO SYSTEMS CHANGE
Slide 14
Slide 14 text
Systems as the unit of work
Not unit of work as a system
Slide 15
Slide 15 text
Ģ
B
ģ
z
SYSTEMS
of
SYSTEMS
Slide 16
Slide 16 text
The thing about real systems is
AUTONOMY
Slide 17
Slide 17 text
CODE SEARCH
An Example
Slide 18
Slide 18 text
Code search the normal wrong way
Pick Framework
Slide 19
Slide 19 text
UI API
DB INDEXER
Code search the wrong way
Slide 20
Slide 20 text
RULES
ARCHITECTURE IS ABOUT
NOT BOXES
Slide 21
Slide 21 text
Code search as Systems
INDEXING SEARCH
Slide 22
Slide 22 text
Code search as Systems
INDEXING SEARCH
CODE
CTAGS
CTAGS
application/html
application/search.v1+json
Slide 23
Slide 23 text
STANDARD FORMATS HELP
PROVIDE AUTONOMY
Slide 24
Slide 24 text
Code search as Systems
INDEXING SEARCH
Scala
Embedded Server
OS for logging
Bourne Shell
Exuberant CTAGS
Deploy as git hook
OS for logging
OS for services
Slide 25
Slide 25 text
now vs later
priorities shift over time,
can't let early traction
sacrifice long term speed
An Aside
Slide 26
Slide 26 text
Ģ
B
ģ
z
EVOLUTION
Slide 27
Slide 27 text
is the default state
LEGACY
Slide 28
Slide 28 text
Thinking ahead is not about avoiding change
Thinking ahead is about letting us change at
different rates for different problems
Slide 29
Slide 29 text
Rates of change
domain
macro
micro
Slide 30
Slide 30 text
Rates of change
domain
macro
micro
Freedom to choose tools and technology by succeeding at domain and protocols
Slide 31
Slide 31 text
STANDARDS HELP
AVOID CHAOS
Slide 32
Slide 32 text
Implications at
Scale
Slide 33
Slide 33 text
Larger Systems of Systems
Slide 34
Slide 34 text
How long does it take to get 1 line
of code to production
Slide 35
Slide 35 text
Larger Systems of Systems
Slide 36
Slide 36 text
Mistaking Modules For Systems
Slide 37
Slide 37 text
Mistaking Modules For Systems
Slide 38
Slide 38 text
Mistaking Modules For Systems
Slide 39
Slide 39 text
AUTONOMOUS SYSTEMS
EVOLVE INDEPENDENTLY
Slide 40
Slide 40 text
Implication of Autonomy
USERS
Slide 41
Slide 41 text
Implication of (lack of) Autonomy
USERS
Slide 42
Slide 42 text
Notice how easy it is to think of
evolution in the large
Slide 43
Slide 43 text
Ģ
B
ģ
z
SYSTEMS
in
ACTION
Slide 44
Slide 44 text
CMSs hold customer data hostage
CONTENT ANALYSIS & DATA MINING PROJECT
Free data
Analyse data
Derive data
CMSs tied to horrible platforms
Goals
Constraints
Slide 45
Slide 45 text
CONTENT ANALYSIS & DATA MINING PROJECT
Prototype Production
1
Slide 46
Slide 46 text
CONTENT ANALYSIS & DATA MINING PROJECT
Prototype Production
1
JSP Java
Slide 47
Slide 47 text
CONTENT ANALYSIS & DATA MINING PROJECT
Prototype Production
1
1 JSP Java
Slide 48
Slide 48 text
CONTENT ANALYSIS & DATA MINING PROJECT
Prototype Production
Smaller Smaller
1
2
Java
1 JSP
Slide 49
Slide 49 text
CONTENT ANALYSIS & DATA MINING PROJECT
Prototype Production
Smaller Smaller
1
2
1 JSP Java
Java
Scala
Slide 50
Slide 50 text
WHAT I IMAGINED
DATA
WEB
CORE EXTRACT PORTAL
CONTENT ANALYSIS & DATA MINING PROJECT
Slide 51
Slide 51 text
WEB / CORE / DATA / EXTRACT
WEBSHERE
REALITY SETTING IN
Slide 52
Slide 52 text
Signs of failure
Systems are not autonomous
Systems share a domain model
Systems need to be evolved in step
Systems need to be built by single team
Systems share a data base
Slide 53
Slide 53 text
Systems as the unit of work
If you can’t add a person to
your project then you are
failing worse than you think
again
Slide 54
Slide 54 text
CONTENT
ANALYSIS
IDENTITY
CONTENT
STORE
QUERY
ENGINE
A Better Approach
Slide 55
Slide 55 text
The Big App
Our Starting Point
Slide 56
Slide 56 text
The Big App
Enabling Moves
IDENTITY
Slide 57
Slide 57 text
The Big App
Protobufs
Thrift
HTTP
Establishing Rules & Autonomy
sbt-assembly
OS package
Containers
IDENTITY
Protocols
Deployment
Slide 58
Slide 58 text
The Big App
Identifying Demand
CONTENT
STORE
IDENTITY
Slide 59
Slide 59 text
The Big App
Validation
CONTENT
STORE
IDENTITY
Slide 60
Slide 60 text
The Big App
Repeat
CONTENT
STORE
IDENTITY
CONTENT
ANALYSIS
Slide 61
Slide 61 text
Repeat
CONTENT
STORE
IDENTITY
CONTENT
ANALYSIS
QUERY
ENGINE
Slide 62
Slide 62 text
leverage architecture changes rather
than force new technology
Slide 63
Slide 63 text
good architecture means never
having to rewrite
Slide 64
Slide 64 text
architecture is an everyday task
Slide 65
Slide 65 text
Ģ
B
ģ
z
RECAP
Slide 66
Slide 66 text
Systems as the unit of work
Not unit of work as a system
Slide 67
Slide 67 text
The thing about real systems is
AUTONOMY
Slide 68
Slide 68 text
is the default state
LEGACY
Slide 69
Slide 69 text
Ģ
B
ģ
z
FIN
@markhibberd
Slide 70
Slide 70 text
References + More Info
Stefan Tilkov's Breaking the Monolith slides video
Coda Hale and Ryan Kennedy on "Streamie" notification service at yammer
Slide Deck v2
v1 v3