Slide 1

Slide 1 text

PROGRAMMING in the LARGE Ģ B ģ z @markhibberd

Slide 2

Slide 2 text

Ģ B ģ z SYSTEM FORMULATION

Slide 3

Slide 3 text

How do we arrive at Systems?

Slide 4

Slide 4 text

Modernization Old New

Slide 5

Slide 5 text

Productionization Prototype Production

Slide 6

Slide 6 text

Consolidation Bigger Small Small

Slide 7

Slide 7 text

Modularization Big Smaller Smaller

Slide 8

Slide 8 text

Greenfield Legacy

Slide 9

Slide 9 text

Incremental Base More More

Slide 10

Slide 10 text

THESE ARE NOT SYSTEMS

Slide 11

Slide 11 text

Any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure Conway’s Law

Slide 12

Slide 12 text

The rules and boundaries of our systems should stem from architecture not organisation

Slide 13

Slide 13 text

How the system interacts What is the systems purpose Macro Architecture Domain Architecture How the system is built Micro Architecture

Slide 14

Slide 14 text

Systems as the unit of work Not unit of work as a system

Slide 15

Slide 15 text

Ģ B ģ z SYSTEMS of SYSTEMS

Slide 16

Slide 16 text

The thing about real systems is AUTONOMY

Slide 17

Slide 17 text

CROSS LANGUAGE CODE SEARCH AS AN EXAMPLE

Slide 18

Slide 18 text

Code search the JVM wrong way Pick Framework

Slide 19

Slide 19 text

Scala + Play Pick Language + Framework Code search the JVM wrong way

Slide 20

Slide 20 text

UI API FILESYSTEM Code search the JVM wrong way Scala + Play

Slide 21

Slide 21 text

UI API DB INDEXER Code search the JVM wrong way Scala + Play

Slide 22

Slide 22 text

SYSTEMS, NOT SUB-SYSTEMS

Slide 23

Slide 23 text

Code search as Systems INDEXING SEARCH

Slide 24

Slide 24 text

Code search as Systems Domain Architecture INDEXING SEARCH

Slide 25

Slide 25 text

Code search as Systems INDEXING SEARCH CODE CTAGS CTAGS application/html application/search.v1+json

Slide 26

Slide 26 text

Code search as Systems INDEXING SEARCH CODE CTAGS CTAGS application/html application/search.v1+json Macro Architecture

Slide 27

Slide 27 text

STANDARD FORMATS HELP GIVE US AUTONOMY

Slide 28

Slide 28 text

Code search as Systems INDEXING SEARCH Scala Embedded Server OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging OS for services

Slide 29

Slide 29 text

Code search as Systems INDEXING SEARCH Micro Architecture Scala Embedded Server OS for services OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging

Slide 30

Slide 30 text

Code search as Systems INDEXING SEARCH Micro Architecture Scala Embedded Server OS for services OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging MORE AUTONOMY

Slide 31

Slide 31 text

RULES ARCHITECTURE IS ABOUT NOT BOXES

Slide 32

Slide 32 text

Signs of failure Systems are not autonomous Systems share a domain model Systems need to be evolved in step Systems need to be built by single team Systems share a data base

Slide 33

Slide 33 text

Systems as the unit of work If you can’t add a person to your project then you are failing worse than you think again An Aside

Slide 34

Slide 34 text

Ģ B ģ z EVOLUTION

Slide 35

Slide 35 text

Thinking ahead is not about avoiding change Thinking ahead is about letting us change at different rates for different problems

Slide 36

Slide 36 text

Indexing V2 INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static Executable OS for services OS for logging OS for services

Slide 37

Slide 37 text

INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static Executable OS for services OS for logging OS for services Micro Architecture Indexing V2

Slide 38

Slide 38 text

now vs later priorities shift over time, can't let early traction sacrifice long term speed An Aside

Slide 39

Slide 39 text

INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static Executable OS for services OS for logging OS for services Indexing V2 DIFFERENT SYSTEMS SAME RULES DIFFERENT SYSTEMS DIFFERENT RULES An Aside

Slide 40

Slide 40 text

Macro Architecture Indexing V3 INDEXING SEARCH CODE CTAGS CTAGS application/html application/search.v1+json Super-Zippy Format

Slide 41

Slide 41 text

Macro Architecture Search V2 INDEXING SEARCH CODE CTAGS Super-Zippy Format application/html application/search.v1+json Super-Zippy Format

Slide 42

Slide 42 text

Notice how easy it is to think of evolution in the large

Slide 43

Slide 43 text

Rates of change domain macro micro

Slide 44

Slide 44 text

Rates of change domain macro micro Freedom to choose tools and technology by succeeding at domain and protocols

Slide 45

Slide 45 text

HOW DO SYSTEMS THINK HOW DO SYSTEMS COMMUNICATE HOW DO SYSTEMS CHANGE

Slide 46

Slide 46 text

Implications at Scale

Slide 47

Slide 47 text

The concept of micro-services are great An Aside The concept of small libraries are great But they are not enough without thinking in systems

Slide 48

Slide 48 text

Larger Systems of Systems

Slide 49

Slide 49 text

How long does it take to get 1 line of code to production

Slide 50

Slide 50 text

Larger Systems of Systems

Slide 51

Slide 51 text

Mistaking Modules For Systems

Slide 52

Slide 52 text

Mistaking Modules For Systems

Slide 53

Slide 53 text

Mistaking Modules For Systems

Slide 54

Slide 54 text

Implication of Autonomy USERS

Slide 55

Slide 55 text

Implication of (lack of) Autonomy USERS

Slide 56

Slide 56 text

Ģ B ģ z SYSTEMS in ACTION

Slide 57

Slide 57 text

CMSs hold customer data hostage CONTENT ANALYSIS & DATA MINING PROJECT Free data Analyse data Derive data CMSs tied to horrible platforms Goals Constraints

Slide 58

Slide 58 text

CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1

Slide 59

Slide 59 text

CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 JSP Java

Slide 60

Slide 60 text

CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 1 JSP Java

Slide 61

Slide 61 text

CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller 1 2 Java 1 JSP

Slide 62

Slide 62 text

CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller 1 2 1 JSP Java Java Scala

Slide 63

Slide 63 text

WHAT I IMAGINED DATA WEB CORE LEECH PORTAL CONTENT ANALYSIS & DATA MINING PROJECT

Slide 64

Slide 64 text

WEB / CORE / DATA / LEECH WEBSHERE REALITY SETTING IN

Slide 65

Slide 65 text

CAUGHT THINKING IN THE SMALL MODULES - LIBRARIES - TECH

Slide 66

Slide 66 text

We should leverage architecture changes rather than force new technology

Slide 67

Slide 67 text

CONTENT ANALYSIS IDENTITY CONTENT STORE QUERY ENGINE A Better Approach

Slide 68

Slide 68 text

Play 1.1 App Our Starting Point

Slide 69

Slide 69 text

Play 1.1 App Enabling Moves IDENTITY

Slide 70

Slide 70 text

Play 1.1 App Protobufs Thrift HTTP Establishing Rules & Autonomy sbt-assembly OS package Containers IDENTITY Protocols Deployment

Slide 71

Slide 71 text

Play 1.1 App Identifying Demand CONTENT STORE IDENTITY

Slide 72

Slide 72 text

Play 1.1 App Validation CONTENT STORE IDENTITY

Slide 73

Slide 73 text

Play 1.1 App Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS

Slide 74

Slide 74 text

Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS QUERY ENGINE

Slide 75

Slide 75 text

architecture is an everyday task

Slide 76

Slide 76 text

good architecture means never having to rewrite

Slide 77

Slide 77 text

All editors are bad, really bad ONLINE CONTENT EDITING Something aaS APIs Ready to change Small team, tight deadline Goals Constraints Lots of ideas, not sure what would work

Slide 78

Slide 78 text

Spelling Identity Templates Editor Customer UI Domain Architecture

Slide 79

Slide 79 text

Spelling Identity Templates Editor Customer UI Macro Architecture HTTP / JSON

Slide 80

Slide 80 text

C Scala Scala Spelling Identity Templates Editor Customer UI Micro Architecture Scala/JS Scala Postgres Postgres Postgres syslog jail per service rc.d 1 executable Minimal stack

Slide 81

Slide 81 text

Editor was working well ONLINE CONTENT EDITING Incremental Multiple Versions Add not modify Needed to add more services Goals Constraints System need to keep running

Slide 82

Slide 82 text

Spelling Identity Templates Content Editor Analysis Customer UI

Slide 83

Slide 83 text

Spelling Identity Templates Content Editor Analysis Customer UI HTTP / JSON

Slide 84

Slide 84 text

Spelling Identity Templates Content Editor Analysis Customer UI Scala/JS Scala JS C Scala Scala Scala multilog jail per service daemon tools 1 executable Minimal stack

Slide 85

Slide 85 text

Demand for standalone features ONLINE CONTENT EDITING Quick Minimal Overhead Maximum Reuse Not tied to our editor Goals Constraints Run in parallel to current system

Slide 86

Slide 86 text

Spelling Identity Templates Content Editor Analysis Customer UI

Slide 87

Slide 87 text

UGLY BITS

Slide 88

Slide 88 text

Spelling Identity Templates Content Editor Analysis Customer UI Glue Glue Glue Glue Glue Glue Glue

Slide 89

Slide 89 text

Spelling Identity Templates Content Editor Analysis Customer UI Config

Slide 90

Slide 90 text

ARCHITECTURE SHOULD HAVE RULES FOR HOW WE SHARE CODE

Slide 91

Slide 91 text

Ģ B ģ z RECAP

Slide 92

Slide 92 text

RULES ARCHITECTURE IS ABOUT NOT BOXES

Slide 93

Slide 93 text

good architecture means never having to rewrite

Slide 94

Slide 94 text

Thinking ahead is not about avoiding change Thinking ahead is about letting us change at different rates for different problems

Slide 95

Slide 95 text

Systems as the unit of work Not unit of work as a system

Slide 96

Slide 96 text

The thing about real systems is AUTONOMY

Slide 97

Slide 97 text

Ģ B ģ z FIN @markhibberd

Slide 98

Slide 98 text

References + More Info Stefan Tilkov's Breaking the Monolith slides video Coda Hale and Ryan Kennedy on "Streamie" notification service at yammer Slide Deck v2 v1