Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Programming in the Large Again

Mark Hibberd
November 27, 2013

Programming in the Large Again

Good architecture frees us to choose the right tools and techniques, allowing us to adapt easily and concentrate on solving real problems rather than our made up ones. In this talk we will run through some stereotypical projects looking at the properties of good architectures and how these play into our ability to adopt better tools like Scala and FP. We will attempt to ground the magical architecture ponies, with real examples of my past projects where things have gone well and probably of more interest where they really have not.

This is a slightly improved version that I presented at NICTA on 28th of November 2013.

Mark Hibberd

November 27, 2013
Tweet

More Decks by Mark Hibberd

Other Decks in Programming

Transcript

  1. PROGRAMMING
    in the
    LARGE
    Ģ
    B
    ģ
    z
    @markhibberd

    View Slide

  2. Ģ
    B
    ģ
    z
    SYSTEM
    FORMULATION

    View Slide

  3. How do we arrive at
    Systems?

    View Slide

  4. Modernization
    Old New

    View Slide

  5. Productionization
    Prototype Production

    View Slide

  6. Consolidation
    Bigger
    Small
    Small

    View Slide

  7. Modularization
    Big
    Smaller
    Smaller

    View Slide

  8. Greenfield
    Legacy

    View Slide

  9. Incremental
    Base
    More More

    View Slide

  10. THESE ARE NOT
    SYSTEMS

    View Slide

  11. Any organization that designs a
    system will produce a design whose
    structure is a copy of the organization's
    communication structure
    Conway’s Law

    View Slide

  12. The rules and boundaries
    of our systems should
    stem from architecture
    not organisation

    View Slide

  13. How the system interacts
    What is the systems purpose
    Macro Architecture
    Domain Architecture
    How the system is built
    Micro Architecture

    View Slide

  14. Systems as the unit of work
    Not unit of work as a system

    View Slide

  15. Ģ
    B
    ģ
    z
    SYSTEMS
    of
    SYSTEMS

    View Slide

  16. The thing about real systems is
    AUTONOMY

    View Slide

  17. CROSS LANGUAGE CODE SEARCH
    AS AN EXAMPLE

    View Slide

  18. Code search the JVM wrong way
    Pick Framework

    View Slide

  19. Scala + Play
    Pick Language + Framework
    Code search the JVM wrong way

    View Slide

  20. UI API
    FILESYSTEM
    Code search the JVM wrong way
    Scala + Play

    View Slide

  21. UI API
    DB INDEXER
    Code search the JVM wrong way
    Scala + Play

    View Slide

  22. SYSTEMS, NOT SUB-SYSTEMS

    View Slide

  23. Code search as Systems
    INDEXING SEARCH

    View Slide

  24. Code search as Systems
    Domain Architecture
    INDEXING SEARCH

    View Slide

  25. Code search as Systems
    INDEXING SEARCH
    CODE
    CTAGS
    CTAGS
    application/html
    application/search.v1+json

    View Slide

  26. Code search as Systems
    INDEXING SEARCH
    CODE
    CTAGS
    CTAGS
    application/html
    application/search.v1+json
    Macro Architecture

    View Slide

  27. STANDARD FORMATS HELP
    GIVE US AUTONOMY

    View Slide

  28. Code search as Systems
    INDEXING SEARCH
    Scala
    Embedded Server
    OS for logging
    Bourne Shell
    Exuberant CTAGS
    Deploy as git hook
    OS for logging
    OS for services

    View Slide

  29. Code search as Systems
    INDEXING SEARCH
    Micro Architecture
    Scala
    Embedded Server
    OS for services
    OS for logging
    Bourne Shell
    Exuberant CTAGS
    Deploy as git hook
    OS for logging

    View Slide

  30. Code search as Systems
    INDEXING SEARCH
    Micro Architecture
    Scala
    Embedded Server
    OS for services
    OS for logging
    Bourne Shell
    Exuberant CTAGS
    Deploy as git hook
    OS for logging
    MORE AUTONOMY

    View Slide

  31. RULES
    ARCHITECTURE IS ABOUT
    NOT BOXES

    View Slide

  32. Signs of failure
    Systems are not autonomous
    Systems share a domain model
    Systems need to be evolved in step
    Systems need to be built by single team
    Systems share a data base

    View Slide

  33. Systems as the unit of work
    If you can’t add a person to
    your project then you are
    failing worse than you think
    again
    An Aside

    View Slide

  34. Ģ
    B
    ģ
    z
    EVOLUTION

    View Slide

  35. Thinking ahead is not about avoiding change
    Thinking ahead is about letting us change at
    different rates for different problems

    View Slide

  36. Indexing V2
    INDEXING SEARCH
    Scala
    Embedded Server
    OS for logging
    Haskell
    Static Executable
    OS for services
    OS for logging
    OS for services

    View Slide

  37. INDEXING SEARCH
    Scala
    Embedded Server
    OS for logging
    Haskell
    Static Executable
    OS for services
    OS for logging
    OS for services
    Micro Architecture
    Indexing V2

    View Slide

  38. now vs later
    priorities shift over time,
    can't let early traction
    sacrifice long term speed
    An Aside

    View Slide

  39. INDEXING SEARCH
    Scala
    Embedded Server
    OS for logging
    Haskell
    Static Executable
    OS for services
    OS for logging
    OS for services
    Indexing V2
    DIFFERENT SYSTEMS SAME RULES
    DIFFERENT SYSTEMS DIFFERENT RULES
    An Aside

    View Slide

  40. Macro Architecture
    Indexing V3
    INDEXING SEARCH
    CODE
    CTAGS
    CTAGS
    application/html
    application/search.v1+json
    Super-Zippy Format

    View Slide

  41. Macro Architecture
    Search V2
    INDEXING SEARCH
    CODE
    CTAGS
    Super-Zippy Format
    application/html
    application/search.v1+json
    Super-Zippy Format

    View Slide

  42. Notice how easy it is to think of
    evolution in the large

    View Slide

  43. Rates of change
    domain
    macro
    micro

    View Slide

  44. Rates of change
    domain
    macro
    micro
    Freedom to choose tools and technology by succeeding at domain and protocols

    View Slide

  45. HOW DO SYSTEMS THINK
    HOW DO SYSTEMS COMMUNICATE
    HOW DO SYSTEMS CHANGE

    View Slide

  46. Implications at
    Scale

    View Slide

  47. The concept of micro-services are great
    An Aside
    The concept of small libraries are great
    But they are not enough without
    thinking in systems

    View Slide

  48. Larger Systems of Systems

    View Slide

  49. How long does it take to get 1 line
    of code to production

    View Slide

  50. Larger Systems of Systems

    View Slide

  51. Mistaking Modules For Systems

    View Slide

  52. Mistaking Modules For Systems

    View Slide

  53. Mistaking Modules For Systems

    View Slide

  54. Implication of Autonomy
    USERS

    View Slide

  55. Implication of (lack of) Autonomy
    USERS

    View Slide

  56. Ģ
    B
    ģ
    z
    SYSTEMS
    in
    ACTION

    View Slide

  57. CMSs hold customer data hostage
    CONTENT ANALYSIS & DATA MINING PROJECT
    Free data
    Analyse data
    Derive data
    CMSs tied to horrible platforms
    Goals
    Constraints

    View Slide

  58. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    1

    View Slide

  59. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    1
    JSP Java

    View Slide

  60. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    1
    1 JSP Java

    View Slide

  61. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    Smaller Smaller
    1
    2
    Java
    1 JSP

    View Slide

  62. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    Smaller Smaller
    1
    2
    1 JSP Java
    Java
    Scala

    View Slide

  63. WHAT I IMAGINED
    DATA
    WEB
    CORE LEECH PORTAL
    CONTENT ANALYSIS & DATA MINING PROJECT

    View Slide

  64. WEB / CORE / DATA / LEECH
    WEBSHERE
    REALITY SETTING IN

    View Slide

  65. CAUGHT THINKING IN THE
    SMALL
    MODULES - LIBRARIES - TECH

    View Slide

  66. We should leverage architecture changes
    rather than force new technology

    View Slide

  67. CONTENT
    ANALYSIS
    IDENTITY
    CONTENT
    STORE
    QUERY
    ENGINE
    A Better Approach

    View Slide

  68. Play 1.1 App
    Our Starting Point

    View Slide

  69. Play 1.1 App
    Enabling Moves
    IDENTITY

    View Slide

  70. Play 1.1 App
    Protobufs
    Thrift
    HTTP
    Establishing Rules & Autonomy
    sbt-assembly
    OS package
    Containers
    IDENTITY
    Protocols
    Deployment

    View Slide

  71. Play 1.1 App
    Identifying Demand
    CONTENT
    STORE
    IDENTITY

    View Slide

  72. Play 1.1 App
    Validation
    CONTENT
    STORE
    IDENTITY

    View Slide

  73. Play 1.1 App
    Repeat
    CONTENT
    STORE
    IDENTITY
    CONTENT
    ANALYSIS

    View Slide

  74. Repeat
    CONTENT
    STORE
    IDENTITY
    CONTENT
    ANALYSIS
    QUERY
    ENGINE

    View Slide

  75. architecture is an everyday task

    View Slide

  76. good architecture means never
    having to rewrite

    View Slide

  77. All editors are bad, really bad
    ONLINE CONTENT EDITING
    Something aaS
    APIs
    Ready to change
    Small team, tight deadline
    Goals
    Constraints
    Lots of ideas, not sure what would work

    View Slide

  78. Spelling Identity Templates
    Editor
    Customer
    UI
    Domain Architecture

    View Slide

  79. Spelling Identity Templates
    Editor
    Customer
    UI
    Macro Architecture
    HTTP / JSON

    View Slide

  80. C Scala Scala
    Spelling Identity Templates
    Editor
    Customer
    UI
    Micro Architecture
    Scala/JS Scala
    Postgres
    Postgres Postgres
    syslog
    jail per service
    rc.d
    1 executable
    Minimal stack

    View Slide

  81. Editor was working well
    ONLINE CONTENT EDITING
    Incremental
    Multiple Versions
    Add not modify
    Needed to add more services
    Goals
    Constraints
    System need to keep running

    View Slide

  82. Spelling Identity Templates Content
    Editor Analysis
    Customer
    UI

    View Slide

  83. Spelling Identity Templates Content
    Editor Analysis
    Customer
    UI
    HTTP / JSON

    View Slide

  84. Spelling Identity Templates Content
    Editor Analysis
    Customer
    UI
    Scala/JS Scala JS
    C Scala Scala Scala
    multilog
    jail per service
    daemon tools
    1 executable
    Minimal stack

    View Slide

  85. Demand for standalone features
    ONLINE CONTENT EDITING
    Quick
    Minimal Overhead
    Maximum Reuse
    Not tied to our editor
    Goals
    Constraints
    Run in parallel to current system

    View Slide

  86. Spelling Identity Templates Content
    Editor Analysis
    Customer
    UI

    View Slide

  87. UGLY BITS

    View Slide

  88. Spelling Identity Templates Content
    Editor Analysis
    Customer
    UI
    Glue
    Glue Glue Glue
    Glue
    Glue
    Glue

    View Slide

  89. Spelling Identity Templates Content
    Editor Analysis
    Customer
    UI
    Config

    View Slide

  90. ARCHITECTURE SHOULD HAVE
    RULES FOR HOW WE SHARE CODE

    View Slide

  91. Ģ
    B
    ģ
    z
    RECAP

    View Slide

  92. RULES
    ARCHITECTURE IS ABOUT
    NOT BOXES

    View Slide

  93. good architecture means never
    having to rewrite

    View Slide

  94. Thinking ahead is not about avoiding change
    Thinking ahead is about letting us change at
    different rates for different problems

    View Slide

  95. Systems as the unit of work
    Not unit of work as a system

    View Slide

  96. The thing about real systems is
    AUTONOMY

    View Slide

  97. Ģ
    B
    ģ
    z
    FIN
    @markhibberd

    View Slide

  98. References + More Info
    Stefan Tilkov's Breaking the Monolith slides video
    Coda Hale and Ryan Kennedy on "Streamie" notification service at yammer
    Slide Deck v2
    v1

    View Slide