Programming in the Large Again

42d9867a0fee0fa6de6534e9df0f1e9b?s=47 Mark Hibberd
November 27, 2013

Programming in the Large Again

Good architecture frees us to choose the right tools and techniques, allowing us to adapt easily and concentrate on solving real problems rather than our made up ones. In this talk we will run through some stereotypical projects looking at the properties of good architectures and how these play into our ability to adopt better tools like Scala and FP. We will attempt to ground the magical architecture ponies, with real examples of my past projects where things have gone well and probably of more interest where they really have not.

This is a slightly improved version that I presented at NICTA on 28th of November 2013.

42d9867a0fee0fa6de6534e9df0f1e9b?s=128

Mark Hibberd

November 27, 2013
Tweet

Transcript

  1. PROGRAMMING in the LARGE Ģ B ģ z @markhibberd

  2. Ģ B ģ z SYSTEM FORMULATION

  3. How do we arrive at Systems?

  4. Modernization Old New

  5. Productionization Prototype Production

  6. Consolidation Bigger Small Small

  7. Modularization Big Smaller Smaller

  8. Greenfield Legacy

  9. Incremental Base More More

  10. THESE ARE NOT SYSTEMS

  11. Any organization that designs a system will produce a design

    whose structure is a copy of the organization's communication structure Conway’s Law
  12. The rules and boundaries of our systems should stem from

    architecture not organisation
  13. How the system interacts What is the systems purpose Macro

    Architecture Domain Architecture How the system is built Micro Architecture
  14. Systems as the unit of work Not unit of work

    as a system
  15. Ģ B ģ z SYSTEMS of SYSTEMS

  16. The thing about real systems is AUTONOMY

  17. CROSS LANGUAGE CODE SEARCH AS AN EXAMPLE

  18. Code search the JVM wrong way Pick Framework

  19. Scala + Play Pick Language + Framework Code search the

    JVM wrong way
  20. UI API FILESYSTEM Code search the JVM wrong way Scala

    + Play
  21. UI API DB INDEXER Code search the JVM wrong way

    Scala + Play
  22. SYSTEMS, NOT SUB-SYSTEMS

  23. Code search as Systems INDEXING SEARCH

  24. Code search as Systems Domain Architecture INDEXING SEARCH

  25. Code search as Systems INDEXING SEARCH CODE CTAGS CTAGS application/html

    application/search.v1+json
  26. Code search as Systems INDEXING SEARCH CODE CTAGS CTAGS application/html

    application/search.v1+json Macro Architecture
  27. STANDARD FORMATS HELP GIVE US AUTONOMY

  28. Code search as Systems INDEXING SEARCH Scala Embedded Server OS

    for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging OS for services
  29. Code search as Systems INDEXING SEARCH Micro Architecture Scala Embedded

    Server OS for services OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging
  30. Code search as Systems INDEXING SEARCH Micro Architecture Scala Embedded

    Server OS for services OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging MORE AUTONOMY
  31. RULES ARCHITECTURE IS ABOUT NOT BOXES

  32. Signs of failure Systems are not autonomous Systems share a

    domain model Systems need to be evolved in step Systems need to be built by single team Systems share a data base
  33. Systems as the unit of work If you can’t add

    a person to your project then you are failing worse than you think again An Aside
  34. Ģ B ģ z EVOLUTION

  35. Thinking ahead is not about avoiding change Thinking ahead is

    about letting us change at different rates for different problems
  36. Indexing V2 INDEXING SEARCH Scala Embedded Server OS for logging

    Haskell Static Executable OS for services OS for logging OS for services
  37. INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static

    Executable OS for services OS for logging OS for services Micro Architecture Indexing V2
  38. now vs later priorities shift over time, can't let early

    traction sacrifice long term speed An Aside
  39. INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static

    Executable OS for services OS for logging OS for services Indexing V2 DIFFERENT SYSTEMS SAME RULES DIFFERENT SYSTEMS DIFFERENT RULES An Aside
  40. Macro Architecture Indexing V3 INDEXING SEARCH CODE CTAGS CTAGS application/html

    application/search.v1+json Super-Zippy Format
  41. Macro Architecture Search V2 INDEXING SEARCH CODE CTAGS Super-Zippy Format

    application/html application/search.v1+json Super-Zippy Format
  42. Notice how easy it is to think of evolution in

    the large
  43. Rates of change domain macro micro

  44. Rates of change domain macro micro Freedom to choose tools

    and technology by succeeding at domain and protocols
  45. HOW DO SYSTEMS THINK HOW DO SYSTEMS COMMUNICATE HOW DO

    SYSTEMS CHANGE
  46. Implications at Scale

  47. The concept of micro-services are great An Aside The concept

    of small libraries are great But they are not enough without thinking in systems
  48. Larger Systems of Systems

  49. How long does it take to get 1 line of

    code to production
  50. Larger Systems of Systems

  51. Mistaking Modules For Systems

  52. Mistaking Modules For Systems

  53. Mistaking Modules For Systems

  54. Implication of Autonomy USERS

  55. Implication of (lack of) Autonomy USERS

  56. Ģ B ģ z SYSTEMS in ACTION

  57. CMSs hold customer data hostage CONTENT ANALYSIS & DATA MINING

    PROJECT Free data Analyse data Derive data CMSs tied to horrible platforms Goals Constraints
  58. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1

  59. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 JSP

    Java
  60. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 1

    JSP Java
  61. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller

    1 2 Java 1 JSP
  62. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller

    1 2 1 JSP Java Java Scala
  63. WHAT I IMAGINED DATA WEB CORE LEECH PORTAL CONTENT ANALYSIS

    & DATA MINING PROJECT
  64. WEB / CORE / DATA / LEECH WEBSHERE REALITY SETTING

    IN
  65. CAUGHT THINKING IN THE SMALL MODULES - LIBRARIES - TECH

  66. We should leverage architecture changes rather than force new technology

  67. CONTENT ANALYSIS IDENTITY CONTENT STORE QUERY ENGINE A Better Approach

  68. Play 1.1 App Our Starting Point

  69. Play 1.1 App Enabling Moves IDENTITY

  70. Play 1.1 App Protobufs Thrift HTTP Establishing Rules & Autonomy

    sbt-assembly OS package Containers IDENTITY Protocols Deployment
  71. Play 1.1 App Identifying Demand CONTENT STORE IDENTITY

  72. Play 1.1 App Validation CONTENT STORE IDENTITY

  73. Play 1.1 App Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS

  74. Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS QUERY ENGINE

  75. architecture is an everyday task

  76. good architecture means never having to rewrite

  77. All editors are bad, really bad ONLINE CONTENT EDITING Something

    aaS APIs Ready to change Small team, tight deadline Goals Constraints Lots of ideas, not sure what would work
  78. Spelling Identity Templates Editor Customer UI Domain Architecture

  79. Spelling Identity Templates Editor Customer UI Macro Architecture HTTP /

    JSON
  80. C Scala Scala Spelling Identity Templates Editor Customer UI Micro

    Architecture Scala/JS Scala Postgres Postgres Postgres syslog jail per service rc.d 1 executable Minimal stack
  81. Editor was working well ONLINE CONTENT EDITING Incremental Multiple Versions

    Add not modify Needed to add more services Goals Constraints System need to keep running
  82. Spelling Identity Templates Content Editor Analysis Customer UI

  83. Spelling Identity Templates Content Editor Analysis Customer UI HTTP /

    JSON
  84. Spelling Identity Templates Content Editor Analysis Customer UI Scala/JS Scala

    JS C Scala Scala Scala multilog jail per service daemon tools 1 executable Minimal stack
  85. Demand for standalone features ONLINE CONTENT EDITING Quick Minimal Overhead

    Maximum Reuse Not tied to our editor Goals Constraints Run in parallel to current system
  86. Spelling Identity Templates Content Editor Analysis Customer UI

  87. UGLY BITS

  88. Spelling Identity Templates Content Editor Analysis Customer UI Glue Glue

    Glue Glue Glue Glue Glue
  89. Spelling Identity Templates Content Editor Analysis Customer UI Config

  90. ARCHITECTURE SHOULD HAVE RULES FOR HOW WE SHARE CODE

  91. Ģ B ģ z RECAP

  92. RULES ARCHITECTURE IS ABOUT NOT BOXES

  93. good architecture means never having to rewrite

  94. Thinking ahead is not about avoiding change Thinking ahead is

    about letting us change at different rates for different problems
  95. Systems as the unit of work Not unit of work

    as a system
  96. The thing about real systems is AUTONOMY

  97. Ģ B ģ z FIN @markhibberd

  98. References + More Info Stefan Tilkov's Breaking the Monolith slides

    video Coda Hale and Ryan Kennedy on "Streamie" notification service at yammer Slide Deck v2 v1