$30 off During Our Annual Pro Sale. View Details »

Programming in the Large ASWEC

Programming in the Large ASWEC

Good architecture frees us to choose the right tools and techniques, allowing us to adapt easily and concentrate on solving real problems rather than our made up ones. In this talk we will run through some stereotypical projects looking at the properties of good architectures and how these play into our ability to adopt better tools and techniques. We will attempt to ground the discussion with real examples of my past projects where things have gone well and probably of more interest where they really have not.

Mark Hibberd

April 07, 2014
Tweet

More Decks by Mark Hibberd

Other Decks in Programming

Transcript

  1. PROGRAMMING
    in the
    LARGE
    Ģ
    B
    ģ
    z
    @markhibberd
    NICTA

    View Slide

  2. Ģ
    B
    ģ
    z
    SYSTEM
    FORMULATION

    View Slide

  3. How do we arrive at
    Systems?

    View Slide

  4. Modernization
    Old New

    View Slide

  5. Productionization
    Prototype Production

    View Slide

  6. Consolidation
    Bigger
    Small
    Small

    View Slide

  7. Modularization
    Big
    Smaller
    Smaller

    View Slide

  8. Greenfield
    Legacy

    View Slide

  9. Incremental
    Base
    More More

    View Slide

  10. THESE ARE NOT
    SYSTEMS

    View Slide

  11. Any organization that designs a
    system will produce a design whose
    structure is a copy of the organization's
    communication structure
    Conway’s Law

    View Slide

  12. The rules and boundaries
    of our systems should
    stem from architecture
    not organization

    View Slide

  13. HOW DO SYSTEMS THINK
    HOW DO SYSTEMS COMMUNICATE
    HOW DO SYSTEMS CHANGE

    View Slide

  14. Systems as the unit of work
    Not unit of work as a system

    View Slide

  15. Ģ
    B
    ģ
    z
    SYSTEMS
    of
    SYSTEMS

    View Slide

  16. The thing about real systems is
    AUTONOMY

    View Slide

  17. CODE SEARCH
    An Example

    View Slide

  18. Code search the normal wrong way
    Pick Framework

    View Slide

  19. UI API
    DB INDEXER
    Code search the wrong way

    View Slide

  20. RULES
    ARCHITECTURE IS ABOUT
    NOT BOXES

    View Slide

  21. Code search as Systems
    INDEXING SEARCH

    View Slide

  22. Code search as Systems
    INDEXING SEARCH
    CODE
    CTAGS
    CTAGS
    application/html
    application/search.v1+json

    View Slide

  23. STANDARD FORMATS HELP
    PROVIDE AUTONOMY

    View Slide

  24. Code search as Systems
    INDEXING SEARCH
    Scala
    Embedded Server
    OS for logging
    Bourne Shell
    Exuberant CTAGS
    Deploy as git hook
    OS for logging
    OS for services

    View Slide

  25. now vs later
    priorities shift over time,
    can't let early traction
    sacrifice long term speed
    An Aside

    View Slide

  26. Ģ
    B
    ģ
    z
    EVOLUTION

    View Slide

  27. is the default state
    LEGACY

    View Slide

  28. Thinking ahead is not about avoiding change
    Thinking ahead is about letting us change at
    different rates for different problems

    View Slide

  29. Rates of change
    domain
    macro
    micro

    View Slide

  30. Rates of change
    domain
    macro
    micro
    Freedom to choose tools and technology by succeeding at domain and protocols

    View Slide

  31. STANDARDS HELP
    AVOID CHAOS

    View Slide

  32. Implications at
    Scale

    View Slide

  33. Larger Systems of Systems

    View Slide

  34. How long does it take to get 1 line
    of code to production

    View Slide

  35. Larger Systems of Systems

    View Slide

  36. Mistaking Modules For Systems

    View Slide

  37. Mistaking Modules For Systems

    View Slide

  38. Mistaking Modules For Systems

    View Slide

  39. AUTONOMOUS SYSTEMS
    EVOLVE INDEPENDENTLY

    View Slide

  40. Implication of Autonomy
    USERS

    View Slide

  41. Implication of (lack of) Autonomy
    USERS

    View Slide

  42. Notice how easy it is to think of
    evolution in the large

    View Slide

  43. Ģ
    B
    ģ
    z
    SYSTEMS
    in
    ACTION

    View Slide

  44. CMSs hold customer data hostage
    CONTENT ANALYSIS & DATA MINING PROJECT
    Free data
    Analyse data
    Derive data
    CMSs tied to horrible platforms
    Goals
    Constraints

    View Slide

  45. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    1

    View Slide

  46. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    1
    JSP Java

    View Slide

  47. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    1
    1 JSP Java

    View Slide

  48. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    Smaller Smaller
    1
    2
    Java
    1 JSP

    View Slide

  49. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    Smaller Smaller
    1
    2
    1 JSP Java
    Java
    Scala

    View Slide

  50. WHAT I IMAGINED
    DATA
    WEB
    CORE EXTRACT PORTAL
    CONTENT ANALYSIS & DATA MINING PROJECT

    View Slide

  51. WEB / CORE / DATA / EXTRACT
    WEBSHERE
    REALITY SETTING IN

    View Slide

  52. Signs of failure
    Systems are not autonomous
    Systems share a domain model
    Systems need to be evolved in step
    Systems need to be built by single team
    Systems share a data base

    View Slide

  53. Systems as the unit of work
    If you can’t add a person to
    your project then you are
    failing worse than you think
    again

    View Slide

  54. CONTENT
    ANALYSIS
    IDENTITY
    CONTENT
    STORE
    QUERY
    ENGINE
    A Better Approach

    View Slide

  55. The Big App
    Our Starting Point

    View Slide

  56. The Big App
    Enabling Moves
    IDENTITY

    View Slide

  57. The Big App
    Protobufs
    Thrift
    HTTP
    Establishing Rules & Autonomy
    sbt-assembly
    OS package
    Containers
    IDENTITY
    Protocols
    Deployment

    View Slide

  58. The Big App
    Identifying Demand
    CONTENT
    STORE
    IDENTITY

    View Slide

  59. The Big App
    Validation
    CONTENT
    STORE
    IDENTITY

    View Slide

  60. The Big App
    Repeat
    CONTENT
    STORE
    IDENTITY
    CONTENT
    ANALYSIS

    View Slide

  61. Repeat
    CONTENT
    STORE
    IDENTITY
    CONTENT
    ANALYSIS
    QUERY
    ENGINE

    View Slide

  62. leverage architecture changes rather
    than force new technology

    View Slide

  63. good architecture means never
    having to rewrite

    View Slide

  64. architecture is an everyday task

    View Slide

  65. Ģ
    B
    ģ
    z
    RECAP

    View Slide

  66. Systems as the unit of work
    Not unit of work as a system

    View Slide

  67. The thing about real systems is
    AUTONOMY

    View Slide

  68. is the default state
    LEGACY

    View Slide

  69. Ģ
    B
    ģ
    z
    FIN
    @markhibberd

    View Slide

  70. References + More Info
    Stefan Tilkov's Breaking the Monolith slides video
    Coda Hale and Ryan Kennedy on "Streamie" notification service at yammer
    Slide Deck v2
    v1 v3

    View Slide