Programming in the Large ASWEC

Programming in the Large ASWEC

Good architecture frees us to choose the right tools and techniques, allowing us to adapt easily and concentrate on solving real problems rather than our made up ones. In this talk we will run through some stereotypical projects looking at the properties of good architectures and how these play into our ability to adopt better tools and techniques. We will attempt to ground the discussion with real examples of my past projects where things have gone well and probably of more interest where they really have not.

42d9867a0fee0fa6de6534e9df0f1e9b?s=128

Mark Hibberd

April 07, 2014
Tweet

Transcript

  1. PROGRAMMING in the LARGE Ģ B ģ z @markhibberd NICTA

  2. Ģ B ģ z SYSTEM FORMULATION

  3. How do we arrive at Systems?

  4. Modernization Old New

  5. Productionization Prototype Production

  6. Consolidation Bigger Small Small

  7. Modularization Big Smaller Smaller

  8. Greenfield Legacy

  9. Incremental Base More More

  10. THESE ARE NOT SYSTEMS

  11. Any organization that designs a system will produce a design

    whose structure is a copy of the organization's communication structure Conway’s Law
  12. The rules and boundaries of our systems should stem from

    architecture not organization
  13. HOW DO SYSTEMS THINK HOW DO SYSTEMS COMMUNICATE HOW DO

    SYSTEMS CHANGE
  14. Systems as the unit of work Not unit of work

    as a system
  15. Ģ B ģ z SYSTEMS of SYSTEMS

  16. The thing about real systems is AUTONOMY

  17. CODE SEARCH An Example

  18. Code search the normal wrong way Pick Framework

  19. UI API DB INDEXER Code search the wrong way

  20. RULES ARCHITECTURE IS ABOUT NOT BOXES

  21. Code search as Systems INDEXING SEARCH

  22. Code search as Systems INDEXING SEARCH CODE CTAGS CTAGS application/html

    application/search.v1+json
  23. STANDARD FORMATS HELP PROVIDE AUTONOMY

  24. Code search as Systems INDEXING SEARCH Scala Embedded Server OS

    for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging OS for services
  25. now vs later priorities shift over time, can't let early

    traction sacrifice long term speed An Aside
  26. Ģ B ģ z EVOLUTION

  27. is the default state LEGACY

  28. Thinking ahead is not about avoiding change Thinking ahead is

    about letting us change at different rates for different problems
  29. Rates of change domain macro micro

  30. Rates of change domain macro micro Freedom to choose tools

    and technology by succeeding at domain and protocols
  31. STANDARDS HELP AVOID CHAOS

  32. Implications at Scale

  33. Larger Systems of Systems

  34. How long does it take to get 1 line of

    code to production
  35. Larger Systems of Systems

  36. Mistaking Modules For Systems

  37. Mistaking Modules For Systems

  38. Mistaking Modules For Systems

  39. AUTONOMOUS SYSTEMS EVOLVE INDEPENDENTLY

  40. Implication of Autonomy USERS

  41. Implication of (lack of) Autonomy USERS

  42. Notice how easy it is to think of evolution in

    the large
  43. Ģ B ģ z SYSTEMS in ACTION

  44. CMSs hold customer data hostage CONTENT ANALYSIS & DATA MINING

    PROJECT Free data Analyse data Derive data CMSs tied to horrible platforms Goals Constraints
  45. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1

  46. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 JSP

    Java
  47. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 1

    JSP Java
  48. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller

    1 2 Java 1 JSP
  49. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller

    1 2 1 JSP Java Java Scala
  50. WHAT I IMAGINED DATA WEB CORE EXTRACT PORTAL CONTENT ANALYSIS

    & DATA MINING PROJECT
  51. WEB / CORE / DATA / EXTRACT WEBSHERE REALITY SETTING

    IN
  52. Signs of failure Systems are not autonomous Systems share a

    domain model Systems need to be evolved in step Systems need to be built by single team Systems share a data base
  53. Systems as the unit of work If you can’t add

    a person to your project then you are failing worse than you think again
  54. CONTENT ANALYSIS IDENTITY CONTENT STORE QUERY ENGINE A Better Approach

  55. The Big App Our Starting Point

  56. The Big App Enabling Moves IDENTITY

  57. The Big App Protobufs Thrift HTTP Establishing Rules & Autonomy

    sbt-assembly OS package Containers IDENTITY Protocols Deployment
  58. The Big App Identifying Demand CONTENT STORE IDENTITY

  59. The Big App Validation CONTENT STORE IDENTITY

  60. The Big App Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS

  61. Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS QUERY ENGINE

  62. leverage architecture changes rather than force new technology

  63. good architecture means never having to rewrite

  64. architecture is an everyday task

  65. Ģ B ģ z RECAP

  66. Systems as the unit of work Not unit of work

    as a system
  67. The thing about real systems is AUTONOMY

  68. is the default state LEGACY

  69. Ģ B ģ z FIN @markhibberd

  70. References + More Info Stefan Tilkov's Breaking the Monolith slides

    video Coda Hale and Ryan Kennedy on "Streamie" notification service at yammer Slide Deck v2 v1 v3