Programming in the Large ASWEC

Good architecture frees us to choose the right tools and techniques, allowing us to adapt easily and concentrate on solving real problems rather than our made up ones. In this talk we will run through some stereotypical projects looking at the properties of good architectures and how these play into our ability to adopt better tools and techniques. We will attempt to ground the discussion with real examples of my past projects where things have gone well and probably of more interest where they really have not.

Transcript

1. PROGRAMMING
   in the
   LARGE
   Ģ
   B
   ģ
   z
   @markhibberd
   NICTA
   

   View Slide

2. Ģ
   B
   ģ
   z
   SYSTEM
   FORMULATION
   

   View Slide

3. How do we arrive at
   Systems?
   

   View Slide

4. Modernization
   Old New
   

   View Slide

5. Productionization
   Prototype Production
   

   View Slide

6. Consolidation
   Bigger
   Small
   Small
   

   View Slide

7. Modularization
   Big
   Smaller
   Smaller
   

   View Slide

8. Greenfield
   Legacy
   

   View Slide

9. Incremental
   Base
   More More
   

   View Slide

10. THESE ARE NOT
    SYSTEMS
    

    View Slide

11. Any organization that designs a
    system will produce a design whose
    structure is a copy of the organization's
    communication structure
    Conway’s Law
    

    View Slide

12. The rules and boundaries
    of our systems should
    stem from architecture
    not organization
    

    View Slide

13. HOW DO SYSTEMS THINK
    HOW DO SYSTEMS COMMUNICATE
    HOW DO SYSTEMS CHANGE
    

    View Slide

14. Systems as the unit of work
    Not unit of work as a system
    

    View Slide

15. Ģ
    B
    ģ
    z
    SYSTEMS
    of
    SYSTEMS
    

    View Slide

16. The thing about real systems is
    AUTONOMY
    

    View Slide

17. CODE SEARCH
    An Example
    

    View Slide

18. Code search the normal wrong way
    Pick Framework
    

    View Slide

19. UI API
    DB INDEXER
    Code search the wrong way
    

    View Slide

20. RULES
    ARCHITECTURE IS ABOUT
    NOT BOXES
    

    View Slide

21. Code search as Systems
    INDEXING SEARCH
    

    View Slide

22. Code search as Systems
    INDEXING SEARCH
    CODE
    CTAGS
    CTAGS
    application/html
    application/search.v1+json
    

    View Slide

23. STANDARD FORMATS HELP
    PROVIDE AUTONOMY
    

    View Slide

24. Code search as Systems
    INDEXING SEARCH
    Scala
    Embedded Server
    OS for logging
    Bourne Shell
    Exuberant CTAGS
    Deploy as git hook
    OS for logging
    OS for services
    

    View Slide

25. now vs later
    priorities shift over time,
    can't let early traction
    sacrifice long term speed
    An Aside
    

    View Slide

26. Ģ
    B
    ģ
    z
    EVOLUTION
    

    View Slide

27. is the default state
    LEGACY
    

    View Slide

28. Thinking ahead is not about avoiding change
    Thinking ahead is about letting us change at
    different rates for different problems
    

    View Slide

29. Rates of change
    domain
    macro
    micro
    

    View Slide

30. Rates of change
    domain
    macro
    micro
    Freedom to choose tools and technology by succeeding at domain and protocols
    

    View Slide

31. STANDARDS HELP
    AVOID CHAOS
    

    View Slide

32. Implications at
    Scale
    

    View Slide

33. Larger Systems of Systems
    

    View Slide

34. How long does it take to get 1 line
    of code to production
    

    View Slide

35. Larger Systems of Systems
    

    View Slide

36. Mistaking Modules For Systems
    

    View Slide

37. Mistaking Modules For Systems
    

    View Slide

38. Mistaking Modules For Systems
    

    View Slide

39. AUTONOMOUS SYSTEMS
    EVOLVE INDEPENDENTLY
    

    View Slide

40. Implication of Autonomy
    USERS
    

    View Slide

41. Implication of (lack of) Autonomy
    USERS
    

    View Slide

42. Notice how easy it is to think of
    evolution in the large
    

    View Slide

43. Ģ
    B
    ģ
    z
    SYSTEMS
    in
    ACTION
    

    View Slide

44. CMSs hold customer data hostage
    CONTENT ANALYSIS & DATA MINING PROJECT
    Free data
    Analyse data
    Derive data
    CMSs tied to horrible platforms
    Goals
    Constraints
    

    View Slide

45. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    1
    

    View Slide

46. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    1
    JSP Java
    

    View Slide

47. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    1
    1 JSP Java
    

    View Slide

48. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    Smaller Smaller
    1
    2
    Java
    1 JSP
    

    View Slide

49. CONTENT ANALYSIS & DATA MINING PROJECT
    Prototype Production
    Smaller Smaller
    1
    2
    1 JSP Java
    Java
    Scala
    

    View Slide

50. WHAT I IMAGINED
    DATA
    WEB
    CORE EXTRACT PORTAL
    CONTENT ANALYSIS & DATA MINING PROJECT
    

    View Slide

51. WEB / CORE / DATA / EXTRACT
    WEBSHERE
    REALITY SETTING IN
    

    View Slide

52. Signs of failure
    Systems are not autonomous
    Systems share a domain model
    Systems need to be evolved in step
    Systems need to be built by single team
    Systems share a data base
    

    View Slide

53. Systems as the unit of work
    If you can’t add a person to
    your project then you are
    failing worse than you think
    again
    

    View Slide

54. CONTENT
    ANALYSIS
    IDENTITY
    CONTENT
    STORE
    QUERY
    ENGINE
    A Better Approach
    

    View Slide

55. The Big App
    Our Starting Point
    

    View Slide

56. The Big App
    Enabling Moves
    IDENTITY
    

    View Slide

57. The Big App
    Protobufs
    Thrift
    HTTP
    Establishing Rules & Autonomy
    sbt-assembly
    OS package
    Containers
    IDENTITY
    Protocols
    Deployment
    

    View Slide

58. The Big App
    Identifying Demand
    CONTENT
    STORE
    IDENTITY
    

    View Slide

59. The Big App
    Validation
    CONTENT
    STORE
    IDENTITY
    

    View Slide

60. The Big App
    Repeat
    CONTENT
    STORE
    IDENTITY
    CONTENT
    ANALYSIS
    

    View Slide

61. Repeat
    CONTENT
    STORE
    IDENTITY
    CONTENT
    ANALYSIS
    QUERY
    ENGINE
    

    View Slide

62. leverage architecture changes rather
    than force new technology
    

    View Slide

63. good architecture means never
    having to rewrite
    

    View Slide

64. architecture is an everyday task
    

    View Slide

65. Ģ
    B
    ģ
    z
    RECAP
    

    View Slide

66. Systems as the unit of work
    Not unit of work as a system
    

    View Slide

67. The thing about real systems is
    AUTONOMY
    

    View Slide

68. is the default state
    LEGACY
    

    View Slide

69. Ģ
    B
    ģ
    z
    FIN
    @markhibberd
    

    View Slide

70. References + More Info
    Stefan Tilkov's Breaking the Monolith slides video
    Coda Hale and Ryan Kennedy on "Streamie" notification service at yammer
    Slide Deck v2
    v1 v3
    

    View Slide