Programming in the Large Again

Good architecture frees us to choose the right tools and techniques, allowing us to adapt easily and concentrate on solving real problems rather than our made up ones. In this talk we will run through some stereotypical projects looking at the properties of good architectures and how these play into our ability to adopt better tools like Scala and FP. We will attempt to ground the magical architecture ponies, with real examples of my past projects where things have gone well and probably of more interest where they really have not.

This is a slightly improved version that I presented at NICTA on 28th of November 2013.

Transcript

1. PROGRAMMING in the LARGE Ģ B ģ z @markhibberd
2. Ģ B ģ z SYSTEM FORMULATION
3. How do we arrive at Systems?
4. Modernization Old New
5. Productionization Prototype Production
6. Consolidation Bigger Small Small
7. Modularization Big Smaller Smaller
8. Greenfield Legacy
9. Incremental Base More More
10. THESE ARE NOT SYSTEMS
11. Any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure Conway’s Law
12. The rules and boundaries of our systems should stem from architecture not organisation
13. How the system interacts What is the systems purpose Macro Architecture Domain Architecture How the system is built Micro Architecture
14. Systems as the unit of work Not unit of work as a system
15. Ģ B ģ z SYSTEMS of SYSTEMS
16. The thing about real systems is AUTONOMY
17. CROSS LANGUAGE CODE SEARCH AS AN EXAMPLE
18. Code search the JVM wrong way Pick Framework
19. Scala + Play Pick Language + Framework Code search the JVM wrong way
20. UI API FILESYSTEM Code search the JVM wrong way Scala + Play
21. UI API DB INDEXER Code search the JVM wrong way Scala + Play
22. SYSTEMS, NOT SUB-SYSTEMS
23. Code search as Systems INDEXING SEARCH
24. Code search as Systems Domain Architecture INDEXING SEARCH
25. Code search as Systems INDEXING SEARCH CODE CTAGS CTAGS application/html application/search.v1+json
26. Code search as Systems INDEXING SEARCH CODE CTAGS CTAGS application/html application/search.v1+json Macro Architecture
27. STANDARD FORMATS HELP GIVE US AUTONOMY
28. Code search as Systems INDEXING SEARCH Scala Embedded Server OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging OS for services
29. Code search as Systems INDEXING SEARCH Micro Architecture Scala Embedded Server OS for services OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging
30. Code search as Systems INDEXING SEARCH Micro Architecture Scala Embedded Server OS for services OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging MORE AUTONOMY
31. RULES ARCHITECTURE IS ABOUT NOT BOXES
32. Signs of failure Systems are not autonomous Systems share a domain model Systems need to be evolved in step Systems need to be built by single team Systems share a data base
33. Systems as the unit of work If you can’t add a person to your project then you are failing worse than you think again An Aside
34. Ģ B ģ z EVOLUTION
35. Thinking ahead is not about avoiding change Thinking ahead is about letting us change at different rates for different problems
36. Indexing V2 INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static Executable OS for services OS for logging OS for services
37. INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static Executable OS for services OS for logging OS for services Micro Architecture Indexing V2
38. now vs later priorities shift over time, can't let early traction sacrifice long term speed An Aside
39. INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static Executable OS for services OS for logging OS for services Indexing V2 DIFFERENT SYSTEMS SAME RULES DIFFERENT SYSTEMS DIFFERENT RULES An Aside
40. Macro Architecture Indexing V3 INDEXING SEARCH CODE CTAGS CTAGS application/html application/search.v1+json Super-Zippy Format
41. Macro Architecture Search V2 INDEXING SEARCH CODE CTAGS Super-Zippy Format application/html application/search.v1+json Super-Zippy Format
42. Notice how easy it is to think of evolution in the large
43. Rates of change domain macro micro
44. Rates of change domain macro micro Freedom to choose tools and technology by succeeding at domain and protocols
45. HOW DO SYSTEMS THINK HOW DO SYSTEMS COMMUNICATE HOW DO SYSTEMS CHANGE
46. Implications at Scale
47. The concept of micro-services are great An Aside The concept of small libraries are great But they are not enough without thinking in systems
48. Larger Systems of Systems
49. How long does it take to get 1 line of code to production
50. Larger Systems of Systems
51. Mistaking Modules For Systems
52. Mistaking Modules For Systems
53. Mistaking Modules For Systems
54. Implication of Autonomy USERS
55. Implication of (lack of) Autonomy USERS
56. Ģ B ģ z SYSTEMS in ACTION
57. CMSs hold customer data hostage CONTENT ANALYSIS & DATA MINING PROJECT Free data Analyse data Derive data CMSs tied to horrible platforms Goals Constraints
58. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1
59. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 JSP Java
60. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 1 JSP Java
61. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller 1 2 Java 1 JSP
62. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller 1 2 1 JSP Java Java Scala
63. WHAT I IMAGINED DATA WEB CORE LEECH PORTAL CONTENT ANALYSIS & DATA MINING PROJECT
64. WEB / CORE / DATA / LEECH WEBSHERE REALITY SETTING IN
65. CAUGHT THINKING IN THE SMALL MODULES - LIBRARIES - TECH
66. We should leverage architecture changes rather than force new technology
67. CONTENT ANALYSIS IDENTITY CONTENT STORE QUERY ENGINE A Better Approach
68. Play 1.1 App Our Starting Point
69. Play 1.1 App Enabling Moves IDENTITY
70. Play 1.1 App Protobufs Thrift HTTP Establishing Rules & Autonomy sbt-assembly OS package Containers IDENTITY Protocols Deployment
71. Play 1.1 App Identifying Demand CONTENT STORE IDENTITY
72. Play 1.1 App Validation CONTENT STORE IDENTITY
73. Play 1.1 App Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS
74. Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS QUERY ENGINE
75. architecture is an everyday task
76. good architecture means never having to rewrite
77. All editors are bad, really bad ONLINE CONTENT EDITING Something aaS APIs Ready to change Small team, tight deadline Goals Constraints Lots of ideas, not sure what would work
78. Spelling Identity Templates Editor Customer UI Domain Architecture
79. Spelling Identity Templates Editor Customer UI Macro Architecture HTTP / JSON
80. C Scala Scala Spelling Identity Templates Editor Customer UI Micro Architecture Scala/JS Scala Postgres Postgres Postgres syslog jail per service rc.d 1 executable Minimal stack
81. Editor was working well ONLINE CONTENT EDITING Incremental Multiple Versions Add not modify Needed to add more services Goals Constraints System need to keep running
82. Spelling Identity Templates Content Editor Analysis Customer UI
83. Spelling Identity Templates Content Editor Analysis Customer UI HTTP / JSON
84. Spelling Identity Templates Content Editor Analysis Customer UI Scala/JS Scala JS C Scala Scala Scala multilog jail per service daemon tools 1 executable Minimal stack
85. Demand for standalone features ONLINE CONTENT EDITING Quick Minimal Overhead Maximum Reuse Not tied to our editor Goals Constraints Run in parallel to current system
86. Spelling Identity Templates Content Editor Analysis Customer UI
87. UGLY BITS
88. Spelling Identity Templates Content Editor Analysis Customer UI Glue Glue Glue Glue Glue Glue Glue
89. Spelling Identity Templates Content Editor Analysis Customer UI Config
90. ARCHITECTURE SHOULD HAVE RULES FOR HOW WE SHARE CODE
91. Ģ B ģ z RECAP
92. RULES ARCHITECTURE IS ABOUT NOT BOXES
93. good architecture means never having to rewrite
94. Thinking ahead is not about avoiding change Thinking ahead is about letting us change at different rates for different problems
95. Systems as the unit of work Not unit of work as a system
96. The thing about real systems is AUTONOMY
97. Ģ B ģ z FIN @markhibberd
98. References + More Info Stefan Tilkov's Breaking the Monolith slides video Coda Hale and Ryan Kennedy on "Streamie" notification service at yammer Slide Deck v2 v1