Programming in the Large ASWEC

Good architecture frees us to choose the right tools and techniques, allowing us to adapt easily and concentrate on solving real problems rather than our made up ones. In this talk we will run through some stereotypical projects looking at the properties of good architectures and how these play into our ability to adopt better tools and techniques. We will attempt to ground the discussion with real examples of my past projects where things have gone well and probably of more interest where they really have not.

Transcript

1. PROGRAMMING in the LARGE Ģ B ģ z @markhibberd NICTA
2. Ģ B ģ z SYSTEM FORMULATION
3. How do we arrive at Systems?
4. Modernization Old New
5. Productionization Prototype Production
6. Consolidation Bigger Small Small
7. Modularization Big Smaller Smaller
8. Greenfield Legacy
9. Incremental Base More More
10. THESE ARE NOT SYSTEMS
11. Any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure Conway’s Law
12. The rules and boundaries of our systems should stem from architecture not organization
13. HOW DO SYSTEMS THINK HOW DO SYSTEMS COMMUNICATE HOW DO SYSTEMS CHANGE
14. Systems as the unit of work Not unit of work as a system
15. Ģ B ģ z SYSTEMS of SYSTEMS
16. The thing about real systems is AUTONOMY
17. CODE SEARCH An Example
18. Code search the normal wrong way Pick Framework
19. UI API DB INDEXER Code search the wrong way
20. RULES ARCHITECTURE IS ABOUT NOT BOXES
21. Code search as Systems INDEXING SEARCH
22. Code search as Systems INDEXING SEARCH CODE CTAGS CTAGS application/html application/search.v1+json
23. STANDARD FORMATS HELP PROVIDE AUTONOMY
24. Code search as Systems INDEXING SEARCH Scala Embedded Server OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging OS for services
25. now vs later priorities shift over time, can't let early traction sacrifice long term speed An Aside
26. Ģ B ģ z EVOLUTION
27. is the default state LEGACY
28. Thinking ahead is not about avoiding change Thinking ahead is about letting us change at different rates for different problems
29. Rates of change domain macro micro
30. Rates of change domain macro micro Freedom to choose tools and technology by succeeding at domain and protocols
31. STANDARDS HELP AVOID CHAOS
32. Implications at Scale
33. Larger Systems of Systems
34. How long does it take to get 1 line of code to production
35. Larger Systems of Systems
36. Mistaking Modules For Systems
37. Mistaking Modules For Systems
38. Mistaking Modules For Systems
39. AUTONOMOUS SYSTEMS EVOLVE INDEPENDENTLY
40. Implication of Autonomy USERS
41. Implication of (lack of) Autonomy USERS
42. Notice how easy it is to think of evolution in the large
43. Ģ B ģ z SYSTEMS in ACTION
44. CMSs hold customer data hostage CONTENT ANALYSIS & DATA MINING PROJECT Free data Analyse data Derive data CMSs tied to horrible platforms Goals Constraints
45. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1
46. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 JSP Java
47. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 1 JSP Java
48. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller 1 2 Java 1 JSP
49. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller 1 2 1 JSP Java Java Scala
50. WHAT I IMAGINED DATA WEB CORE EXTRACT PORTAL CONTENT ANALYSIS & DATA MINING PROJECT
51. WEB / CORE / DATA / EXTRACT WEBSHERE REALITY SETTING IN
52. Signs of failure Systems are not autonomous Systems share a domain model Systems need to be evolved in step Systems need to be built by single team Systems share a data base
53. Systems as the unit of work If you can’t add a person to your project then you are failing worse than you think again
54. CONTENT ANALYSIS IDENTITY CONTENT STORE QUERY ENGINE A Better Approach
55. The Big App Our Starting Point
56. The Big App Enabling Moves IDENTITY
57. The Big App Protobufs Thrift HTTP Establishing Rules & Autonomy sbt-assembly OS package Containers IDENTITY Protocols Deployment
58. The Big App Identifying Demand CONTENT STORE IDENTITY
59. The Big App Validation CONTENT STORE IDENTITY
60. The Big App Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS
61. Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS QUERY ENGINE
62. leverage architecture changes rather than force new technology
63. good architecture means never having to rewrite
64. architecture is an everyday task
65. Ģ B ģ z RECAP
66. Systems as the unit of work Not unit of work as a system
67. The thing about real systems is AUTONOMY
68. is the default state LEGACY
69. Ģ B ģ z FIN @markhibberd
70. References + More Info Stefan Tilkov's Breaking the Monolith slides video Coda Hale and Ryan Kennedy on "Streamie" notification service at yammer Slide Deck v2 v1 v3