Programming in the Large

Good architecture frees us to choose the right tools and techniques, allowing us to adapt easily and concentrate on solving real problems rather than our made up ones. In this talk we will run through some stereotypical projects looking at the properties of good architectures and how these play into our ability to adopt better tools like Scala and FP. We will attempt to ground the magical architecture ponies, with real examples of my past projects where things have gone well and probably of more interest where they really have not.

Transcript

1. PROGRAMMING in the LARGE Ģ B ģ z @markhibberd
2. Ģ B ģ z SYSTEM FORMULATION
3. How do we arrive at Systems?
4. Modernization Old New
5. Productionization Prototype Production
6. Consolidation Bigger Small Small
7. Modularization Big Smaller Smaller
8. Greenfield Legacy
9. Incremental Base More More
10. THESE ARE NOT SYSTEMS
11. Any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure Conway’s Law
12. The rules and boundaries of our systems should stem from architecture not organisation
13. How the system interacts What is the systems purpose Macro Architecture Domain Architecture How the system is built Micro Architecture
14. Systems as the unit of work Not unit of work as a system
15. Ģ B ģ z SYSTEMS of SYSTEMS
16. The thing about real systems is AUTONOMY
17. CROSS LANGUAGE CODE SEARCH AS AN EXAMPLE
18. Code search the JVM wrong way Pick Framework
19. Scala + Play Pick Language + Framework Code search the JVM wrong way
20. UI API FILESYSTEM Code search the JVM wrong way Scala + Play
21. UI API DB INDEXER Code search the JVM wrong way Scala + Play
22. SYSTEMS, NOT SUB-SYSTEMS
23. Code search as Systems INDEXING SEARCH
24. Code search as Systems Domain Architecture INDEXING SEARCH
25. Code search as Systems INDEXING SEARCH CODE CTAGS CTAGS application/html application/search.v1+json
26. Code search as Systems INDEXING SEARCH CODE CTAGS CTAGS application/html application/search.v1+json Macro Architecture
27. STANDARD FORMATS HELP GIVE US AUTONOMY
28. Code search as Systems INDEXING SEARCH Scala Embedded Server OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging OS for services
29. Code search as Systems INDEXING SEARCH Micro Architecture Scala Embedded Server OS for services OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging
30. Code search as Systems INDEXING SEARCH Micro Architecture Scala Embedded Server OS for services OS for logging Bourne Shell Exuberant CTAGS Deploy as git hook OS for logging MORE AUTONOMY
31. Signs of failure Systems are not autonomous Systems share a domain model Systems need to be evolved in step Systems need to be built by single team Systems share a data base
32. The concept of micro-services are great An Aside The concept of small libraries are great But they are not enough without thinking in systems
33. Ģ B ģ z EVOLUTION
34. Thinking ahead is not about avoiding change Thinking ahead is about letting us change at different rates for different problems
35. Indexing V2 INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static Executable OS for services OS for logging OS for services
36. INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static Executable OS for services OS for logging OS for services Micro Architecture Indexing V2
37. now vs later priorities shift over time, can't let early traction sacrifice long term speed An Aside
38. INDEXING SEARCH Scala Embedded Server OS for logging Haskell Static Executable OS for services OS for logging OS for services Indexing V2 DIFFERENT SYSTEMS SAME RULES DIFFERENT SYSTEMS DIFFERENT RULES An Aside
39. Macro Architecture Indexing V3 INDEXING SEARCH CODE CTAGS CTAGS application/html application/search.v1+json Super-Zippy Format
40. Macro Architecture Search V2 INDEXING SEARCH CODE CTAGS Super-Zippy Format application/html application/search.v1+json Super-Zippy Format
41. Notice how easy it is to think of evolution in the large
42. Rates of change domain macro micro
43. Rates of change domain macro micro Freedom to choose tools and technology by nailing domain and protocols
44. Implications at Scale
45. Larger Systems of Systems
46. How long does it take to get 1 line of code to production
47. Larger Systems of Systems
48. Mistaking Modules For Systems
49. Systems as the unit of work If you can’t add a person to your project then you are failing worse than you think again An Aside
50. Ģ B ģ z SYSTEMS in ACTION
51. CMSs hold customer data hostage CONTENT ANALYSIS & DATA MINING PROJECT Free data Analyse data Derive data CMSs tied to horrible platforms Goals Constraints
52. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1
53. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 JSP Java
54. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production 1 1 JSP Java
55. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller 1 2 Java 1 JSP
56. CONTENT ANALYSIS & DATA MINING PROJECT Prototype Production Smaller Smaller 1 2 1 JSP Java Java Scala
57. WHAT I IMAGINED DATA WEB CORE LEECH PORTAL CONTENT ANALYSIS & DATA MINING PROJECT
58. WEB / CORE / DATA / LEECH WEBSHERE REALITY SETTING IN
59. CAUGHT THINKING IN THE SMALL MODULES - LIBRARIES - TECH
60. We should leverage architecture changes rather than force new technology
61. CONTENT ANALYSIS IDENTITY CONTENT STORE QUERY ENGINE A Better Approach
62. Play 1.1 App Our Starting Point
63. Play 1.1 App Enabling Moves IDENTITY
64. Play 1.1 App Protobufs Thrift HTTP Establishing Rules & Autonomy sbt-assembly OS package Containers IDENTITY Protocols Deployment
65. Play 1.1 App Identifying Demand CONTENT STORE IDENTITY
66. Play 1.1 App Validation CONTENT STORE IDENTITY
67. Play 1.1 App Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS
68. Repeat CONTENT STORE IDENTITY CONTENT ANALYSIS QUERY ENGINE
69. architecture is an everyday task
70. All editors are bad, really bad ONLINE CONTENT EDITING Something aaS APIs Ready to change Small team, tight deadline Goals Constraints Lots of ideas, not sure what would work
71. Spelling Identity Templates Editor Customer UI Domain Architecture
72. Spelling Identity Templates Editor Customer UI Macro Architecture HTTP / JSON
73. C Scala Scala Spelling Identity Templates Editor Customer UI Micro Architecture Scala/JS Scala Postgres Postgres Postgres syslog jail per service rc.d 1 executable Minimal stack
74. Editor was working well ONLINE CONTENT EDITING Incremental Multiple Versions Add not modify Needed to add more services Goals Constraints System need to keep running
75. Spelling Identity Templates Content Editor Analysis Customer UI
76. Spelling Identity Templates Content Editor Analysis Customer UI HTTP / JSON
77. Spelling Identity Templates Content Editor Analysis Customer UI Scala/JS Scala JS C Scala Scala Scala multilog jail per service daemon tools 1 executable Minimal stack
78. Demand for standalone features ONLINE CONTENT EDITING Quick Minimal Overhead Maximum Reuse Not tied to our editor Goals Constraints Run in parallel to current system
79. Spelling Identity Templates Content Editor Analysis Customer UI
80. UGLY BITS
81. Spelling Identity Templates Content Editor Analysis Customer UI Glue Glue Glue Glue Glue Glue Glue
82. Spelling Identity Templates Content Editor Analysis Customer UI Config
83. Ģ B ģ z FIN @markhibberd
84. References + More Info Stefan Tilkov's Breaking the Monolith slides video Coda Hale and Ryan Kennedy on "Streamie" notification service at yammer