Cheating Gall's Law

How the npm registry rewrite cheated Gall's Law and split a monolith into a set of microservices. Delivered at WebRebels 2015.

Transcript

1. Cheating Gall's Law
2. C J Silverio director of engineering, @ceejbot
3. Cheating Gall's Law How we split a monolith and lived to tell the tale
4. Gall's Law A complex system that works is invariably found to have evolved from a simple system that worked.
6. monolith everything in one process
8. npm's monolith: embedded in couchdb
9. monoliths work just fine
10. whatever it takes to build a system that satisfies your users
11. success! now scale it.
13. scaling monoliths many copies of the full thing
15. Exponential growth of node resulted in exponential growth of the npm registry
16. exponential monoliths were going to be expensive
17. splitting the monolith
18. yay microservices?
19. just rewrite everything! what's the problem?
20. Your monolith is complex. A split system is more complex.
21. what did that Gall guy say about complex working systems?
22. Gall's Law in full “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”
23. Systemantics: How Systems Really Work and How They Fail
24. "A simple system may or may not work."
25. "If a system is working, leave it alone. Don't change anything."
26. Gall's Law warns us about that rewrite
27. how do you split a monolith successfully?
28. Let's cheat.
29. Q: How do you cheat? A: By not rewriting the whole thing.
30. slice off a part of the system into a module with a clearly-defined interface
31. then write a second implementation of that interface
32. send requests to that second implementation with a proxy
37. now let's make our proxy smart Replace Varnish with a node service
40. first step just send everything through
41. record what the proxy does how do people use your service? measure performance
42. second step divide & conquer
45. beat Gall's Law with modularity aka information hiding
46. each service is a module a simple, testable system when viewed in isolation
47. best part: now you can change everything
48. your platform your database
51. why stop there?
54. the awesome
55. Each piece is a simple system
56. more scaling dials to turn finer-grained perf data
57. isolating tasks let us debug even without splitting the monolith
58. the system overall is complex but loosely coupled
59. you have a working system every step of the way
60. a proxy service lets you flip back & forth between implementations
61. the pitfalls
62. now you've got a distributed system
63. @architectclippy says “I see you have a poorly structured monolith. Would you like me to convert it into a poorly structured set of microservices?”
64. Second system syndrome "this time we'll do it right"
65. Generalize only when forced
66. let's recap! how do you survive a rewrite?
67. simple working service first scale it later
68. respect Gall: rewrite in pieces
69. use a proxy to divide & conquer
70. Be bold you can change your system
71. Nobody noticed when we changed the entire npm registry
72. npm loves you npm install -g npm@latest