Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cheating Gall's Law

Cheating Gall's Law

How the npm registry rewrite cheated Gall's Law and split a monolith into a set of microservices. Delivered at WebRebels 2015.


C J Silverio

May 22, 2015


  1. Cheating Gall's Law

  2. C J Silverio director of engineering, @ceejbot

  3. Cheating Gall's Law How we split a monolith and lived

    to tell the tale
  4. Gall's Law A complex system that works is invariably found

    to have evolved from a simple system that worked.
  5. None
  6. monolith everything in one process

  7. None
  8. npm's monolith: embedded in couchdb

  9. monoliths work just fine

  10. whatever it takes to build a system that satisfies your

  11. success! now scale it.

  12. None
  13. scaling monoliths many copies of the full thing

  14. None
  15. Exponential growth of node resulted in exponential growth of the

    npm registry
  16. exponential monoliths were going to be expensive

  17. splitting the monolith

  18. yay microservices?

  19. just rewrite everything! what's the problem?

  20. Your monolith is complex. A split system is more complex.

  21. what did that Gall guy say about complex working systems?

  22. Gall's Law in full “A complex system that works is

    invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”
  23. Systemantics: How Systems Really Work and How They Fail

  24. "A simple system may or may not work."

  25. "If a system is working, leave it alone. Don't change

  26. Gall's Law warns us about that rewrite

  27. how do you split a monolith successfully?

  28. Let's cheat.

  29. Q: How do you cheat? A: By not rewriting the

    whole thing.
  30. slice off a part of the system into a module

    with a clearly-defined interface
  31. then write a second implementation of that interface

  32. send requests to that second implementation with a proxy

  33. None
  34. None
  35. None
  36. None
  37. now let's make our proxy smart Replace Varnish with a

    node service
  38. None
  39. None
  40. first step just send everything through

  41. record what the proxy does how do people use your

    service? measure performance
  42. second step divide & conquer

  43. None
  44. None
  45. beat Gall's Law with modularity aka information hiding

  46. each service is a module a simple, testable system when

    viewed in isolation
  47. best part: now you can change everything

  48. your platform your database

  49. None
  50. None
  51. why stop there?

  52. None
  53. None
  54. the awesome

  55. Each piece is a simple system

  56. more scaling dials to turn finer-grained perf data

  57. isolating tasks let us debug even without splitting the monolith

  58. the system overall is complex but loosely coupled

  59. you have a working system every step of the way

  60. a proxy service lets you flip back & forth between

  61. the pitfalls

  62. now you've got a distributed system

  63. @architectclippy says “I see you have a poorly structured monolith.

    Would you like me to convert it into a poorly structured set of microservices?”
  64. Second system syndrome "this time we'll do it right"

  65. Generalize only when forced

  66. let's recap! how do you survive a rewrite?

  67. simple working service first scale it later

  68. respect Gall: rewrite in pieces

  69. use a proxy to divide & conquer

  70. Be bold you can change your system

  71. Nobody noticed when we changed the entire npm registry

  72. npm loves you npm install -g npm@latest