Maintaining consistency in distributed systems with an effect machine

In the beginning there was nothing. So the programmers said “let there be a monolith”.

And a monolith was created, but it was coupled and not scalable, and the programmers saw that it was not good.

Then they said “let there be microservices”. And microservices were created.

At first it was good, but with microservices came chaos and problems, like: consistency, testing and concurrency issues.

And the programmers thought “What if we could isolate all the side effects leveraging idempotency and functional programming to solve these issues?”.

So they said “let there be a side effect machine”. And a side effect machine was created.

And It was very good. A magic new world was out there.

Transcript

1. Maintaining consistency in distributed systems with an effect machine
2. Carolina Pascale Software Engineer @ Nubank @CarolinaPascale github.com/carolpc instagram.com/carolpc
3. Marcelo Piva Software Engineer @ Nubank @mpivaa github.com/mpivaa instagram.com/m.pivaa
4. In the beginning there was nothing.
5. So the programmers said: “let there be a monolith”
6. And a monolith was created
7. But it was coupled and not scalable, and the programmers saw that it was not good.
8. But it was coupled and not scalable, and the programmers saw that it was not good.
9. Emergency-driven context switching
10. but it was coupled and not scalable, and the programmers saw that it was not good.
11. Hard to implement new architectural designs
12. Long onboarding for the newcomer
13. Biggest source of shared state: database as a global variable
14. Conway's law
15. But it was coupled and not scalable, and the programmers saw that it was not good.
16. So they said: “let there be microservices”.
17. and microservices were created.
18. At first it was good, but with microservices came chaos and problems
20. At first it was good, but with microservices came chaos and problems
23. consistency and testing issues
24. Idempotency
25. At least once
26. Replayable
28. Avoid effects duplication
29. Pure functions
30. Order of effects matter
31. A B C
32. A B C
33. A B C ❌
34. A B C ❌
35. A B C ❌
36. ❌
37. consistency and testing issues eventual
38. A B C
39. ❌ A B C
40. Coordinating
41. db-time db-time db-time A B C sync
42. db-time db-time db-time A B C
43. consistency and testing issues
44. Services need to communicate
46. IO is hard to test
47. Unit Integration E2E
48. Isolate side-effects
49. Pure functions 2
50. What can we do?
51. First attempt: hexagonal architecture
52. Inner layer: domain model
53. Second layer: application
54. Third layer: IO
55. Imperative code and side effects on the second layer
56. Why not imperative? Harder to reason about Harder to compose Harder to test Slower feedback
57. Why not imperative? Harder to reason about Harder to compose Harder to test Slower feedback
58. Why not imperative? (block! card datomic producer) ; =>
59. Why not imperative? (defn block! [card datomic producer] (when (logic.card/can-block? card) (let [blocked-card (logic.card/block card)] (datomic.card/update! blocked-card datomic) (producer.card/status-changed! blocked-card producer))))
60. Why not imperative? Harder to reason about Harder to compose Harder to test Slower feedback
61. Why not imperative? (defn block-all! [customer cards datomic producer] (doseq [card cards] (block! card datomic producer)) (cards-blocked-notify! customer producer))
62. Why not imperative? (defn block-all! [customer cards datomic producer] (doseq [card cards] (block! card datomic producer)) (cards-blocked-notify! customer producer))
63. Why not imperative? (defn block-all! [customer cards datomic producer] (doseq [card cards] (block! card datomic producer)) (cards-blocked-notify! customer producer))
64. Why not imperative? Harder to reason about Harder to compose Harder to test Slower feedback
65. Why not imperative? Mocks & Integration Tests
67. Why not imperative? Harder to reason about Harder to compose Harder to test Slower feedback
69. Tests ❌ Consistency ❌
70. Second attempt: Free monads
71. CODE
72. CODE AST
73. CODE AST RUNNER IMPLEMENTATION
75. Tests ✅ Consistency ❌
76. Not simple
77. Not easy
79. Third attempt
80. isolate all side effects leveraging functional programming
82. (defn run-effects! [message] (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t))))
83. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
84. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
85. A deterministic ID to each message
86. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
87. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
88. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
89. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
90. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
91. so they said: “let there be an effect machine”
92. And an effect machine was created
93. Decouple decisions from effects
94. Decouple intent from execution
96. Data as first class citizens
97. {:effects/datoms [{:db/id [:card/id "some-uuid"] :card/status :status/blocked}] :effects/requests [{:url :some-url :method :post}] :effects/messages [{:topic :card-status-changed :message {:card/status :status/blocked}}]}
98. {:effects/datoms [{:db/id [:card/id "some-uuid"] :card/status :status/blocked}] :effects/requests [{:url :some-url :method :post}] :effects/messages [{:topic :card-status-changed :message {:card/status :status/blocked}}]}
99. Declarative
100. (block card) ; => [{:effects/datoms [...] ; :effects/messages [...]}] Easier to reason about
101. (s/defschema BlockCard {:messages [(s/one StatusChanged)] :datoms [(s/one BlockedCard)]}) (defn block :- (s/either t.effects/BlockCard t.effects/Empty) [card] (if (logic.card/can-block? card) (let [blocked-card (logic.card/block card)] (-> e/empty-effects (effect.card/update blocked-card) (effect.card/status-changed blocked-card)))) e/empty-effects))
102. (defn block-all [customer cards] (-> (apply e/merge (map block cards)) (effect.card/card-blocked-notify customer))) Easier to compose
103. Atomic operations
104. (deftest block-card-test (is (= {:effects/datoms [{:db/id [:card/id "id"] :card/status :card/blocked}]} :effects/messages [{:topic :card-status-changed :message {:card/status :status/blocked}}] (block card)))) Easier to test
105. (defspec block-card-always-return-valid-effects-prop (props/for-all [card gen-card] (s/validate t.effects/BlockCard (block card)))) Easier to test
106. Global dependencies
107. Global dependencies Dependency injection
108. Dependency rejection Global dependencies Dependency injection
109. (defn block [card] (if (logic.card/can-block? card) (let [blocked-card (logic.card/block card)] (-> e/empty-effects (effect.card/update blocked-card) (effect.card/status-changed blocked-card)))) e/empty-effects))
110. (defn block [card] (if (logic.card/can-block? card) (let [blocked-card (logic.card/block card)] (-> e/empty-effects (effect.card/update blocked-card) (effect.card/status-changed blocked-card)))) e/empty-effects))
111. (s/defn run-effects! [effects :- Effects+Id components] (-> effects add-idempotency-check (run-datoms! components) (run-requests! components) after-run-requests (run-messages! components)))
112. Tests ✅ Consistency ✅
114. https://t.me/clojurebrasil https://www.meetup.com/pt-BR/ClojureSP/
115. Thanks! We're hiring! @mpivaa github.com/mpivaa instagram.com/m.pivaa https://boards.greenhouse.io/nubank @CarolinaPascale github.com/carolpc instagram.com/carolpc