Actor Systems in Clojure: What are your options?

[Presented @ClojureBerlin]
Wunderlist, with millions of active users, continues to grow, bending its realtime sync technology to the limits. This pushed them to rethink and experiment with different architectural approaches. In this talk Andrei will cover what they learned from Akka’s Actor System and how to achieve something similar in Clojure.

Transcript

1. Actor Systems in Clojure What are your options? Clojure Berlin 2015.12.09 #ClojureBerlin @andreiursan 1
2. WHOAMI (def me {:name "Andrei Ursan" :twitter "@andreiursan" :work "Software Engineer, @Microsoft" :builds "Wunderlist" :tags #{:polyglot :microservices :scalability}}) @andreiursan 2
3. Disclaimer The opinions presented here are my own and not my employer’s. This is my personal exploration of Clojure Actor Systems as an alternative to Akka’s Actor System. @andreiursan 3
4. Agenda • actors • real world example • Wunderlist • actors in Clojure @andreiursan 4
5. clojure.org/state Message Passing and Actors I chose not to use the Erlang-style actor model for same-process state management in Clojure for several reasons — Rich Hickey @andreiursan 5
6. clojure.org/state Message Passing and Actors ! complex programming model ! lose efficiency of being in the same process ! reduces your flexibility in modelling ! ? " eventual support for distributed programming. @andreiursan 6
7. Actors • 1973, Carl Hewitt et al.: A universal modular ACTOR formalism for artificial intelligence • 1986, Gul Agha et. al.: Actors • 1995, Ericson: first commercial use of Erlang/OTP • 2006, Erlang/OTP SMP (adds Symmetrical MultiProcessing) • 2009, Jonas Bonér: creation of Akka @andreiursan 7
8. Actors (Erlang) • are lightweight processes • a scheduler gives them “CPU time” • “communicate” via message passing • messages are processed one at a time • they are particularly good for controlling access to resources @andreiursan 8
9. Wunderlist @andreiursan 9
10. What’s behind the client? @andreiursan 10
11. @andreiursan 11
12. @andreiursan 12
13. @andreiursan 13
14. Akka • a toolkit for distributed applications • emphasises actor based concurrency • actors are async 1 • part of Scala’s standard library 1 callback hell is avoided by abstractions like Futures @andreiursan 14
15. Socket “Stage” Take 01 • play app creates WebSocketActors • an actor is created for each connection • the WebSocketActor creates two others: • for http requests • one that listens for RabbitMQ events @andreiursan 15
16. Socket “Stage” Take 01 • we had to scale… a lot. • only a limited number of rabbit connections were supported per node. @andreiursan 16
17. Socket “Stage” Take 02 • Akka DistributedPubSub2 • distributed messaging hub 2 akka.contrib.pattern.DistributedPubSubMediator @andreiursan 17
18. • this is not a complete overview of the WebSocket Server. • e.g. http requests are throttled 3 • akka includes a throttler for message delivery. Conclusion Akka is a battery included Distributed Actor System. 3 akka.contrib.throttle.TimerBasedThrottler @andreiursan 18
19. Great What are my options in the Clojure World? @andreiursan 19
20. Actors in Clojure • Akka wrappers • not complete • not maintained @andreiursan 20
21. Actors in Clojure • Akka wrappers • not complete • not maintained • puniverse/pulsar • Idiomatic Clojure wrapper around quasar @andreiursan 21
22. @andreiursan 22
23. @andreiursan 23
24. Quasar > Pulsar @andreiursan 24
25. Pulsar • like erlang • lightweight processes (a.k.a. fibers) • actors can block • gen-servers • supervisors • core.async like channels @andreiursan 25
26. Popularity Akka vs Quasar porter.io/quasar, porter.io/akka @andreiursan 26
27. Pulsar Actors @andreiursan 27
28. (defsfn actor-name [] body) @andreiursan 28
29. (defsfn actor-name [] (receive)) @andreiursan 29
30. (defsfn actor-name [] (receive) (recur)) @andreiursan 30
31. (defsfn actor-name [] (rcv @mailbox) (recur)) @andreiursan 31
32. (defsfn actor-name [] (receive pattern-one (do (action-one …) (recur)) [pattern-two msg] (do (action-two msg) (recur)) pattern-three (do (action-three …)))) @andreiursan 32
33. (defsfn actor-name [] (receive pattern-one (receive pattern-two (do (action-one …) (recur))) pattern-three (do (action-three …) (recur)) pattern-four (do (action-four …)))) @andreiursan 33
34. (require '[co.paralleluniverse.pulsar.core :refer [defsfn]]) (require '[co.paralleluniverse.pulsar.actors :refer [! receive spawn register!]]) (defsfn inc-actor [n] (receive :inc (recur (inc n)) [:inc number] (recur (+ n number)) :print (do (println n) (recur n)) :stop (println "Bye!"))) @andreiursan 34
35. (defsfn inc-actor [n] (receive :inc (recur (inc n)) [:inc number] (recur (+ n number)) :print (do (println n) (recur n)) :stop (println "Bye!"))) (register! :my-inc (spawn inc-actor 5)) (! :my-inc :inc) (! :my-inc [:inc 4]) (! :my-inc :print) ; => 10 @andreiursan 35
36. Actors and Pubsub Code Sketch @andreiursan 36
37. (defn sync [out user-id] (let [user-proxy (spawn proxy-actor user-id) web-socket (spawn web-socket-actor out user-proxy) user-events (spawn events-actor user-id web-socket pub-events)] web-socket)) @andreiursan 37
38. WebSockets? Use Comsat @andreiursan 38
39. (defsfn user-web-socket [out http-actor] (loop [] (receive [:write-message msg] (do (! out msg) (recur)) [:socket-message msg] (do (! http-actor [:request @self msg]) (recur)) :stop :stopped))) @andreiursan 39
40. (defsfn http-actor [user-id] (receive [:request user-socket msg] (! user-socket [:write-message (http-request msg)]) (recur))) (defn http-request [msg] (...)) @andreiursan 40
41. (defsfn events-actor [user-socket user-id events-pub] (let [user-events (chan) events-sub (sub events-pub user-id user-events)] (loop [] (! user-socket [:write-message (<! user-events)]) (recur)))) @andreiursan 41
42. (require '[co.paralleluniverse.pulsar.async :refer [chan <! >! pub sub]]) (def events-chan (chan)) (def events-pub (pub events-chan #{:topic %})) (defn publish-rabbit-events[] ; … (!> events-chan {:topic topic :message message})) @andreiursan 42
43. (defsfn user-web-socket [out http-actor] (loop [] (receive [:write-message msg] (do (! out msg) (recur)) [:socket-message msg] (do (! http-actor [:request @self msg]) (recur)) :stop :stopped))) (defsfn http-actor [user-id] (receive [:request user-socket msg] (! user-socket :write-message (http-request msg)) (recur))) (defsfn events-actor [user-socket user-id events-pub] (let [user-events (chan) events-sub (sub events-pub user-id user-events)] (loop [] (! user-socket [:write-message (<! user-events)]) (recur)))) @andreiursan 43
44. Testing Actors4 4 github.com/.../pulsar/actors_test.clj @andreiursan 44
45. (fact "Test actor matching receive" (let [actor (spawn #(receive :abc "yes!" :else "oy"))] (! actor :abc) (join actor)) => "yes!") @andreiursan 45
46. (fact "When matching receive and timeout then run :after clause" (let [actor (spawn #(receive [:foo] nil :else (println "got it!") :after 30 :timeout))] (Thread/sleep 150) (! actor 1) (join actor)) => :timeout) @andreiursan 46
47. Final Thoughts • actor model is a useful tool • simple resource access • fits well for some event-driven/async problems • tool diversity for the Clojure Ecosystem is GOOD. @andreiursan 47
48. Resources • The Actor Model (video) • What are fibers and why should you care? • Pulsar Docs • Web Actors: Comsat • Distributed Actors in Java and Clojure • Galaxy part 1, part 2, part 3 @andreiursan 48
49. Big thanks to Fabio! @ftudone, github/circlespainter @andreiursan 49
50. Thank you! @andreiursan 50
51. ? @andreiursan 51