Finagle & Clojure

Finagle & Clojure

FinagleCon & Scala by the Bay 2015

B9b7a5ffa24e2af6f877a7950461ba0f?s=128

Oleksii Kachaiev

August 13, 2015
Tweet

Transcript

  1. Finagle & Clojure by Alexey Kachayev for #FinagleCon 2015

  2. About Me ‣ Alexey Kachayev, @kachayev ‣ CTO at Attendify.com

    • Almost-all-day-coding-kind-of-CTO ‣ Active open source contributor
  3. Attendify Use Case Detailed Event information. Private event community and

    social network. A fully featured event app with essential content.
  4. Attendify Use Case ‣ Private social networks for events (thousands)

    ‣ A lot of microservices both in Scala & Clojure ‣ Finagle to run services written in Scala ‣ Finagle to run services written in Clojure (???)
  5. finagle-clojure ‣ https://github.com/finagle/finagle-clojure ‣ “A thin Clojure wrapper around Finagle”

    ‣ Scala interop ‣ Bridge to Thrift/ThriftMux and more ‣ lein template (quick start)
  6. lein template $  lein  new  finagle-­‐clojure  schedule   $  tree

     -­‐L  2   .   !""  schedule          #""  README.md          #""  project.clj          #""  schedule-­‐client          #""  schedule-­‐core          !""  schedule-­‐service
  7. Thrift definition namespace  java  schedule.thrift   struct  SessionRequest  {  

       1:  string  id   }   struct  Session  {      1:  string  id      2:  string  title      3:  string  speaker   }   service  Schedule  {          Session  fetchSession(1:  SessionRequest  req)   }
  8. Service implementation (ns  schedule.service      (:import  [schedule.thrift  Schedule  Session])

         (:require  [finagle-­‐clojure.futures  :as  f]                          [finagle-­‐clojure.thrift  :as  thrift])      (:gen-­‐class))   (defn  make-­‐service      []      (thrift/service  Schedule          (fetchSession  [req]              (let  [id  (.id  req)]                  (f/value  (Session.  id  "Clojure"  "Alexey"))))))   (defn  -­‐main      [&  args]      (f/await  (thrift/serve  ":9999"  (make-­‐service))))
  9. Client implementation (ns  schedule.client      (:import  [schedule.thrift  Schedule])  

       (:require  [finagle-­‐clojure.futures  :as  f]                          [finagle-­‐clojure.thrift  :as  thrift]))   (defn  make-­‐client      [address]      (thrift/client  address  Schedule))   (defn  session-­‐title  [client  id]      (-­‐>  (.fetchSession  client  (SessionRequest.  id))              (f/map  [v]  (.title  v))))
  10. Client implementation (defn  speaker-­‐name  [client  session-­‐id]      (-­‐>  (.fetchSession

     client  (SessionRequest.  session-­‐id))              (f/flatmap  [v]                  (let  [speaker-­‐id  (.speaker  v)                              req  (SpeakerRequest.  speaker-­‐id)]                      (-­‐>  (.fetchSpeaker  client  req)                              (f/map  [v]  (.firstName  v)))))))   (defn  -­‐main  [&  args]      (let  [c  (make-­‐client  "localhost:9999")]          (println  (f/await  (session-­‐title  c  "42")))          (println  (f/await  (speaker-­‐name  c  "42")))))
  11. finagle-clojure? ‣ Scala API is not idiomatic for Clojure •

    easy to translate examples from the Internet • hard to play with the rest of the code ‣ Should we write Scala code using Clojure syntax?
  12. Everything is a Data ‣ Builder API for servers &

    clients ‣ Hash-maps instead chained mutators ‣ Hash-maps instead of HTTP Request/Response builders
  13. Not A Clojure (-­‐>  (builder-­‐server/builder)          (builder-­‐server/codec

     http-­‐codec/http)          (builder-­‐server/bind-­‐to  3000)          (builder-­‐server/named  “test”)          (builder-­‐server/build  hello-­‐world)) (new-­‐server  hello-­‐world  {:codec  http-­‐codec/http                                                    :tls  nil                                                    :max-­‐request-­‐size  200                                                    :max-­‐response-­‐size  400                                                    :bind-­‐to  3000                                                    :name  :test})
  14. Clojure (-­‐>  (builder-­‐server/builder)          (builder-­‐server/codec  http-­‐codec/http)  

           (builder-­‐server/bind-­‐to  3000)          (builder-­‐server/named  “test”)          (builder-­‐server/build  hello-­‐world)) (new-­‐server  hello-­‐world  {:codec  http-­‐codec/http                                                    :tls  nil                                                    :max-­‐request-­‐size  200                                                    :max-­‐response-­‐size  400                                                    :bind-­‐to  3000                                                    :name  :test})
  15. Scala Futures in Clojure ‣ Two branches of execution (success

    & errors) ‣ Exception-driven errors handling ‣ Hard to manage execution context ‣ There is a better way to work with chaining
  16. Clojure Futures ‣ A lot of “native” primitives: • futures,

    promises • agents • pmap, parallel reducers ‣ Clojure Futures are not that great for doing I/O
  17. Manifold ‣ https://github.com/ztellman/manifold ‣ Abstraction for event-based async programming ‣

    Streams & deferreds ‣ Deferreds: @, zip, chain, success, catch ‣ Pluggable concurrent executors
  18. Finagle & Deferred (def  req  (SessionRequest.  "42"))   (def  f1

     (.fetchSession  client  req))   (def  d4  (d/deferred))   (f/on-­‐success  f1  [v]  (d/success!  d4  v))   (f/on-­‐failure  f1  [v]  (d/error!  d4  v))   user>  @d4   user>  #object[Session[…]]   user>  @(d/chain  d4  #(.title  %))   user>  "Finagle  &  Clojure"
  19. Finagle & Manifold (defn  future-­‐>deferred  [sf]      (let  [d1

     (d/deferred)]          (f/on-­‐success  sf  [v]  (d/success!  d1  v))          (f/on-­‐failure  sf  [v]  (d/error!  d1  v))          d1))   (defn  fetch-­‐session  [c  id]      (f-­‐>d  (.fetchSession  c  (SessionRequest.  id))))   (defn  fetch-­‐speaker  [c  id]      (f-­‐>d  (.fetchSpeaker  c  (SpeakerRequest.  id))))
  20. Let-Flow (defn  speaker-­‐name  [c  session]      (let-­‐flow  [session  (fetch-­‐session

     c  session)                            sid  (.speaker  session)                            speaker  (fetch-­‐speaker  c  sid)]          (str  (.firstName  speaker)  "  "                      (.lastName  speaker))))   user>  (speaker-­‐name  client  "42")   user>  #object[Deferred]   user>  @(speaker-­‐name  client  "42")   user>  "Alexey  Kachayev"
  21. Let-Flow (defn  speaker-­‐name  [c  session]      (let-­‐flow  [session  (fetch-­‐session

     c  session)                            sid  (.speaker  session)                            speaker  (fetch-­‐speaker  c  sid)]          (str  (.firstName  speaker)  "  "                      (.lastName  speaker)))) values deferreds
  22. core.async ‣ https://github.com/clojure/core.async ‣ “Library designed to provide facilities for

    async programming and communication” ‣ CSP as a library ‣ go macro to turn async code into a state machine ‣ Widely adopted by Clojure community
  23. Finagle & core.async (defn  future-­‐>chan  [sf]      (let  [c1

     (a/chan  1)]          (f/on-­‐success  sf  [v]  (a/put!  c1  v))          (f/on-­‐failure  sf  [v]  (a/put!  c1  (e/left  v)))          c1))   (defn  fetch-­‐session  [c  id]      (f-­‐>c  (.fetchSession  c  (SessionRequest.  id))))   (defn  fetch-­‐speaker  [c  id]      (f-­‐>c  (.fetchSpeaker  c  (SpeakerRequest.  id))))
  24. Finagle & core.async (defn  speaker-­‐name  [c  session-­‐id]      (go

             (let  [session  (<!  (fetch-­‐session  c  session-­‐id))                      speaker-­‐id  (.speaker  session)                      speaker  (<!  (fetch-­‐speaker  c  speaker-­‐id))]              (str  (.firstName  speaker)  "  "                        (.lastName  speaker)))))   user>  (speaker-­‐name  client  "42")   user>  #object[ManyToManyChannel]   user>  (<!!  (speaker-­‐name  client  "42"))   user>  "Alexey  Kachayev"
  25. Finagle & core.async (defn  speaker-­‐name  [c  session-­‐id]      (go

             (let  [session  (<!  (fetch-­‐session  c  session-­‐id))                      speaker-­‐id  (.speaker  session)                      speaker  (<!  (fetch-­‐speaker  c  speaker-­‐id))]              (str  (.firstName  speaker)  "  "                        (.lastName  speaker)))))   state machine
  26. What About Server (:import  [com.twitter.util  Promise])   (defn  propagate-­‐to  [promise

     value]      (if  (e/left?  value)          (.setException  promise  (e/left-­‐value  value))          (.setValue  promise  (e/right-­‐value  value))))   (defn  chan-­‐>future  [c]      (let  [promise  (Promise.)]          (a/take!  c  (partial  propagate-­‐to  promise))          promise))
  27. What About Server (:import  [com.twitter.util  Promise])   (defn  propagate-­‐to  [promise

     value]      (if  (e/left?  value)          (.setException  promise  (e/left-­‐value  value))          (.setValue  promise  (e/right-­‐value  value))))   (defn  chan-­‐>future  [c]      (let  [promise  (Promise.)]          (a/take!  c  (partial  propagate-­‐to  promise))          promise))
  28. Finagle & core.async ‣ Clean & concise async code ‣

    A lot of built-in primitives (timeout, pub/sub etc) ‣ Compatibility with tons of Clojure libraries based on core.async ‣ Limited usage in request-reply world though
  29. Stitch → Muse ‣ Stitch is a Scala library for

    composing RPC services ‣ Created in Twitter, introduces by Jake Donham ‣ As Stitch is not open sourced… ‣ github.com/kachayev/muse ‣ EuroClojure talk about it
  30. Stitch → Muse ‣ Runs independent data fetches concurrently •

    Uses BFS to group fetches level-by-level ‣ Caches previously made fetches during execution ‣ Batches requests when applicable ‣ Uses the idea of building and interpreting AST ‣ Uses core.async to deal with concurrency
  31. More Clojure Codec ‣ Scala has Scodec & finagle-serial ‣

    Clojure has Fressian • Designed with EDN in mind • Rich set of core types & extensions • Middleware-friendly with tagged objects ‣ Working on Fressian codec right now
  32. EDN & Fressian {:id  42    :title  "Finagle  &  Clojure"

       :tracks  ["Libraries"  "Practice"]    :tags  #{"Finagle"  "Clojure"}    :speaker  {:name  "Alexey  Kachayev"}    :at  #inst  "2015-­‐08-­‐13T14:30:00"    :duration  #duration  "10s"}
  33. EDN & Fressian {:id  42    :title  "Finagle  &  Clojure"

       :tracks  ["Libraries"  "Practice"]    :tags  #{"Finagle"  "Clojure"}    :speaker  {:name  "Alexey  Kachayev"}    :at  #inst  "2015-­‐08-­‐13T14:30:00"    :duration  #duration  "10s"} tags
  34. Conclusion ‣ Finagle is great ‣ Clojure is amazing ‣

    Finagle can solve a lot of problems for Clojure ecosystem ‣ Finagle & Clojure integration still requires a lot of work
  35. Thank You! Questions?