Powerful Data Access in Clojure

Presented at Clojure Munich Meetup

Interactive version at:
https://xsc.github.io/slides/clojure-munich/dec-2016/powerful-data-access-in-clojure/

Transcript

1. POWERFUL DATA ACCESS IN CLOJURE YANNICK SCHERER  xsc  xscys
2. THE PERFECT DATA SOURCE
3.   ?   one endpoint one shape () queriable dimensions () self-contained no unnecessary data THE PERFECT DATA SOURCE
4. (defn fetch-cubes! [datasource cube-query] ...)
5. (defn fetch-view-data! [datasource view] (->> (build-cube-query view) (fetch-cubes! datasource) (transform-for-view)))
6. REALITY!
7. A FEDERATION OF PERFECT DATA SOURCES             one endpoint per shape queriable dimensions self-contained no unnecessary data
8. (defn fetch-cubes! [datasource cube-query] ...) (defn fetch-circles! [datasource circle-query] ...) ;; ...
9. (defn fetch-view-data! [datasources view] (let [cubes (->> (build-cube-query view) (fetch-cubes! (:cube-source datasources))) circles (->> (build-circle-query view) (fetch-circles! (:circle-source datasources)))] (transform-for-view cubes circles)))
10. AN OPPORTUNITY FOR PARALLELISATION (defn fetch-view-data! [datasources view] (let [cubes (->> (build-cube-query view) (fetch-cubes! (:cube-source datasources)) (future)) circles (->> (build-circle-query view) (fetch-circles! (:circle-source datasources)) (future))] (transform-for-view @cubes @circles)))
11. MORE REALITY!
12. LINKED DATA SOURCES            one endpoint per shape queriable dimensions explicit dependencies no unnecessary data  
13. (defn fetch-view-data! [datasources view] (let [floppies (->> (build-floppy-query view) (fetch-floppies! (:floppy-source datasources))) gears (->> (extract-gear-references floppies) (build-gear-subquery view) (fetch-gears! (:gear-source datasources)))] (transform-for-view (attach-gears floppies gears))))
14. MORE LEVELS (defn fetch-view-data! [datasources view] (let [floppies ... gears ... cubes (->> (concat (extract-floppy-cube-references floppies) (extract-gear-cube-references gears)) (build-cube-subquery view) (fetch-cubes! (:cube-source datasources)))] (transform-for-view (attach-gears (attach-floppy-cubes floppies cubes) (attach-gear-cubes gears cubes)))))
15. VARIATIONS OF THE SAME SHAPE (defn fetch-view-data! [datasources view] (let [floppies ... gears ... g-cubes (->> (extract-floppy-cube-references floppies) (build-floppy-cube-subquery view) (fetch-cubes! (:cube-source datasources))) f-cubes (->> (extract-gear-cube-references gears) (build-gear-cube-subquery view) (fetch-cubes! (:cube-source datasources)))] (transform-for-view (attach-gears (attach-floppy-cubes floppies f-cubes) (attach-gear-cubes gears g-cubes)))))
16. (defn fetch-view-data! [datasources view] (let [floppies ... gears ... g-cubes (->> (extract-floppy-cube-references floppies) (build-floppy-cube-subquery view) (fetch-cubes! (:cube-source datasources)) (future)) f-cubes (->> (extract-gear-cube-references gears) (build-gear-cube-subquery view) (fetch-cubes! (:cube-source datasources)) (future))] (transform-for-view (attach-gears (attach-floppy-cubes floppies @f-cubes) (attach-gear-cubes gears @g-cubes)))))
17. THINGS YOU DON'T WANT TO CARE ABOUT Reference Locations Fetching Order/Parallelism Redundant Fetches Soundness
18. HAXL  facebook/haxl
19. A Haskell library that simpli es access to remote data, such as databases or web-based services. HAXL
20. HAXL Implicit concurrency using applicative functors. Implicit caching.
21. commonFriendsOfFriends id1 id2   concat <$> mapM friendsOf    intersect <$>  <*>    concat <$> mapM friendsOf     friendsOf id1  friendsOf id2
22. REQUEST DEFINITION data FriendReq a where FriendsOf :: Id -> FriendReq [Id] deriving (Typeable) instance DataSource u FriendReq where fetch _state _flags _userEnv blockedFetches = AsyncFetch $ \inner -> do ... friendsOf :: Id -> Haxl [Id] friendsOf id = dataFetch (FriendsOf id) commonFriendsOfFriends id1 id2 = do friends1 <- friendsOf id1 friends2 <- friendsOf id2 fofs1 <- concat <$> mapM friendsOf friends1 fofs2 <- concat <$> mapM friendsOf friends2 intersect fofs1 fofs2
23. REQUEST EXECUTION main = do env <- ... id1 <- ... id2 <- ... fofs <- runHaxl env $ commonFriendsOfFriends id1 id2 print fofs
24. MUSE & URANIA  kachayev/muse  funcool/urania ()
25. A Clojure library that works hard to make your relationship with remote data simple & enjoyable. MUSE
26. MUSE Implements functors using the cats library. Leverages protocols and records to declare data sources. Uses core.async channels for concurrency.
27. DATA SOURCES (defrecord FriendsOf [id] muse/DataSource (fetch [_] (go (set (fetch-friend-ids! id)))))
28. COMPOSITION (FMAP) (muse/fmap count (->FriendsOf id1)) (muse/fmap set/intersection (->FriendsOf id1) (->FriendsOf id2)) BIND (FLAT-MAP) (->> (->FriendsOf id1) (muse/fmap first) (muse/flat-map ->FriendsOf)) MAP (TRAVERSE) (defn friends-of-friends [id] (->> (->FriendsOf id) (muse/traverse ->FriendsOf) (muse/fmap #(reduce set/union %))))
29. THE GOOD (defn common-friends-of-friends [id1 id2] (muse/fmap set/intersection (friends-of-friends id1) (friends-of-friends id2))) (muse/run!! (common-friends-of-friends 1 2)) ;; => #{3 4 5}
30. THE BATCH (defrecord FriendsOf [id] muse/DataSource (fetch [_] (go (set (fetch-friend-ids! id)))) muse/BatchedSource (fetch-multi [_ users] (let [ids (cons id (map :id users))] (->> ids ...))))
31. THE UGLY? (defn user-and-friends-by-name [name] (muse/flat-map (fn [{:keys [id] :as user}] (muse/fmap #(assoc user :friends %) (->FriendsOf id))) (->User name)))
32. URANIA Fork of Muse. Promises instead of channels. fmap  map flat-map  mapcat Allows passing of environment to fetch function.
33. THINGS YOU DON'T WANT TO CARE ABOUT Reference Locations  Fetching Order/Parallelism  Redundant Fetches  Soundness 
34. GRAPHQL  facebook/graphql
35. GRAPHQL Describe your data. Ask for what you want. Get predictable results.
36. { commonFriends(id1: 1, id2: 2) { name, address { city } } } { "commonFriends": [ { "name": "Nobody", "address": { "city": "Nowhere" }}, ... ] }
37. { commonFriends(id1: 1, id2: 2) { name, address { city }, friends { friends { friends { name } } } } } { "commonFriends": [ { "name": "Nobody", "address": { "city": "Nowhere" }, "friends": [{ "friends": [{ "friends": [{ "name": "Who" }]}]}] }, ... ] }
38. CLARO  xsc/claro
39. A library that allows you to streamline your data access, providing powerful optimisations and abstractions along the way. CLARO
40. ANY DATASTRUCTURE IS A DATASOURCE
41. ANY DATASTRUCTURE IS A DATASOURCE: (engine/run!! {:friends1 (->FriendsOf 1) :friends2 (->FriendsOf 2)}) ;; => {:friends1 #{2 3}, :friends2 #{2 5}} COMPARE: (muse/run!! (muse/fmap #(hash-map :friends1 %1 :friends2 %2) (->FriendsOf 1) (->FriendsOf 2))) ;; => {:friends1 #{2 3}, :friends2 #{2 5}}
42. DATASOURCES CAN PRODUCE MORE DATASOURCES
43. DATASOURCES CAN PRODUCE MORE DATASOURCES: (defrecord User [id] data/Resolvable (resolve! [_ env] (d/future (when-let [user (fetch-user! (:db env) id)] (assoc user :friends (->FriendsOf (:id user)))))))  fewer (fmap ...) calls  unnecessary data‽
44. DATASOURCES CAN PRODUCE INFINITE TREES
45. DATASOURCES CAN PRODUCE INFINITE TREES: (defrecord FriendsOf [id] data/Resolvable (resolve! [_ env] (d/future (let [friends (fetch-friend-ids! (:db env) id)] (set (map #(->User %) friends)))))) (engine/run!! (->User 1)) ;; => IllegalStateException: maximum batch count exceeded (33/32).
46. TREE PROJECTIONS
47. SELECTION {:id projection/leaf :name projection/leaf :friends [{:name projection/leaf}]} (engine/run!! (projection/apply (->User 1) ...)) ;; => {:id 1, ;; :name "Someone", ;; :friends [{:name "Nobody"} {:name "NobodyElse"}]}
48. MERGE (projection/union base-user {:friends [(projection/extract :name)]}) (engine/run!! (projection/apply (->User 1) ...)) ;; => {:id 1, ;; :name "Someone", ;; :friends ["Nobody" "NobodyElse"]}  tree projections are composable!
49. TRANSFORM {:id projection/leaf :name projection/leaf :friends (projection/transform count [{}])} (engine/run!! (projection/apply (->User 1) ...)) ;; => {:id 1, ;; :name "Someone", ;; :friends 2}
50. CLARO QUE SÌ Create one rich, reusable, in nite tree of data. Select/transform the parts you need ad-hoc. (def Root {:user (map->User {}) :users (map->Users {}) ...}) (defn fetch-view-data! [view] (->> (build-projection view) (projection/apply Root) (engine/run!!)))
51.   ?   THE PERFECT DATA SOURCE
52. THANK YOU!