Designing a Clojure library for working with FoundationDB

This talk introduces the reader to the basic concepts of FoundationDB and takes them through the journey of building a Clojure library - https://github.com/vedang/clj_fdb - for interacting with it.

Transcript

1. Designing a Clojure library for working with FoundationDB Vedang Manerikar <2021-07-03 Sat> Bangalore Clojure Meetup Group
6. FDB is a fully ACID compliant globally lexicographical ordered KV store. 5
7. • The basic interface of a Key/Value store • ++ performant range queries (because known ordering) • ++ performant transactions (ACID!) 6
8. FDB has language bindings in Java, Python, Go (and some other languages as well, I think) 7
9. Let’s write a Clojure wrapper! How hard can it be? • It’s a KV store. • get/set/clear • Question: What’s missing here? 8
10. • STORAGE • Range Queries • Transactions 9
11. FDB itself only deals in byte-arrays, for both keys as well as values. 10
12. Question: How does the Clojure world write data to databases? 11
13. How does the Clojure world write data to databases? • Protocols (cutting edge: extend-via-metadata) • Custom Serialization (eg: nippy, transit) I looked at: • michaelklishin/monger • ptaoussanis/carmine • seancorfield/next-jdbc (and some others …) 12
14. Lesson: Review how your library handles data! 13
15. clj-fdb | Leave the serialization to the user 14
16. This simplifies the library significantly, but is probably a serious drawback. 15
17. Let’s recap the API so far 16
18. (with-open [db (cfdb/open fdb)] (fc/set db "blr-clj:20210703" "hello-world" :keyfn bs/to-byte-array :valfn bs/to-byte-array) ;; => nil (fc/get db "blr-clj:20210703" :keyfn bs/to-byte-array :valfn bs/to-string) ;; => "hello-world" (fc/clear db "blr-clj:20210703" :keyfn bs/to-byte-array)) ;; => nil 17
19. ;; The Shape of the API (fc/set db k v) ; Key/Val must be byte-array (fc/get db k) ; Key must be byte-array, returns a byte-array (fc/clear db k) ; Key must be byte-array 18
20. Performant Range Queries & The brilliant idea of layers Tuples, Subspaces and Directories 19
21. ;;; Examples of different Tuples (ftup/from "meetups") (ftup/from "meetups" "bangalore") (ftup/from "meetups" "pune") (ftup/from "meetups" "bangalore" "blr-clj") (ftup/from "meetups" "bangalore" "blr-py") (ftup/from "meetups" "pune" "pnq-clj") (ftup/from "meetups" "pune" "pnq-distsys") 20
22. (fc/set db ["meetups"] []) ;; => key Tuple("meetup") stores value empty (fc/set db ["meetups" "bangalore"] []) (fc/set db ["meetups" "pune"] []) (fc/set db ["meetups" "bangalore" "blr-clj"] []) (fc/set db ["meetups" "bangalore" "blr-py"] []) (fc/set db ["meetups" "pune" "pnq-clj"] []) (fc/set db ["meetups" "pune" "pnq-distsys"] []) 21
23. ;;; Querying ranges of Tuples (fc/get-range db ["meetups"]) ;; => {["meetups" "bangalore"] [], ;; ["meetups" "pune"] [], ;; ["meetups" "bangalore" "blr-clj"] [] ;; ["meetups" "bangalore" "blr-py"] [] ;; ["meetups" "pune" "pnq-clj"] [] ;; ["meetups" "pune" "pnq-distsys"] []} ;; Similarly: (fc/get-range db ["meetups" "bangalore"]) ;; => {["meetups" "bangalore" "blr-clj"] [] ;; ["meetups" "bangalore" "blr-py"] []} (fc/get-range db ["meetups" "pune"]) ;; => {["meetups" "pune" "pnq-clj"] [] ;; ["meetups" "pune" "pnq-distsys"] []} 22
24. Lesson: Leverage familiarity, hide complexity deep inside the interface. 23
25. ;; The Shape of the API (fc/get db k) ; Key must be byte-array / Tuple, returns byte-ar (fc/set db k v) ; Key/Val must be byte-array or Tuple (fc/clear db k) ; Key must be byte-array or Tuple (fc/get-range db k) ; Returns a map of all kvs under k (fc/clear-range db k) ; Clears all kvs under k 24
26. Subspaces are a namespace abstraction for your Tuples. 25
27. Directories are an indirection abstraction for your Tuples (and Subspaces) 26
28. Tuples, Subspaces and Directories are just different ways to create keys 27
29. ;; API evolution / support for Subspaces/Directories ;; somewhere else (def events-spc (fdir/create-or-open! db ["meetup" "events"])) (def members-spc (fdir/create-or-open! db ["meetup" "members"])) ;; in mark-attendance function (let [m-a (fc/encode members-spc [member-id "attends" event-id]) ev-a (fc/encode events-spc [event-id "attendance_counter"]) prev (to-int (fc/get db attendance))] (fc/set db m-a []) (fc/set db ev-a (to-byte-array (inc prev)))) 28
30. ;; The Shape of the API (fc/get db (fc/encode s k)) (fc/set db (fc/encode s k) v) (fc/clear db (fc/encode s k)) (fc/get-range db (fc/encode s k)) (fc/clear-range db (fc/encode s k)) 29
31. ;; Upcoming (fc/get-range db k), (fc/get-range db k opts) ;; opts contains: {:async? true :limit 10 ;; ... } ;; If opts takes up the 4th arity, having subspace in the ;; top-level becomes awkward. ;; Clojure's arglists metadata help here. (fc/get-range db k) (fc/get-range db k opts) ; These two clash, but okay (fc/get-range db s k) ; These two clash, but okay (fc/get-range db s k opts) 30
32. ;; The Shape of the API: Latest (fc/get db k), (fc/get db s k) (fc/set db k v), (fc/set db s k v) (fc/clear db k), (fc/clear db s k) (fc/get-range db k), (fc/get-range db s k) (fc/clear-range db k), (fc/clear-range db s k) 31
33. ”Different Layers, Different Abstractions” ~ John Ousterhout 32
35. ;; We saw this bit of code previously ;; Updating it for the latest API (let [prev (to-int (fc/get db attendance))] (fc/set db members-spc [member-id "attends" event-id] []) (fc/set db events-spc [event-id "attendance_counter"] (to-byte-array (inc prev)))) 34
36. ;;; Running the full code in a single transation: (ftr/run db (fn [tr] (let [prev (to-int (fc/get tr attendance))] (fc/set tr members-spc [member-id "attends" event-id] []) (fc/set tr events-spc [event-id "attendance_counter"] (to-byte-array (inc prev)))))) ;; Done. 35
37. (defn mark-attendance [db member-id event-id] (let [prev (to-int (fc/get db attendance))] (fc/set db members-spc [member-id "attends" event-id] []) (fc/set db events-spc [event-id "attendance_counter"] (to-byte-array (inc prev))))) ;; ... somewhere in our code ... (ftr/run db (fn [tr] (mark-attendance tr member-id event-id))) 36
38. Composing multiple transactional functions just works! 37
39. (defn signup-student [db sid cid] (ftr/run db (fn [tr] (let [seats (to-int (fc/get tr cspc ["class" cid]))] (fc/set tr cspc ["attends" cid sid] []) (fc/set tr cspc ["attends" sid cid] []) (fc/set tr cspc ["class" cid] (to-byte-array (dec seats))))))) ;; (signup-student db "S1" "C10") => nil 38
40. (defn drop-student [db sid cid] (ftr/run db (fn [tr] (let [seats (to-int (fc/get tr cspc ["class" cid]))] (fc/clear tr cspc ["attends" cid sid]) (fc/clear tr cspc ["attends" sid cid]) (fc/set tr cspc ["class" cid] (to-byte-array (inc seats))))))) ;; (drop-student db "S1" "C10") => nil 39
41. (defn switch-classes [db sid old-cid new-cid] (ftr/run db (fn [tr] (drop-student tr sid old-cid) (signup-student tr sid new-cid)))) ;; (switch-student db "S1" "C10" "C11") => nil 40
42. ;;; Different ways to run transactions ;; Blocking, returns the result of the function (ftr/run db (fn [tr])) ;; Blocking, only accepts reads on DB keys (ftr/read db (fn [tr])) ;; Non-blocking, immediately returns future (ftr/run-async! db (fn [tr])) ;; Non-blocking, immediately returns future (ftr/read-async! db (fn [tr])) 41
43. Currently thinking about: dealing gracefully with async operations • handling errors • avoiding callbacks • core.async! 42
44. ;;; The Shape of the API ;; Core [fc/get, fc/set, fc/clear, fc/get-range, fc/clear-range, fc/mutate!] ;; Transactions [ftr/run, ftr/read, ftr/run-async!, ftr/read-async!] ;; Directories, Subspaces, Tuples [fdir/create-or-open!, fdir/remove!, fdir/list fsub/create, fsub/range ftup/create, ftup/range] 43
45. Features I need to add to the library • Watches: Unblocked after async • Versionstamps: Need to understand this better 44
46. When choosing a new library • List down what tasks you expect to accomplish 45
47. When choosing a new library • List down what tasks you expect to accomplish • Look at the API. How deep is it? 45
48. When choosing a new library • List down what tasks you expect to accomplish • Look at the API. How deep is it? • Look at the API. How much magic is it? 45
49. When choosing a new library • List down what tasks you expect to accomplish • Look at the API. How deep is it? • Look at the API. How much magic is it? • When you need it, can you drop down easily? 45
50. A Big Thanks to Jan Rychter for discussing FDB with me and sharing his code! Talk to me about: 1. This Library: Use v0.3.0 and above • vedang/clj_fdb 2. Data modeling using FDB 3. Design patterns. 4. Working at Helpshift! mailto:ved.manerikar@gmail.com / @vedang on Twitter 46