DataScript for Web Development

DataScript is an open-source implementation of Datalog and Datomic in-memory storage, in ClojureScript, for browser. Here I talk about motivation behind DataScript, how it was created, how it can be used, what kind of problems it solves and how simple and sane web app development is when you have a database in your browser.

Transcript

1. DataScript for Web Development
2. Nikita Prokopov tonsky.me Novosibirsk, Russia
4. Server-side traditionally benefits from relational storage + queries Rationale
5. Why web developers should limit themselves?
6. Datomic model feels right and was within my reach
7. DataScript is a post-modern DB: someone else’s good idea redone on a more primitive level
8. <entity, attribute, value> triple store What’s DataScript?
9. Database as an immutable value
10. Completely in-memory
11. Datalog queries over it
12. Tries to mimic Datomic API
13. Written in ClojureScript, JS bindings available
14. (def conn (d/create-conn {:aka {:db/cardinality :db.cardinality/many }})] (d/transact! conn [ {:db/id 1 :name “Ivan” :aka [“The Terrible”, “Durak”]}]) (d/q ‘[:find ?name :where [?e :aka “Durak”] [?e :name ?name]] @conn)
15. (defn create-conn [schema] (atom (empty-db schema) :meta { :listeners (atom {}) })) How it works?
16. (defrecord DB [ schema eavt aevt avet max-eid max-tx ])
17. (defn with [db datoms] (→ db (update-in [:eavt] into datoms) (update-in [:aevt] into datoms)
18. (defn transact [conn datoms] (swap! conn with datom))
19. In-house implementation of B+ tree (sorted set) BTSet
20. Performance comparable with sorted-set: ~ 50% slower conj ~200% faster iterate
21. Binary search lookups, fast range scans, reverse iteration, fast first-time creation
22. ClojureScript, but heavy use of JS arrays and APIs
23. 700 loc btset 550 loc query engine 1700 loc total 1200 loc tests DataScript is lightweight
24. Unfortunate to be associated with the word “database”: no networking, query over memory,
25. no log, no history & compound indexes, no external segments, constant space operation
26. DataScript is more of a persistent data structure. It’s just three sorted sets, literally.
27. Every app has an ad-hoc state Put everything in a database How to use it
28. Non-trivial SPA has complex state KV stores do not cut it
29. No inherent hierarchy, very natural for any data: sparse, irregular, hierarchical, graph
30. Faster data retrieval from big datasets: indexes + hash joins
31. Uniform solution for storage encourages decoupling and reusable components: server sync, undo/redo, local caching, audit
32. Datoms, DB and transactions have standard way to be serialized No need to invent format for delta exchange
33. Immutability + db as a value + transactions = always consistent render
34. Encourages to write decoupled, pure renders Good fit for React and Flux
35. Example architecture: CatChat github/tonsky/datascript-chat
38. Raw react render, pure, decoupled
39. Re-render triggered by database mutations, always top-down, full re-render
40. “Server” sync, db cleanup all decoupled from render code
41. Operates in a bound space via cleanup
42. Second DB used to emulate server
43. Example architecture: Acha-Acha acha-acha.co
45. Whole DB prefetch: no server fetch on any navigation, all queries and aggregations hap- pen on a client
46. Always up-to-date server sync: initial dump + deltas via websock- et (tricky to do correctly)
47. Non-hierarchical layout: user page, repo page, main page
48. 14K datoms in 100..150ms
49. Example architecture: Menu [WIP] github/tonsky/datascript-menu
51. Component can listen for a specific [e a v] pattern
52. Optimized for huge amount of listeners, e.g. every person component ever can subscribe to [(:db/id person) :person/name _]
53. Similar to Om ref-cursors Have to figure out how to play nicely with top-down rendering
54. Considered alpha stage Project status
55. No docs, must know/learn Datomic Extensive acceptance tests suite helps
56. Almost no validation, silent fails/ unexpected results instead
57. Occasional breaking changes (not significant though) If it’s in Datomic and DataScript already, it probably won’t change
58. Cover all Datomic APIs components lookup refs find specifications pull API Future plans
59. Entry barrier: docs, tutorials, error reporting
60. JVM Clojure port
61. Nikita Prokopov github/tonsky/datascript tonsky.me Thanks! — Questions?