DataScript for Web Development

DataScript for Web Development

DataScript is an open-source implementation of Datalog and Datomic in-memory storage, in ClojureScript, for browser. Here I talk about motivation behind DataScript, how it was created, how it can be used, what kind of problems it solves and how simple and sane web app development is when you have a database in your browser.

4188c62c28a196e3e82363217c56fca5?s=128

Nikita Prokopov

December 04, 2014
Tweet

Transcript

  1. DataScript for Web Development

  2. Nikita Prokopov tonsky.me Novosibirsk, Russia

  3. None
  4. Server-side traditionally benefits from relational storage + queries Rationale

  5. Why web developers should limit themselves?

  6. Datomic model feels right and was within my reach

  7. DataScript is a post-modern DB: someone else’s good idea redone

    on a more primitive level
  8. <entity, attribute, value> triple store What’s DataScript?

  9. Database as an immutable value

  10. Completely in-memory

  11. Datalog queries over it

  12. Tries to mimic Datomic API

  13. Written in ClojureScript, JS bindings available

  14. (def conn (d/create-conn {:aka {:db/cardinality :db.cardinality/many }})] (d/transact! conn [

    {:db/id 1 :name “Ivan” :aka [“The Terrible”, “Durak”]}]) (d/q ‘[:find ?name :where [?e :aka “Durak”] [?e :name ?name]] @conn)
  15. (defn create-conn [schema] (atom (empty-db schema) :meta { :listeners (atom

    {}) })) How it works?
  16. (defrecord DB [ schema eavt aevt avet max-eid max-tx ])

  17. (defn with [db datoms] (→ db (update-in [:eavt] into datoms)

    (update-in [:aevt] into datoms)
  18. (defn transact [conn datoms] (swap! conn with datom))

  19. In-house implementation of B+ tree (sorted set) BTSet

  20. Performance comparable with sorted-set: ~ 50% slower conj ~200% faster

    iterate
  21. Binary search lookups, fast range scans, reverse iteration, fast first-time

    creation
  22. ClojureScript, but heavy use of JS arrays and APIs

  23. 700 loc btset 550 loc query engine 1700 loc total

    1200 loc tests DataScript is lightweight
  24. Unfortunate to be associated with the word “database”: no networking,

    query over memory,
  25. no log, no history & compound indexes, no external segments,

    constant space operation
  26. DataScript is more of a persistent data structure. It’s just

    three sorted sets, literally.
  27. Every app has an ad-hoc state Put everything in a

    database How to use it
  28. Non-trivial SPA has complex state KV stores do not cut

    it
  29. No inherent hierarchy, very natural for any data: sparse, irregular,

    hierarchical, graph
  30. Faster data retrieval from big datasets: indexes + hash joins

  31. Uniform solution for storage encourages decoupling and reusable components: server

    sync, undo/redo, local caching, audit
  32. Datoms, DB and transactions have standard way to be serialized

    No need to invent format for delta exchange
  33. Immutability + db as a value + transactions = always

    consistent render
  34. Encourages to write decoupled, pure renders Good fit for React

    and Flux
  35. Example architecture: CatChat github/tonsky/datascript-chat

  36. None
  37. None
  38. Raw react render, pure, decoupled

  39. Re-render triggered by database mutations, always top-down, full re-render

  40. “Server” sync, db cleanup all decoupled from render code

  41. Operates in a bound space via cleanup

  42. Second DB used to emulate server

  43. Example architecture: Acha-Acha acha-acha.co

  44. None
  45. Whole DB prefetch: no server fetch on any navigation, all

    queries and aggregations hap- pen on a client
  46. Always up-to-date server sync: initial dump + deltas via websock-

    et (tricky to do correctly)
  47. Non-hierarchical layout: user page, repo page, main page

  48. 14K datoms in 100..150ms

  49. Example architecture: Menu [WIP] github/tonsky/datascript-menu

  50. None
  51. Component can listen for a specific [e a v] pattern

  52. Optimized for huge amount of listeners, e.g. every person component

    ever can subscribe to [(:db/id person) :person/name _]
  53. Similar to Om ref-cursors Have to figure out how to

    play nicely with top-down rendering
  54. Considered alpha stage Project status

  55. No docs, must know/learn Datomic Extensive acceptance tests suite helps

  56. Almost no validation, silent fails/ unexpected results instead

  57. Occasional breaking changes (not significant though) If it’s in Datomic

    and DataScript already, it probably won’t change
  58. Cover all Datomic APIs components lookup refs find specifications pull

    API Future plans
  59. Entry barrier: docs, tutorials, error reporting

  60. JVM Clojure port

  61. Nikita Prokopov github/tonsky/datascript tonsky.me Thanks! — Questions?