Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Datomic for the 96% - Redux

Ryan Neufeld
November 06, 2014

Datomic for the 96% - Redux

Introductory Datomic presentation delivered at Øredev 2014 in Malmö, Sweden

Ryan Neufeld

November 06, 2014
Tweet

More Decks by Ryan Neufeld

Other Decks in Programming

Transcript

  1. 96%

  2. Transience Persistence Sharing Difficult Trivial Distribution Difficult Trivial Concurrency Difficult

    Trivial Access Pattern Eager Eager or Lazy Caching Difficult Trivial
  3. The Laws 1. Memory is expensive 2. Storage is expensive

    3. Machines are precious 4. Resources are dedicated
  4. 1, 2, 3: Functional, Lazy “Peers” (require ‘[datomic.api :as d])

    ;; Connect to a remote database (def conn (d/connect “datomic:ddb://us-east-1/mb/mbrainz)) ;; Fetch the current database value (def db (d/db conn)) ;; Evaluate a query (d/q […<some query>…] db) ;; Join across multiple databases (d/q […<some query>…] db1 db2) ;; Retrieve an entity (d/entity db 42)
  5. 1, 2, 3: Functional, Lazy “Peers” ;; Connect to a

    remote database (def conn (d/connect “datomic:ddb://us-east-1/mb/mbrainz))
  6. 1, 2, 3: Functional, Lazy “Peers” ;; Connect to a

    remote database (def conn (d/connect “datomic:ddb://us-east-1/mb/mbrainz)) mem sql dev riak couchbase inf cassandra { Pluggable Storage
  7. 1, 2, 3: Functional, Lazy “Peers” ;; Fetch the current

    database value (def db (d/db conn)) Immutable lazy database value at time t
  8. 1, 2, 3: Functional, Lazy “Peers” ;; Evaluate a query

    (d/q […<some query>…] db) Query a database value, not a transient connection
  9. 1, 2, 3: Functional, Lazy “Peers” ;; Join across multiple

    databases (d/q […<some query>…] db1 db2) Query multiple, potentially disparate data sources
  10. 4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx

    data>) Simple, native data e.g. [:db/add :ryan :likes :pizza]
  11. 4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx

    data>) Tx enqueued on Txor, returns immediately.
  12. 4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx

    data>) (keys @result) ;; (:db-before :db-after :tx-data :tempids)
  13. 4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx

    data>) (keys @result) ;; (:db-before :db-after :tx-data :tempids) Old, new values of database
  14. 5: Time-aware ;; View the db, as of a week

    ago (def db-before (d/as-of db <1 week ago>)) ;; Or, pretend to transact new data (def db-later (d/with db <proposed tx data>)) ;; History of a db, in its entirety (d/history db) ;; Listen for new changes (d/tx-report-queue conn) ;; Or, process old ones (d/tx-range (d/log conn) <1 week ago> nil)
  15. 5: Time-aware ;; View the db, as of a week

    ago (def db-before (d/as-of db <1 week ago>)) Zero-effort, time travel
  16. 5: Time-aware ;; Or, pretend to transact new data (def

    db-later (d/with db <proposed tx data>)) Zero-effort, time travel
  17. 5: Time-aware ;; History of a db, in its entirety

    (d/history db) Every event across time
  18. 5: Time-aware ;; Or, process old ones (d/tx-range (d/log conn)

    <1 week ago> nil) Or introspect on the prev. log of changes
  19. 7: Programmable Database ;; A simple query (defn people-who-like [db

    food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db food))
  20. 7: Programmable Database ;; A simple query (defn people-who-like [db

    food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db food)) Query any database
  21. 7: Programmable Database ;; A simple query (defn people-who-like [db

    food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db food)) Query is Data, not strings
  22. 7: Programmable Database ;; A simple query (defn people-who-like [db

    food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db)) And so much more…
  23. Entity Attribute Value 42 :email [email protected] 43 :email [email protected] 42

    :orders 107 42 :orders 41 Data Pattern [?customer :email ?email]
  24. Entity Attribute Value 42 :email [email protected] 43 :email [email protected] 42

    :orders 107 42 :orders 41 Constants Anywhere [42 :email ?email]
  25. Entity Attribute Value 42 :email [email protected] 43 :email [email protected] 42

    :orders 107 42 :orders 41 Variables Anywhere [42 ?attribute]
  26. Entity Attribute Value 42 :email [email protected] 43 :email [email protected] 42

    :orders 107 42 :orders 41 Variables Anywhere [42 ?attribute ?value]
  27. Parameterized Query (d/q ‘[:find ?customer :in $db ?email :where [$db

    ?customer :email ?email]] db “[email protected]”) “Find customer by email”
  28. Ignore DB in clauses entirely (d/q ‘[:find ?customer :in $

    ?email :where [?customer :email ?email]] db “[email protected]”)
  29. Functions (defn shipping “Estimate the cost to ship goods to

    a given postal region.” [postal weight] (* (cost-by-postal postal) weight)) Plain, native functions
  30. Using Functions [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr

    :postal-code ?postal] [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] “Find me customer/product combos where shipping cost dominates the price”
  31. [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal]

    [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Navigate from customer to postal code Using Functions
  32. [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal]

    [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Retrieve needed product facts Using Functions
  33. [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal]

    [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Retrieve estimated shipping cost Using Functions
  34. [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal]

    [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Constraint based upon price & shipping Using Functions
  35. Using Functions [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr

    :postal-code ?postal] [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Return customer, product pairs
  36. Entity API {:db/id 42 :email “[email protected]” :orders [{…} {…}] Lazy,

    immutable, map-like view of entity’s attributes & values 42 :email [email protected] 42 :orders 107 42 :orders 41 (d/entity db 42)
  37. Transaction Functions • Run inside transaction • Can access current

    DB-value • Expand short-form into 1+ assertions/retractions
  38. Transaction Functions (defn inc [db e attr val] (let [entity

    (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
  39. Transaction Functions (defn inc [db e attr val] (let [entity

    (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
  40. Transaction Functions (defn inc [db e attr val] (let [entity

    (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
  41. Transaction Functions (defn inc [db e attr val] (let [entity

    (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
  42. Transaction Functions (defn inc [db e attr val] (let [entity

    (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
  43. Transaction Short Form (let [ryan (d/tempid :db.part/user)] [[:db/add ryan :name

    “Ryan”] [:db/add ryan :email “[email protected]”] [:db/add ryan :homepage “rkn.io”] …])
  44. Attribute Type Cardinality :story/title string 1 :story/url string 1 :story/slug

    string 1 :news/comment ref many Modelling News Stories
  45. Attribute Type Cardinality :story/title string 1 Modelling News Stories {:db/id

    … :db/ident :story/title :db/valueType :db.type/string :db/cardinality :db.cardinality/one …}
  46. Schema is data too… “What are all the attributes in

    the database?” [?e :db/valueType]
  47. Attribute Type Cardinality :user/email string 1 Unique Identity {:db/ident :user/email

    :db/unique :db.unique/identity …} Value is unique, identifies entity
  48. Attribute Type Cardinality :comment/body string 1 :comment/author ref 1 :news/comment

    ref many Modelling Comments “Type” does not dictate attributes
  49. Attribute Type Cardinality :comment/body string 1 :comment/author ref 1 :news/comment

    ref many Modelling Comments How do ask for all of a user’s comments?
  50. Relations are bi-directional ;; Get a comment’s author (:comment/author some-comment)

    ;; Get an author’s comments (:comment/_author some-author) Navigate backwards
  51. Structure Name Included? Row EAVT All Datoms Column AEVT All

    Datoms Lookup AVET “Indexed” attributes Graph VAET :db.type/ref attributes Available Indices
  52. Attribute Type Cardinality :comment/body string 1 :comment/author ref 1 :news/comment

    ref many Modelling Comments “What are all the comments for a story”
  53. complexities mitigated Lost Data Log Analysis ORM Inheritance Structural Rigidity

    Model Caching App Caching Managing Time Test Setup Defensive Copying Join Tables Relationship Direction Logic Duplication Imperative Code Read Transactions Denormalization Eventual Consistency DAOs DTOs Objects Strings String Injection Data Duplication Isolation Levels 113