Datomic for the 96% - Redux

58dc7342c17dbc3a0848fe78e5b6d3a4?s=47 Ryan Neufeld
November 06, 2014

Datomic for the 96% - Redux

Introductory Datomic presentation delivered at Øredev 2014 in Malmö, Sweden

58dc7342c17dbc3a0848fe78e5b6d3a4?s=128

Ryan Neufeld

November 06, 2014
Tweet

Transcript

  1. 7.

    96%

  2. 14.
  3. 17.

    Transience Persistence Sharing Difficult Trivial Distribution Difficult Trivial Concurrency Difficult

    Trivial Access Pattern Eager Eager or Lazy Caching Difficult Trivial
  4. 18.

    The Laws 1. Memory is expensive 2. Storage is expensive

    3. Machines are precious 4. Resources are dedicated
  5. 19.
  6. 28.

    1, 2, 3: Functional, Lazy “Peers” (require ‘[datomic.api :as d])

    ;; Connect to a remote database (def conn (d/connect “datomic:ddb://us-east-1/mb/mbrainz)) ;; Fetch the current database value (def db (d/db conn)) ;; Evaluate a query (d/q […<some query>…] db) ;; Join across multiple databases (d/q […<some query>…] db1 db2) ;; Retrieve an entity (d/entity db 42)
  7. 29.

    1, 2, 3: Functional, Lazy “Peers” ;; Connect to a

    remote database (def conn (d/connect “datomic:ddb://us-east-1/mb/mbrainz))
  8. 30.

    1, 2, 3: Functional, Lazy “Peers” ;; Connect to a

    remote database (def conn (d/connect “datomic:ddb://us-east-1/mb/mbrainz)) mem sql dev riak couchbase inf cassandra { Pluggable Storage
  9. 31.

    1, 2, 3: Functional, Lazy “Peers” ;; Fetch the current

    database value (def db (d/db conn)) Immutable lazy database value at time t
  10. 32.

    1, 2, 3: Functional, Lazy “Peers” ;; Evaluate a query

    (d/q […<some query>…] db) Query a database value, not a transient connection
  11. 33.

    1, 2, 3: Functional, Lazy “Peers” ;; Join across multiple

    databases (d/q […<some query>…] db1 db2) Query multiple, potentially disparate data sources
  12. 34.
  13. 36.

    4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx

    data>) Simple, native data e.g. [:db/add :ryan :likes :pizza]
  14. 37.

    4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx

    data>) Tx enqueued on Txor, returns immediately.
  15. 38.

    4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx

    data>) (keys @result) ;; (:db-before :db-after :tx-data :tempids)
  16. 39.

    4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx

    data>) (keys @result) ;; (:db-before :db-after :tx-data :tempids) Old, new values of database
  17. 40.

    5: Time-aware ;; View the db, as of a week

    ago (def db-before (d/as-of db <1 week ago>)) ;; Or, pretend to transact new data (def db-later (d/with db <proposed tx data>)) ;; History of a db, in its entirety (d/history db) ;; Listen for new changes (d/tx-report-queue conn) ;; Or, process old ones (d/tx-range (d/log conn) <1 week ago> nil)
  18. 41.

    5: Time-aware ;; View the db, as of a week

    ago (def db-before (d/as-of db <1 week ago>)) Zero-effort, time travel
  19. 42.

    5: Time-aware ;; Or, pretend to transact new data (def

    db-later (d/with db <proposed tx data>)) Zero-effort, time travel
  20. 43.

    5: Time-aware ;; History of a db, in its entirety

    (d/history db) Every event across time
  21. 45.

    5: Time-aware ;; Or, process old ones (d/tx-range (d/log conn)

    <1 week ago> nil) Or introspect on the prev. log of changes
  22. 48.

    7: Programmable Database ;; A simple query (defn people-who-like [db

    food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db food))
  23. 49.

    7: Programmable Database ;; A simple query (defn people-who-like [db

    food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db food)) Query any database
  24. 50.

    7: Programmable Database ;; A simple query (defn people-who-like [db

    food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db food)) Query is Data, not strings
  25. 51.

    7: Programmable Database ;; A simple query (defn people-who-like [db

    food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db)) And so much more…
  26. 52.
  27. 61.

    Entity Attribute Value 42 :email jack@example.com 43 :email jill@example.com 42

    :orders 107 42 :orders 41 Data Pattern [?customer :email ?email]
  28. 63.

    Entity Attribute Value 42 :email jack@example.com 43 :email jill@example.com 42

    :orders 107 42 :orders 41 Constants Anywhere [42 :email ?email]
  29. 65.

    Entity Attribute Value 42 :email jack@example.com 43 :email jill@example.com 42

    :orders 107 42 :orders 41 Variables Anywhere [42 ?attribute]
  30. 67.

    Entity Attribute Value 42 :email jack@example.com 43 :email jill@example.com 42

    :orders 107 42 :orders 41 Variables Anywhere [42 ?attribute ?value]
  31. 70.
  32. 74.

    Parameterized Query (d/q ‘[:find ?customer :in $db ?email :where [$db

    ?customer :email ?email]] db “jake@example.com”) “Find customer by email”
  33. 75.

    Ignore DB in clauses entirely (d/q ‘[:find ?customer :in $

    ?email :where [?customer :email ?email]] db “jake@example.com”)
  34. 79.

    Functions (defn shipping “Estimate the cost to ship goods to

    a given postal region.” [postal weight] (* (cost-by-postal postal) weight)) Plain, native functions
  35. 80.

    Using Functions [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr

    :postal-code ?postal] [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] “Find me customer/product combos where shipping cost dominates the price”
  36. 81.

    [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal]

    [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Navigate from customer to postal code Using Functions
  37. 82.

    [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal]

    [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Retrieve needed product facts Using Functions
  38. 83.

    [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal]

    [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Retrieve estimated shipping cost Using Functions
  39. 84.

    [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal]

    [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Constraint based upon price & shipping Using Functions
  40. 85.

    Using Functions [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr

    :postal-code ?postal] [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Return customer, product pairs
  41. 87.

    Entity API {:db/id 42 :email “jack@example.com” :orders [{…} {…}] Lazy,

    immutable, map-like view of entity’s attributes & values 42 :email jack@example.com 42 :orders 107 42 :orders 41 (d/entity db 42)
  42. 95.

    Transaction Functions • Run inside transaction • Can access current

    DB-value • Expand short-form into 1+ assertions/retractions
  43. 96.

    Transaction Functions (defn inc [db e attr val] (let [entity

    (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
  44. 97.

    Transaction Functions (defn inc [db e attr val] (let [entity

    (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
  45. 98.

    Transaction Functions (defn inc [db e attr val] (let [entity

    (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
  46. 99.

    Transaction Functions (defn inc [db e attr val] (let [entity

    (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
  47. 100.

    Transaction Functions (defn inc [db e attr val] (let [entity

    (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
  48. 102.

    Transaction Short Form (let [ryan (d/tempid :db.part/user)] [[:db/add ryan :name

    “Ryan”] [:db/add ryan :email “ryan@rkn.io”] [:db/add ryan :homepage “rkn.io”] …])
  49. 104.
  50. 105.
  51. 108.

    Attribute Type Cardinality :story/title string 1 :story/url string 1 :story/slug

    string 1 :news/comment ref many Modelling News Stories
  52. 109.

    Attribute Type Cardinality :story/title string 1 Modelling News Stories {:db/id

    … :db/ident :story/title :db/valueType :db.type/string :db/cardinality :db.cardinality/one …}
  53. 111.

    Schema is data too… “What are all the attributes in

    the database?” [?e :db/valueType]
  54. 114.
  55. 115.

    Attribute Type Cardinality :user/email string 1 Unique Identity {:db/ident :user/email

    :db/unique :db.unique/identity …} Value is unique, identifies entity
  56. 118.

    Attribute Type Cardinality :comment/body string 1 :comment/author ref 1 :news/comment

    ref many Modelling Comments “Type” does not dictate attributes
  57. 119.

    Attribute Type Cardinality :comment/body string 1 :comment/author ref 1 :news/comment

    ref many Modelling Comments How do ask for all of a user’s comments?
  58. 120.

    Relations are bi-directional ;; Get a comment’s author (:comment/author some-comment)

    ;; Get an author’s comments (:comment/_author some-author) Navigate backwards
  59. 121.

    Structure Name Included? Row EAVT All Datoms Column AEVT All

    Datoms Lookup AVET “Indexed” attributes Graph VAET :db.type/ref attributes Available Indices
  60. 122.

    Attribute Type Cardinality :comment/body string 1 :comment/author ref 1 :news/comment

    ref many Modelling Comments “What are all the comments for a story”
  61. 124.
  62. 126.

    Database by Anton Outkine from The Noun Project Original presentation

    by Stuart Halloway @rkneufeld – ryan@rkn.io Questions?
  63. 127.
  64. 128.

    complexities mitigated Lost Data Log Analysis ORM Inheritance Structural Rigidity

    Model Caching App Caching Managing Time Test Setup Defensive Copying Join Tables Relationship Direction Logic Duplication Imperative Code Read Transactions Denormalization Eventual Consistency DAOs DTOs Objects Strings String Injection Data Duplication Isolation Levels 113