Datomic for the 96% - Redux

Introductory Datomic presentation delivered at Øredev 2014 in Malmö, Sweden

Transcript

1. Datomic for the 96% @rkneufeld – rkn.io
2. @rkneufeld – rkn.io Ryan Neufeld
3. Datomic?
4. Databases are hard
5. Five Challenges
6. Constrained Writes
7. 96%
8. Constrained Reads
9. RIGID Ops
10. Storage Query I/O Tx Indexing
11. RIGID Model
12. “Person can belong multiple clubs” join table person table club table FK constraints foreign keys ids
13. Update In-place!
14. x → y
15. Transience Sharing Difficult Distribution Difficult Concurrency Difficult Access Pattern Eager Caching Difficult
16. [x, y] “then…”
17. Transience Persistence Sharing Difficult Trivial Distribution Difficult Trivial Concurrency Difficult Trivial Access Pattern Eager Eager or Lazy Caching Difficult Trivial
18. The Laws 1. Memory is expensive 2. Storage is expensive 3. Machines are precious 4. Resources are dedicated
20. The Goals 1. Functional Database
21. The Goals 2. Local, Lazy Reads
22. The Goals 3. Persistent Data
23. The Goals 4. Serialized, ACID Transactions
24. The Goals 5. Aware of Time
25. The Goals 6. Scalable Reads
26. The Goals 7. Programmable Database
27. a quick tour
28. 1, 2, 3: Functional, Lazy “Peers” (require ‘[datomic.api :as d]) ;; Connect to a remote database (def conn (d/connect “datomic:ddb://us-east-1/mb/mbrainz)) ;; Fetch the current database value (def db (d/db conn)) ;; Evaluate a query (d/q […<some query>…] db) ;; Join across multiple databases (d/q […<some query>…] db1 db2) ;; Retrieve an entity (d/entity db 42)
29. 1, 2, 3: Functional, Lazy “Peers” ;; Connect to a remote database (def conn (d/connect “datomic:ddb://us-east-1/mb/mbrainz))
30. 1, 2, 3: Functional, Lazy “Peers” ;; Connect to a remote database (def conn (d/connect “datomic:ddb://us-east-1/mb/mbrainz)) mem sql dev riak couchbase inf cassandra { Pluggable Storage
31. 1, 2, 3: Functional, Lazy “Peers” ;; Fetch the current database value (def db (d/db conn)) Immutable lazy database value at time t
32. 1, 2, 3: Functional, Lazy “Peers” ;; Evaluate a query (d/q […<some query>…] db) Query a database value, not a transient connection
33. 1, 2, 3: Functional, Lazy “Peers” ;; Join across multiple databases (d/q […<some query>…] db1 db2) Query multiple, potentially disparate data sources
34. 1, 2, 3: Functional, Lazy “Peers” ;; Retrieve an entity (d/entity db 42) Direct, lazy access
35. 4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx data>)
36. 4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx data>) Simple, native data e.g. [:db/add :ryan :likes :pizza]
37. 4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx data>) Tx enqueued on Txor, returns immediately.
38. 4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx data>) (keys @result) ;; (:db-before :db-after :tx-data :tempids)
39. 4: Serialized, ACID Transactions (def result (d/transact-async conn <some tx data>) (keys @result) ;; (:db-before :db-after :tx-data :tempids) Old, new values of database
40. 5: Time-aware ;; View the db, as of a week ago (def db-before (d/as-of db <1 week ago>)) ;; Or, pretend to transact new data (def db-later (d/with db <proposed tx data>)) ;; History of a db, in its entirety (d/history db) ;; Listen for new changes (d/tx-report-queue conn) ;; Or, process old ones (d/tx-range (d/log conn) <1 week ago> nil)
41. 5: Time-aware ;; View the db, as of a week ago (def db-before (d/as-of db <1 week ago>)) Zero-effort, time travel
42. 5: Time-aware ;; Or, pretend to transact new data (def db-later (d/with db <proposed tx data>)) Zero-effort, time travel
43. 5: Time-aware ;; History of a db, in its entirety (d/history db) Every event across time
44. 5: Time-aware ;; Listen for new changes (d/tx-report-queue conn) Monitor & react to changes made elsewhere
45. 5: Time-aware ;; Or, process old ones (d/tx-range (d/log conn) <1 week ago> nil) Or introspect on the prev. log of changes
46. 6: Scalable Reads Backing Storage DB Instance Client
47. Backing Storage Transactor Peer 6: Scalable Reads
48. 7: Programmable Database ;; A simple query (defn people-who-like [db food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db food))
49. 7: Programmable Database ;; A simple query (defn people-who-like [db food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db food)) Query any database
50. 7: Programmable Database ;; A simple query (defn people-who-like [db food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db food)) Query is Data, not strings
51. 7: Programmable Database ;; A simple query (defn people-who-like [db food] (d/q ‘[:find ?person :in $ ?food :where [?person :likes ?food]] db)) And so much more…
52. Query
53. Query is Datalog
54. Datalog is* Relations + Recursion *I mean, kinda…
55. Datalog means: 1. Order-independent clauses 2. Guaranteed Termination 3. Simple pattern-matching
56. Entity Attribute Value 42 :email jack@example.com 43 :email jill@example.com 42 :orders 107 42 :orders 41 A Sample Database
57. Data Pattern [entity attribute value] Constrain the results returned, bind variables
58. Data Pattern [?customer :email ?email] “What are our customer’s emails?”
59. Data Pattern [?customer :email ?email] Constants
60. Data Pattern [?customer :email ?email] Variables
61. Entity Attribute Value 42 :email jack@example.com 43 :email jill@example.com 42 :orders 107 42 :orders 41 Data Pattern [?customer :email ?email]
62. Constants Anywhere [42 :email ?email] “Find a particular customer’s email”
63. Entity Attribute Value 42 :email jack@example.com 43 :email jill@example.com 42 :orders 107 42 :orders 41 Constants Anywhere [42 :email ?email]
64. Variables Anywhere [42 ?attribute] “What attributes does customer 42 have?”
65. Entity Attribute Value 42 :email jack@example.com 43 :email jill@example.com 42 :orders 107 42 :orders 41 Variables Anywhere [42 ?attribute]
66. Variables Anywhere [42 ?attribute ?value] “What attributes and values does customer 42 have?”
67. Entity Attribute Value 42 :email jack@example.com 43 :email jill@example.com 42 :orders 107 42 :orders 41 Variables Anywhere [42 ?attribute ?value]
68. Where Clause [:find ?customer :where [?customer :email]] Data Pattern
69. Implicit Join [:find ?customer :where [?customer :email] [?customer :orders]] “Find all customers who have placed orders”
70. Query API (require ‘[datomic.api :as d]) (d/q ‘[:find ?customer :where [?customer :email] [?customer :orders]] db)
71. Query API (d/q ‘[:find ?customer :where [?customer :email] [?customer :orders]] db) Query
72. Query API (d/q ‘[:find ?customer :where [?customer :email] [?customer :orders]] db …) Inputs
73. In Clause :in $database ?email Name inputs for reference elsewhere in query
74. Parameterized Query (d/q ‘[:find ?customer :in $db ?email :where [$db ?customer :email ?email]] db “jake@example.com”) “Find customer by email”
75. Ignore DB in clauses entirely (d/q ‘[:find ?customer :in $ ?email :where [?customer :email ?email]] db “jake@example.com”)
76. Predicates [(> ?price 100)] Functional constraints that appear within a :where clause
77. Predicates [:find ?item :where [?item :price ?price] [(> ?price 100)]] “Find the expensive items”
78. Functions [(shipping ?postal ?weight) ?cost]) Take bound variables as input, bind variable with output
79. Functions (defn shipping “Estimate the cost to ship goods to a given postal region.” [postal weight] (* (cost-by-postal postal) weight)) Plain, native functions
80. Using Functions [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal] [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] “Find me customer/product combos where shipping cost dominates the price”
81. [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal] [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Navigate from customer to postal code Using Functions
82. [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal] [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Retrieve needed product facts Using Functions
83. [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal] [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Retrieve estimated shipping cost Using Functions
84. [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal] [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Constraint based upon price & shipping Using Functions
85. Using Functions [:find ?customer ?product :where [?customer :shipAddr ?addr] [?addr :postal-code ?postal] [?product :product/weight ?weight] [?product :product/price ?price] [(shipping ?postal ?weight) ?shipping] [(<= ?price ?shipping)]] Return customer, product pairs
86. Entity Attribute Value 42 :email jack@example.com 43 :email jill@example.com 42 :orders 107 42 :orders 41 Entity API
87. Entity API {:db/id 42 :email “jack@example.com” :orders [{…} {…}] Lazy, immutable, map-like view of entity’s attributes & values 42 :email jack@example.com 42 :orders 107 42 :orders 41 (d/entity db 42)
88. Transactions
89. Transactions = Lists of Datoms [[:db/add ryan :likes pizza] [:db/retract ryan :likes pizza]]
90. Datom Structure [op entity attr val]
91. Datom Structure [op entity attr val] :db/add :db/retract :db.fn/retractEntity
92. Atomic Increment [:db/add ryan :balance <old> + 10] ?
93. Datom Structure [op entity attr val] Provide your own functions
94. Atomic Increment [:inc ryan :balance 10]
95. Transaction Functions • Run inside transaction • Can access current DB-value • Expand short-form into 1+ assertions/retractions
96. Transaction Functions (defn inc [db e attr val] (let [entity (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
97. Transaction Functions (defn inc [db e attr val] (let [entity (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
98. Transaction Functions (defn inc [db e attr val] (let [entity (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
99. Transaction Functions (defn inc [db e attr val] (let [entity (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
100. Transaction Functions (defn inc [db e attr val] (let [entity (d/entity db e) prev (get entity attr)] [[:db/add e attr (+ prev val)]]))
101. Transaction Functions [:inc ryan :balance 10] [:db/add ryan :balance 110]
102. Transaction Short Form (let [ryan (d/tempid :db.part/user)] [[:db/add ryan :name “Ryan”] [:db/add ryan :email “ryan@rkn.io”] [:db/add ryan :homepage “rkn.io”] …])
103. Transaction Short Form {:db/id (d/tempid :db.part/user) :name “Ryan” :email “ryan@rkn.io” :homepage “rkn.io” …}* Expands to list of datoms
104. Schema
105. “Person can belong multiple clubs” join table person table club table FK constraints foreign keys ids
106. “Person can belong multiple clubs” [?person :club ?club]
107. Modeling a Social Media Site • Stories • Users • Comments
108. Attribute Type Cardinality :story/title string 1 :story/url string 1 :story/slug string 1 :news/comment ref many Modelling News Stories
109. Attribute Type Cardinality :story/title string 1 Modelling News Stories {:db/id … :db/ident :story/title :db/valueType :db.type/string :db/cardinality :db.cardinality/one …}
110. isn’t that a short-form transaction? Wait…
111. Schema is data too… “What are all the attributes in the database?” [?e :db/valueType]
112. Attribute Type Cardinality :user/full-name string 1 :user/email* string 1 :user/votes ref many Modelling Users
113. Attribute Type Cardinality :user/full-name string 1 :user/email* string 1 :user/votes ref many Modelling Users What about uniqueness?
114. Attribute Type Cardinality :user/email string 1 Unique Value {:db/ident :user/email :db/unique :db.unique/value …} Value is unique
115. Attribute Type Cardinality :user/email string 1 Unique Identity {:db/ident :user/email :db/unique :db.unique/identity …} Value is unique, identifies entity
116. Unique Identity (d/entity db [:user/email “ryan@rkn.io”]) Lookup entities by identifier, not arbitrary database IDs
117. Attribute Type Cardinality :comment/body string 1 :comment/author ref 1 :news/comment ref many Modelling Comments
118. Attribute Type Cardinality :comment/body string 1 :comment/author ref 1 :news/comment ref many Modelling Comments “Type” does not dictate attributes
119. Attribute Type Cardinality :comment/body string 1 :comment/author ref 1 :news/comment ref many Modelling Comments How do ask for all of a user’s comments?
120. Relations are bi-directional ;; Get a comment’s author (:comment/author some-comment) ;; Get an author’s comments (:comment/_author some-author) Navigate backwards
121. Structure Name Included? Row EAVT All Datoms Column AEVT All Datoms Lookup AVET “Indexed” attributes Graph VAET :db.type/ref attributes Available Indices
122. Attribute Type Cardinality :comment/body string 1 :comment/author ref 1 :news/comment ref many Modelling Comments “What are all the comments for a story”
123. Recursive Queries [(story-comment ?story ?comment) [?story :story/title] [?story :news/comment ?comment]] Base Case: A story has a comment
124. Recursive Queries [(story-comment ?story ?comment) [?parent :news/comment ?comment] (story-comment ?story ?parent)] Recursive Case: Comment is a child of a story-comment
125. Datomic: You should try it
126. Database by Anton Outkine from The Noun Project Original presentation by Stuart Halloway @rkneufeld – ryan@rkn.io Questions?
127. Scraps
128. complexities mitigated Lost Data Log Analysis ORM Inheritance Structural Rigidity Model Caching App Caching Managing Time Test Setup Defensive Copying Join Tables Relationship Direction Logic Duplication Imperative Code Read Transactions Denormalization Eventual Consistency DAOs DTOs Objects Strings String Injection Data Duplication Isolation Levels 113