A Gentle Intro to Datomic

A gentle introduction to Datomic.

I gave this talk at the November 2015 meeting of Clojure Toronto (http://www.meetup.com/Clojure-Toronto/)

Transcript

1. A Gentle Intro to Datomic Max Veytsman max@appcanary.com @mveystman ClojureTO, November 2015
2. What the Mongo people say » Schemas suck! » SQL sucks! » ORMs suck! ClojureTO, November 2015
3. This talk is not about that ClojureTO, November 2015
4. Motivation » Relational Databases are from 1970s » Storage is expensive » Memory is expensive » Today » Storage is cheap » Memory is cheap ClojureTO, November 2015
5. Storage is cheap » It's usually cheaper to not delete things » ALTER TABLE users ADD isDeleted BOOL; » You generally want a log of updates » log/history tables » temporal tables ClojureTO, November 2015
6. Memory is cheap (sql/query "postgresql://localhost:5432/somedb" ["select id from users where name = 'max'"]) » Application constructs a query and sends it over network » Database processes query, sends response over network » Application processes result » Maybe the application caches the query? ClojureTO, November 2015
7. Memory is cheap » Most datasets fit into RAM » Why not store db in app memory? » If only my database was immutable! ClojureTO, November 2015
8. Database as a value » Database is a value, at a point-in-time » All transactions are handled by one process, the Transactor » App runs query engine locally, client is a "peer" » The datonmic query engine is just a library ClojureTO, November 2015
9. Architecture ClojureTO, November 2015
10. Motivating example (d/as-of db t) - what was the db at a point in time t? (how-well-is-my-startup-doing db) ;=> "100 points" (def db-last-month (d/as-of db #inst "2015-10-17") (how-well-is-my-startup-doing db-last-month) ;=> "80 points" » We grew 25% boss! ClojureTO, November 2015
11. Datomic Log of "datoms" and Index ClojureTO, November 2015
12. Datoms - 5 tuples "Max likes pizza" (d/transact conn [[:db/add 17592186045418 :user/likes "pizza"]]) » Entity (17592186045418) » Attribute (:user/likes) » Value ("pizza") » Transaction (?) » Added? (true - because :db/add) ClojureTO, November 2015
13. Datoms - 5 tuples "Max no longer likes pizza" (d/transact conn [[:db/retract 17592186045418 :user/likes "pizza"]]) » Entity (17592186045418) » Attribute (:user/likes) » Value ("pizza") » Transaction (?) » Added? (false - because :db/add) ClojureTO, November 2015
14. Transactions (d/transact conn [[:db/add 17592186045418 :user/likes "pizza"]]) => {:db-before datomic.db.Db@d418eda9 :db-after datomic.db.Db@1d2eceea :tx-data [#datom[17592186045418 :user/likes "pizza" 13194139534313 true] ...] :tempids {}} » Transaction - 13194139534313 ClojureTO, November 2015
15. Datalog » A subset of prolog » Positional » ?'s are things we query for ClojureTO, November 2015
16. Basic Query (d/q '[:find ?e :where [?e :age 42]] data) ;=>#{[ethel] [fred]} (def data '[[sally :age 21] [fred :age 42] [ethel :age 42] [fred :likes pizza] [sally :likes opera] [ethel :likes sushi] ClojureTO, November 2015
17. Unification (d/q '[:find ?e :where [?e :age 42] [?e :likes pizza]] data) ;=>#{[fred]} (def data '[[sally :age 21] [fred :age 42] [ethel :age 42] [fred :likes pizza] [sally :likes opera] [ethel :likes sushi]] ClojureTO, November 2015
18. Or clauses (d/q '[:find ?e :where (or [?e :age 42] [?e :likes opera])] data) ;=>#{[ethel] [fred] [sally]} (def data '[[sally :age 21] [fred :age 42] [ethel :age 42] [fred :likes pizza] [sally :likes opera] [ethel :likes sushi]] ClojureTO, November 2015
19. Multiple query variables (d/q '[:find ?e ?l :where [?e :likes ?l]] data) ;=>#{[sally opera] [ethel sushi] [fred pizza]} (def data '[[sally :age 21] [fred :age 42] [ethel :age 42] [fred :likes pizza] [sally :likes opera] [ethel :likes sushi]] ClojureTO, November 2015
20. Querying a database (def db (d/db conn)) (d/q '[:find ?e :where [?e :user/likes "pizza"]] db) ;=> #{[17592186045418]} (d/q '[:find ?e ?l :where [?e :user/likes ?l]] db) ;=> #{[17592186045418 "pizza"]} ClojureTO, November 2015
21. Querying a database (d/q '[:find ?e ?l ?t :where [?e :user/likes ?l ?t]] db) ;=> #{[17592186045418 "pizza" 13194139534313]} (d/q '[:find ?e ?l ?t ?a :where [?e :user/likes ?l ?t ?a]] db) ;=> #{[17592186045418 "pizza" 13194139534313 true]} ClojureTO, November 2015
22. Transactions as entities (d/transact conn [[:db/add 17592186045418 :user/likes "pizza"]]) => {:db-before datomic.db.Db@d418eda9 :db-after datomic.db.Db@1d2eceea :tx-data [#datom[17592186045418 :user/likes "pizza" 13194139534313 true] [#datom[13194139534313 :db/txInstant #inst "2015-11-17T20:34:31.654-00:00" 13194139534313 true]] :tempids {}} ClojureTO, November 2015
23. (d/q '[:find ?e ?l ?inst :where [?e :user/likes ?l ?t] [?t :db/txInstant ?inst]] db) => {[17592186045418 "pizza" #inst "2015-11-17T20:34:31.654-00:00"]} ClojureTO, November 2015
24. Another motivating example (d/since db t) - what is the db if we only had the txes since time t (all-the-emails-we-sent db) ;=> [... 10 million emails ...] (def db-since-this-morning (d/as-of db #inst "2015-11-17") (all-the-emails-we-sent db-since-this-morning) ;=> [... 17 emails ...] ClojureTO, November 2015
25. Creating new entities ClojureTO, November 2015
26. Tempids (d/tempid :db.part/user) ;=> #db/id[:db.part/user -1002040] (d/tempid :db.part/user) ;=> #db/id[:db.part/user -1002041]] (d/tempid :db.part/user) ;=> #db/id[:db.part/user -1] ClojureTO, November 2015
27. Tempids (d/transact conn [[:db/add (d/tempid :db.part/user -1) :user/likes "cereal"] [:db/add (d/tempid :db.part/user -1) :user/name "Sarah"]]) => {... :tx-data [#datom[13194139534317 50 #inst "2015-11-17T21:41:55.192-00:00" 13194139534317 true] #datom[17592186045422 63 "cereal" 13194139534317 true] #datom[17592186045422 64 "Sarah" 13194139534317 true]], :tempids {-9223350046622220289 17592186045422}} ClojureTO, November 2015
28. Transacting Entities (d/transact conn [[:db/add (d/tempid :db.part/user -1) :user/likes "smores"] [:db/add (d/tempid :db.part/user -1) :user/name "Kyle"]]) is the same as (d/transact conn [{:db/id (d/tempid :db.part/user) :user/likes "smores" :user/name "Kyle"}]) ClojureTO, November 2015
29. Making new attributes ClojureTO, November 2015
30. Schemas are data! (d/transact conn [{:db/id #db/id[:db.part/db] :db/ident :user/hates :db/valueType :db.type/string :db/cardinality :db.cardinality/one :db.install/_attribute :db.part/db}]) » Types: string, boolean, long, bigint, float, double, bigdec, ref, instant, uuid, uri, bytes » Cardinality: one, many ClojureTO, November 2015
31. The end ClojureTO, November 2015