Slide 1

Slide 1 text

Our new architecture with Clojure and Datomic Robert Stuttaford Notes for this talk: http://goo.gl/9dDnq Friday 19 April 13

Slide 2

Slide 2 text

Notes: http://goo.gl/9dDnq Before we begin, a disclaimer... • Launched the rebuild mid Jan 2013. • Not running at scale, yet; 1000 users. • You’re about to see our strategy for scale. We think it’s a good one :-) • I’d like to do a follow-up at ScaleConf 2014 and share how things are going! Friday 19 April 13

Slide 3

Slide 3 text

Notes: http://goo.gl/9dDnq About our startup • Started Cognician with Barry and Patrick Kayton in early 2010 • Proved the concept over 18 months with good old LAMP and Adobe Flex/AIR • Started V2 app rebuild with Google Closure Javascript late 2011 Friday 19 April 13

Slide 4

Slide 4 text

Notes: http://goo.gl/9dDnq So, why Clojure? • In April 2012, after several months and 11,000 lines of (soul-crushingly verbose) Javascript development, discovered Clojure • Rewrote the model portion of this new app in two weeks and 1500 lines of Clojure • Evaluated it for back end, and decided to go all-in! Friday 19 April 13

Slide 5

Slide 5 text

Notes: http://goo.gl/9dDnq What is Cognician? Friday 19 April 13

Slide 6

Slide 6 text

Notes: http://goo.gl/9dDnq Friday 19 April 13

Slide 7

Slide 7 text

Notes: http://goo.gl/9dDnq Friday 19 April 13

Slide 8

Slide 8 text

Notes: http://goo.gl/9dDnq Friday 19 April 13

Slide 9

Slide 9 text

Notes: http://goo.gl/9dDnq Friday 19 April 13

Slide 10

Slide 10 text

Notes: http://goo.gl/9dDnq Friday 19 April 13

Slide 11

Slide 11 text

Notes: http://goo.gl/9dDnq Friday 19 April 13

Slide 12

Slide 12 text

Notes: http://goo.gl/9dDnq Friday 19 April 13

Slide 13

Slide 13 text

Notes: http://goo.gl/9dDnq Who uses Cognician? Friday 19 April 13

Slide 14

Slide 14 text

Notes: http://goo.gl/9dDnq So. Clojure. Friday 19 April 13

Slide 15

Slide 15 text

Notes: http://goo.gl/9dDnq Clojure •Modern Lisp dialect •General purpose functional language •Immutable data structures •JVM, Javascript, CLI, and more Friday 19 April 13

Slide 16

Slide 16 text

Notes: http://goo.gl/9dDnq Hello ScaleConf! (defn handler [request] {:status 200 :headers {"Content-Type" "text/html"} :body (html [:div [:h1 "Hello ScaleConf! You said:" [:pre (:body request)]]])}) (run-jetty handler {:port 3000}) Friday 19 April 13

Slide 17

Slide 17 text

Notes: http://goo.gl/9dDnq Our Architecture Friday 19 April 13

Slide 18

Slide 18 text

Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13

Slide 19

Slide 19 text

Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13

Slide 20

Slide 20 text

Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13

Slide 21

Slide 21 text

Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13

Slide 22

Slide 22 text

Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13

Slide 23

Slide 23 text

Notes: http://goo.gl/9dDnq Datomic Friday 19 April 13

Slide 24

Slide 24 text

Notes: http://goo.gl/9dDnq Datomic is a database of flexible, time-based facts, supporting queries and joins, with elastic scalability, and ACID transactions. Friday 19 April 13

Slide 25

Slide 25 text

Notes: http://goo.gl/9dDnq How Datomic models data • Entity-attribute-value tuples, or ‘datoms’ • Entities are really just IDs • Any mix of attributes can be put on any entity • Schema defines what attributes can do • Entity relations are just a type of attribute Friday 19 April 13

Slide 26

Slide 26 text

Notes: http://goo.gl/9dDnq Schema definition {:db/doc "User email address" :db/ident :user/email :db/valueType :db.type/string :db/cardinality :db.cardinality/one :db/unique :db.unique/value :db/index true} {:db/doc "User group references" :db/ident :user/groups :db/valueType :db.type/ref :db/cardinality :db.cardinality/many} Friday 19 April 13

Slide 27

Slide 27 text

Notes: http://goo.gl/9dDnq Robert likes beer Friday 19 April 13

Slide 28

Slide 28 text

Notes: http://goo.gl/9dDnq Robert likes beer today Friday 19 April 13

Slide 29

Slide 29 text

Notes: http://goo.gl/9dDnq How Datomic models time • Novelty enters as transactions, of • Assertions and retractions of datoms • Every datom linked to its transaction • Every transaction is timestamped • So, it’s actually EAVT, a.k.a. facts! Friday 19 April 13

Slide 30

Slide 30 text

Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids that have an email address [:find ?entity :where [?entity :user/email]] Friday 19 April 13

Slide 31

Slide 31 text

Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids that have an email address [:find ?entity :where [?entity :user/email]] ; find a user id and password for a given email address [:find ?user ?password :where [?user :user/email "[email protected]"] [?user :user/password ?password]] Friday 19 April 13

Slide 32

Slide 32 text

Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids that have an email address [:find ?entity :where [?entity :user/email]] ; find a user id and password for a given email address [:find ?user ?password :where [?user :user/email "[email protected]"] [?user :user/password ?password]] ; find all the users in all the groups I belong to [:find ?user :in $ ?me [?me :user/groups ?group] [?user :user/groups ?group]] Friday 19 April 13

Slide 33

Slide 33 text

Notes: http://goo.gl/9dDnq Datomic’s Architecture Friday 19 April 13

Slide 34

Slide 34 text

Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13

Slide 35

Slide 35 text

Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13

Slide 36

Slide 36 text

Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13

Slide 37

Slide 37 text

Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13

Slide 38

Slide 38 text

Notes: http://goo.gl/9dDnq In Summary • Event-sourcing and immutable data are a great match • Clojure code is concise and easier to reason about (and has a lot of parenthesis) • Datomic cuts out a lot of the hassle in building scalable, distributed systems Friday 19 April 13

Slide 39

Slide 39 text

Notes: http://goo.gl/9dDnq One more thing! Friday 19 April 13

Slide 40

Slide 40 text

Notes: http://goo.gl/9dDnq Cape Town Clojure User Group •Next meet: 18 May 2013 •Venue: Codebridge in Claremont •Intro to Clojure session •Talks on Datomic, ClojureScript, etc •Details in the notes: http://goo.gl/9dDnq Friday 19 April 13

Slide 41

Slide 41 text

Notes: http://goo.gl/9dDnq Thank you! Friday 19 April 13