Cognician's new architecture with Clojure and Datomic

Cognician's new architecture with Clojure and Datomic

Slides for a talk I gave on the 19th of April at ScaleConf 2013.

Video: http://www.youtube.com/watch?v=KlDJy3yXVsc

96857e2f67b65f97825d2de491a17808?s=128

Robert Stuttaford

April 18, 2013
Tweet

Transcript

  1. 1.

    Our new architecture with Clojure and Datomic Robert Stuttaford Notes

    for this talk: http://goo.gl/9dDnq Friday 19 April 13
  2. 2.

    Notes: http://goo.gl/9dDnq Before we begin, a disclaimer... • Launched the

    rebuild mid Jan 2013. • Not running at scale, yet; 1000 users. • You’re about to see our strategy for scale. We think it’s a good one :-) • I’d like to do a follow-up at ScaleConf 2014 and share how things are going! Friday 19 April 13
  3. 3.

    Notes: http://goo.gl/9dDnq About our startup • Started Cognician with Barry

    and Patrick Kayton in early 2010 • Proved the concept over 18 months with good old LAMP and Adobe Flex/AIR • Started V2 app rebuild with Google Closure Javascript late 2011 Friday 19 April 13
  4. 4.

    Notes: http://goo.gl/9dDnq So, why Clojure? • In April 2012, after

    several months and 11,000 lines of (soul-crushingly verbose) Javascript development, discovered Clojure • Rewrote the model portion of this new app in two weeks and 1500 lines of Clojure • Evaluated it for back end, and decided to go all-in! Friday 19 April 13
  5. 15.

    Notes: http://goo.gl/9dDnq Clojure •Modern Lisp dialect •General purpose functional language

    •Immutable data structures •JVM, Javascript, CLI, and more Friday 19 April 13
  6. 16.

    Notes: http://goo.gl/9dDnq Hello ScaleConf! (defn handler [request] {:status 200 :headers

    {"Content-Type" "text/html"} :body (html [:div [:h1 "Hello ScaleConf! You said:" [:pre (:body request)]]])}) (run-jetty handler {:port 3000}) Friday 19 April 13
  7. 18.

    Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  8. 19.

    Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  9. 20.

    Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  10. 21.

    Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  11. 22.

    Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  12. 24.

    Notes: http://goo.gl/9dDnq Datomic is a database of flexible, time-based facts,

    supporting queries and joins, with elastic scalability, and ACID transactions. Friday 19 April 13
  13. 25.

    Notes: http://goo.gl/9dDnq How Datomic models data • Entity-attribute-value tuples, or

    ‘datoms’ • Entities are really just IDs • Any mix of attributes can be put on any entity • Schema defines what attributes can do • Entity relations are just a type of attribute Friday 19 April 13
  14. 26.

    Notes: http://goo.gl/9dDnq Schema definition {:db/doc "User email address" :db/ident :user/email

    :db/valueType :db.type/string :db/cardinality :db.cardinality/one :db/unique :db.unique/value :db/index true} {:db/doc "User group references" :db/ident :user/groups :db/valueType :db.type/ref :db/cardinality :db.cardinality/many} Friday 19 April 13
  15. 29.

    Notes: http://goo.gl/9dDnq How Datomic models time • Novelty enters as

    transactions, of • Assertions and retractions of datoms • Every datom linked to its transaction • Every transaction is timestamped • So, it’s actually EAVT, a.k.a. facts! Friday 19 April 13
  16. 30.

    Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] Friday 19 April 13
  17. 31.

    Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] ; find a user id and password for a given email address [:find ?user ?password :where [?user :user/email "robert@cognician.com"] [?user :user/password ?password]] Friday 19 April 13
  18. 32.

    Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] ; find a user id and password for a given email address [:find ?user ?password :where [?user :user/email "robert@cognician.com"] [?user :user/password ?password]] ; find all the users in all the groups I belong to [:find ?user :in $ ?me [?me :user/groups ?group] [?user :user/groups ?group]] Friday 19 April 13
  19. 34.

    Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  20. 35.

    Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  21. 36.

    Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  22. 37.

    Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  23. 38.

    Notes: http://goo.gl/9dDnq In Summary • Event-sourcing and immutable data are

    a great match • Clojure code is concise and easier to reason about (and has a lot of parenthesis) • Datomic cuts out a lot of the hassle in building scalable, distributed systems Friday 19 April 13
  24. 40.

    Notes: http://goo.gl/9dDnq Cape Town Clojure User Group •Next meet: 18

    May 2013 •Venue: Codebridge in Claremont •Intro to Clojure session •Talks on Datomic, ClojureScript, etc •Details in the notes: http://goo.gl/9dDnq Friday 19 April 13