Cognician's new architecture with Clojure and Datomic

Cognician's new architecture with Clojure and Datomic

Slides for a talk I gave on the 19th of April at ScaleConf 2013.

Video: http://www.youtube.com/watch?v=KlDJy3yXVsc

96857e2f67b65f97825d2de491a17808?s=128

Robert Stuttaford

April 18, 2013
Tweet

Transcript

  1. Our new architecture with Clojure and Datomic Robert Stuttaford Notes

    for this talk: http://goo.gl/9dDnq Friday 19 April 13
  2. Notes: http://goo.gl/9dDnq Before we begin, a disclaimer... • Launched the

    rebuild mid Jan 2013. • Not running at scale, yet; 1000 users. • You’re about to see our strategy for scale. We think it’s a good one :-) • I’d like to do a follow-up at ScaleConf 2014 and share how things are going! Friday 19 April 13
  3. Notes: http://goo.gl/9dDnq About our startup • Started Cognician with Barry

    and Patrick Kayton in early 2010 • Proved the concept over 18 months with good old LAMP and Adobe Flex/AIR • Started V2 app rebuild with Google Closure Javascript late 2011 Friday 19 April 13
  4. Notes: http://goo.gl/9dDnq So, why Clojure? • In April 2012, after

    several months and 11,000 lines of (soul-crushingly verbose) Javascript development, discovered Clojure • Rewrote the model portion of this new app in two weeks and 1500 lines of Clojure • Evaluated it for back end, and decided to go all-in! Friday 19 April 13
  5. Notes: http://goo.gl/9dDnq What is Cognician? Friday 19 April 13

  6. Notes: http://goo.gl/9dDnq Friday 19 April 13

  7. Notes: http://goo.gl/9dDnq Friday 19 April 13

  8. Notes: http://goo.gl/9dDnq Friday 19 April 13

  9. Notes: http://goo.gl/9dDnq Friday 19 April 13

  10. Notes: http://goo.gl/9dDnq Friday 19 April 13

  11. Notes: http://goo.gl/9dDnq Friday 19 April 13

  12. Notes: http://goo.gl/9dDnq Friday 19 April 13

  13. Notes: http://goo.gl/9dDnq Who uses Cognician? Friday 19 April 13

  14. Notes: http://goo.gl/9dDnq So. Clojure. Friday 19 April 13

  15. Notes: http://goo.gl/9dDnq Clojure •Modern Lisp dialect •General purpose functional language

    •Immutable data structures •JVM, Javascript, CLI, and more Friday 19 April 13
  16. Notes: http://goo.gl/9dDnq Hello ScaleConf! (defn handler [request] {:status 200 :headers

    {"Content-Type" "text/html"} :body (html [:div [:h1 "Hello ScaleConf! You said:" [:pre (:body request)]]])}) (run-jetty handler {:port 3000}) Friday 19 April 13
  17. Notes: http://goo.gl/9dDnq Our Architecture Friday 19 April 13

  18. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  19. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  20. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  21. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  22. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  23. Notes: http://goo.gl/9dDnq Datomic Friday 19 April 13

  24. Notes: http://goo.gl/9dDnq Datomic is a database of flexible, time-based facts,

    supporting queries and joins, with elastic scalability, and ACID transactions. Friday 19 April 13
  25. Notes: http://goo.gl/9dDnq How Datomic models data • Entity-attribute-value tuples, or

    ‘datoms’ • Entities are really just IDs • Any mix of attributes can be put on any entity • Schema defines what attributes can do • Entity relations are just a type of attribute Friday 19 April 13
  26. Notes: http://goo.gl/9dDnq Schema definition {:db/doc "User email address" :db/ident :user/email

    :db/valueType :db.type/string :db/cardinality :db.cardinality/one :db/unique :db.unique/value :db/index true} {:db/doc "User group references" :db/ident :user/groups :db/valueType :db.type/ref :db/cardinality :db.cardinality/many} Friday 19 April 13
  27. Notes: http://goo.gl/9dDnq Robert likes beer Friday 19 April 13

  28. Notes: http://goo.gl/9dDnq Robert likes beer today Friday 19 April 13

  29. Notes: http://goo.gl/9dDnq How Datomic models time • Novelty enters as

    transactions, of • Assertions and retractions of datoms • Every datom linked to its transaction • Every transaction is timestamped • So, it’s actually EAVT, a.k.a. facts! Friday 19 April 13
  30. Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] Friday 19 April 13
  31. Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] ; find a user id and password for a given email address [:find ?user ?password :where [?user :user/email "robert@cognician.com"] [?user :user/password ?password]] Friday 19 April 13
  32. Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] ; find a user id and password for a given email address [:find ?user ?password :where [?user :user/email "robert@cognician.com"] [?user :user/password ?password]] ; find all the users in all the groups I belong to [:find ?user :in $ ?me [?me :user/groups ?group] [?user :user/groups ?group]] Friday 19 April 13
  33. Notes: http://goo.gl/9dDnq Datomic’s Architecture Friday 19 April 13

  34. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  35. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  36. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  37. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  38. Notes: http://goo.gl/9dDnq In Summary • Event-sourcing and immutable data are

    a great match • Clojure code is concise and easier to reason about (and has a lot of parenthesis) • Datomic cuts out a lot of the hassle in building scalable, distributed systems Friday 19 April 13
  39. Notes: http://goo.gl/9dDnq One more thing! Friday 19 April 13

  40. Notes: http://goo.gl/9dDnq Cape Town Clojure User Group •Next meet: 18

    May 2013 •Venue: Codebridge in Claremont •Intro to Clojure session •Talks on Datomic, ClojureScript, etc •Details in the notes: http://goo.gl/9dDnq Friday 19 April 13
  41. Notes: http://goo.gl/9dDnq Thank you! Friday 19 April 13