Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cognician's new architecture with Clojure and Datomic

Cognician's new architecture with Clojure and Datomic

Slides for a talk I gave on the 19th of April at ScaleConf 2013.

Video: http://www.youtube.com/watch?v=KlDJy3yXVsc

96857e2f67b65f97825d2de491a17808?s=128

Robert Stuttaford

April 18, 2013
Tweet

More Decks by Robert Stuttaford

Other Decks in Programming

Transcript

  1. Our new architecture with Clojure and Datomic Robert Stuttaford Notes

    for this talk: http://goo.gl/9dDnq Friday 19 April 13
  2. Notes: http://goo.gl/9dDnq Before we begin, a disclaimer... • Launched the

    rebuild mid Jan 2013. • Not running at scale, yet; 1000 users. • You’re about to see our strategy for scale. We think it’s a good one :-) • I’d like to do a follow-up at ScaleConf 2014 and share how things are going! Friday 19 April 13
  3. Notes: http://goo.gl/9dDnq About our startup • Started Cognician with Barry

    and Patrick Kayton in early 2010 • Proved the concept over 18 months with good old LAMP and Adobe Flex/AIR • Started V2 app rebuild with Google Closure Javascript late 2011 Friday 19 April 13
  4. Notes: http://goo.gl/9dDnq So, why Clojure? • In April 2012, after

    several months and 11,000 lines of (soul-crushingly verbose) Javascript development, discovered Clojure • Rewrote the model portion of this new app in two weeks and 1500 lines of Clojure • Evaluated it for back end, and decided to go all-in! Friday 19 April 13
  5. Notes: http://goo.gl/9dDnq What is Cognician? Friday 19 April 13

  6. Notes: http://goo.gl/9dDnq Friday 19 April 13

  7. Notes: http://goo.gl/9dDnq Friday 19 April 13

  8. Notes: http://goo.gl/9dDnq Friday 19 April 13

  9. Notes: http://goo.gl/9dDnq Friday 19 April 13

  10. Notes: http://goo.gl/9dDnq Friday 19 April 13

  11. Notes: http://goo.gl/9dDnq Friday 19 April 13

  12. Notes: http://goo.gl/9dDnq Friday 19 April 13

  13. Notes: http://goo.gl/9dDnq Who uses Cognician? Friday 19 April 13

  14. Notes: http://goo.gl/9dDnq So. Clojure. Friday 19 April 13

  15. Notes: http://goo.gl/9dDnq Clojure •Modern Lisp dialect •General purpose functional language

    •Immutable data structures •JVM, Javascript, CLI, and more Friday 19 April 13
  16. Notes: http://goo.gl/9dDnq Hello ScaleConf! (defn handler [request] {:status 200 :headers

    {"Content-Type" "text/html"} :body (html [:div [:h1 "Hello ScaleConf! You said:" [:pre (:body request)]]])}) (run-jetty handler {:port 3000}) Friday 19 April 13
  17. Notes: http://goo.gl/9dDnq Our Architecture Friday 19 April 13

  18. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  19. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  20. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  21. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  22. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  23. Notes: http://goo.gl/9dDnq Datomic Friday 19 April 13

  24. Notes: http://goo.gl/9dDnq Datomic is a database of flexible, time-based facts,

    supporting queries and joins, with elastic scalability, and ACID transactions. Friday 19 April 13
  25. Notes: http://goo.gl/9dDnq How Datomic models data • Entity-attribute-value tuples, or

    ‘datoms’ • Entities are really just IDs • Any mix of attributes can be put on any entity • Schema defines what attributes can do • Entity relations are just a type of attribute Friday 19 April 13
  26. Notes: http://goo.gl/9dDnq Schema definition {:db/doc "User email address" :db/ident :user/email

    :db/valueType :db.type/string :db/cardinality :db.cardinality/one :db/unique :db.unique/value :db/index true} {:db/doc "User group references" :db/ident :user/groups :db/valueType :db.type/ref :db/cardinality :db.cardinality/many} Friday 19 April 13
  27. Notes: http://goo.gl/9dDnq Robert likes beer Friday 19 April 13

  28. Notes: http://goo.gl/9dDnq Robert likes beer today Friday 19 April 13

  29. Notes: http://goo.gl/9dDnq How Datomic models time • Novelty enters as

    transactions, of • Assertions and retractions of datoms • Every datom linked to its transaction • Every transaction is timestamped • So, it’s actually EAVT, a.k.a. facts! Friday 19 April 13
  30. Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] Friday 19 April 13
  31. Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] ; find a user id and password for a given email address [:find ?user ?password :where [?user :user/email "robert@cognician.com"] [?user :user/password ?password]] Friday 19 April 13
  32. Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] ; find a user id and password for a given email address [:find ?user ?password :where [?user :user/email "robert@cognician.com"] [?user :user/password ?password]] ; find all the users in all the groups I belong to [:find ?user :in $ ?me [?me :user/groups ?group] [?user :user/groups ?group]] Friday 19 April 13
  33. Notes: http://goo.gl/9dDnq Datomic’s Architecture Friday 19 April 13

  34. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  35. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  36. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  37. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  38. Notes: http://goo.gl/9dDnq In Summary • Event-sourcing and immutable data are

    a great match • Clojure code is concise and easier to reason about (and has a lot of parenthesis) • Datomic cuts out a lot of the hassle in building scalable, distributed systems Friday 19 April 13
  39. Notes: http://goo.gl/9dDnq One more thing! Friday 19 April 13

  40. Notes: http://goo.gl/9dDnq Cape Town Clojure User Group •Next meet: 18

    May 2013 •Venue: Codebridge in Claremont •Intro to Clojure session •Talks on Datomic, ClojureScript, etc •Details in the notes: http://goo.gl/9dDnq Friday 19 April 13
  41. Notes: http://goo.gl/9dDnq Thank you! Friday 19 April 13