Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cognician's new architecture with Clojure and Datomic

Cognician's new architecture with Clojure and Datomic

Slides for a talk I gave on the 19th of April at ScaleConf 2013.

Video: http://www.youtube.com/watch?v=KlDJy3yXVsc

Robert Stuttaford

April 18, 2013
Tweet

More Decks by Robert Stuttaford

Other Decks in Programming

Transcript

  1. Our new architecture with Clojure and Datomic Robert Stuttaford Notes

    for this talk: http://goo.gl/9dDnq Friday 19 April 13
  2. Notes: http://goo.gl/9dDnq Before we begin, a disclaimer... • Launched the

    rebuild mid Jan 2013. • Not running at scale, yet; 1000 users. • You’re about to see our strategy for scale. We think it’s a good one :-) • I’d like to do a follow-up at ScaleConf 2014 and share how things are going! Friday 19 April 13
  3. Notes: http://goo.gl/9dDnq About our startup • Started Cognician with Barry

    and Patrick Kayton in early 2010 • Proved the concept over 18 months with good old LAMP and Adobe Flex/AIR • Started V2 app rebuild with Google Closure Javascript late 2011 Friday 19 April 13
  4. Notes: http://goo.gl/9dDnq So, why Clojure? • In April 2012, after

    several months and 11,000 lines of (soul-crushingly verbose) Javascript development, discovered Clojure • Rewrote the model portion of this new app in two weeks and 1500 lines of Clojure • Evaluated it for back end, and decided to go all-in! Friday 19 April 13
  5. Notes: http://goo.gl/9dDnq Clojure •Modern Lisp dialect •General purpose functional language

    •Immutable data structures •JVM, Javascript, CLI, and more Friday 19 April 13
  6. Notes: http://goo.gl/9dDnq Hello ScaleConf! (defn handler [request] {:status 200 :headers

    {"Content-Type" "text/html"} :body (html [:div [:h1 "Hello ScaleConf! You said:" [:pre (:body request)]]])}) (run-jetty handler {:port 3000}) Friday 19 April 13
  7. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  8. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  9. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  10. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  11. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  12. Notes: http://goo.gl/9dDnq Datomic is a database of flexible, time-based facts,

    supporting queries and joins, with elastic scalability, and ACID transactions. Friday 19 April 13
  13. Notes: http://goo.gl/9dDnq How Datomic models data • Entity-attribute-value tuples, or

    ‘datoms’ • Entities are really just IDs • Any mix of attributes can be put on any entity • Schema defines what attributes can do • Entity relations are just a type of attribute Friday 19 April 13
  14. Notes: http://goo.gl/9dDnq Schema definition {:db/doc "User email address" :db/ident :user/email

    :db/valueType :db.type/string :db/cardinality :db.cardinality/one :db/unique :db.unique/value :db/index true} {:db/doc "User group references" :db/ident :user/groups :db/valueType :db.type/ref :db/cardinality :db.cardinality/many} Friday 19 April 13
  15. Notes: http://goo.gl/9dDnq How Datomic models time • Novelty enters as

    transactions, of • Assertions and retractions of datoms • Every datom linked to its transaction • Every transaction is timestamped • So, it’s actually EAVT, a.k.a. facts! Friday 19 April 13
  16. Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] Friday 19 April 13
  17. Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] ; find a user id and password for a given email address [:find ?user ?password :where [?user :user/email "[email protected]"] [?user :user/password ?password]] Friday 19 April 13
  18. Notes: http://goo.gl/9dDnq Query with Datalog ; find all entity ids

    that have an email address [:find ?entity :where [?entity :user/email]] ; find a user id and password for a given email address [:find ?user ?password :where [?user :user/email "[email protected]"] [?user :user/password ?password]] ; find all the users in all the groups I belong to [:find ?user :in $ ?me [?me :user/groups ?group] [?user :user/groups ?group]] Friday 19 April 13
  19. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  20. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  21. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  22. Notes: http://goo.gl/9dDnq Cognician app SERVER SIDE Web nodes (2) AWS

    ELB AWS S3 Datomic Transactor Datomic Storage (Postgres) Worker nodes (1) Our code Self-hosted Services Memcached RabbitMQ Managed Services CLIENT SIDE Cognician’s Architecture Friday 19 April 13
  23. Notes: http://goo.gl/9dDnq In Summary • Event-sourcing and immutable data are

    a great match • Clojure code is concise and easier to reason about (and has a lot of parenthesis) • Datomic cuts out a lot of the hassle in building scalable, distributed systems Friday 19 April 13
  24. Notes: http://goo.gl/9dDnq Cape Town Clojure User Group •Next meet: 18

    May 2013 •Venue: Codebridge in Claremont •Intro to Clojure session •Talks on Datomic, ClojureScript, etc •Details in the notes: http://goo.gl/9dDnq Friday 19 April 13