Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cognician's new architecture with Clojure and Datomic

Cognician's new architecture with Clojure and Datomic

Slides for a talk I gave on the 19th of April at ScaleConf 2013.

Video: http://www.youtube.com/watch?v=KlDJy3yXVsc

Robert Stuttaford

April 18, 2013
Tweet

More Decks by Robert Stuttaford

Other Decks in Programming

Transcript

  1. Our new architecture
    with Clojure and Datomic
    Robert Stuttaford
    Notes for this talk: http://goo.gl/9dDnq
    Friday 19 April 13

    View Slide

  2. Notes: http://goo.gl/9dDnq
    Before we begin, a
    disclaimer...
    • Launched the rebuild mid Jan 2013.
    • Not running at scale, yet; 1000 users.
    • You’re about to see our strategy for scale.
    We think it’s a good one :-)
    • I’d like to do a follow-up at ScaleConf 2014
    and share how things are going!
    Friday 19 April 13

    View Slide

  3. Notes: http://goo.gl/9dDnq
    About our startup
    • Started Cognician with Barry and
    Patrick Kayton in early 2010
    • Proved the concept over 18 months with
    good old LAMP and Adobe Flex/AIR
    • Started V2 app rebuild with
    Google Closure Javascript late 2011
    Friday 19 April 13

    View Slide

  4. Notes: http://goo.gl/9dDnq
    So, why Clojure?
    • In April 2012, after several months and 11,000
    lines of (soul-crushingly verbose) Javascript
    development, discovered Clojure
    • Rewrote the model portion of this new app in
    two weeks and 1500 lines of Clojure
    • Evaluated it for back end, and decided to go
    all-in!
    Friday 19 April 13

    View Slide

  5. Notes: http://goo.gl/9dDnq
    What is Cognician?
    Friday 19 April 13

    View Slide

  6. Notes: http://goo.gl/9dDnq
    Friday 19 April 13

    View Slide

  7. Notes: http://goo.gl/9dDnq
    Friday 19 April 13

    View Slide

  8. Notes: http://goo.gl/9dDnq
    Friday 19 April 13

    View Slide

  9. Notes: http://goo.gl/9dDnq
    Friday 19 April 13

    View Slide

  10. Notes: http://goo.gl/9dDnq
    Friday 19 April 13

    View Slide

  11. Notes: http://goo.gl/9dDnq
    Friday 19 April 13

    View Slide

  12. Notes: http://goo.gl/9dDnq
    Friday 19 April 13

    View Slide

  13. Notes: http://goo.gl/9dDnq
    Who uses Cognician?
    Friday 19 April 13

    View Slide

  14. Notes: http://goo.gl/9dDnq
    So. Clojure.
    Friday 19 April 13

    View Slide

  15. Notes: http://goo.gl/9dDnq
    Clojure
    •Modern Lisp dialect
    •General purpose functional language
    •Immutable data structures
    •JVM, Javascript, CLI, and more
    Friday 19 April 13

    View Slide

  16. Notes: http://goo.gl/9dDnq
    Hello ScaleConf!
    (defn handler [request]
    {:status 200
    :headers {"Content-Type" "text/html"}
    :body (html
    [:div
    [:h1 "Hello ScaleConf! You said:"
    [:pre (:body request)]]])})
    (run-jetty handler {:port 3000})
    Friday 19 April 13

    View Slide

  17. Notes: http://goo.gl/9dDnq
    Our Architecture
    Friday 19 April 13

    View Slide

  18. Notes: http://goo.gl/9dDnq
    Cognician app
    SERVER SIDE
    Web nodes (2)
    AWS ELB
    AWS S3
    Datomic
    Transactor
    Datomic Storage
    (Postgres)
    Worker nodes (1)
    Our code
    Self-hosted Services
    Memcached
    RabbitMQ
    Managed Services
    CLIENT SIDE
    Cognician’s Architecture
    Friday 19 April 13

    View Slide

  19. Notes: http://goo.gl/9dDnq
    Cognician app
    SERVER SIDE
    Web nodes (2)
    AWS ELB
    AWS S3
    Datomic
    Transactor
    Datomic Storage
    (Postgres)
    Worker nodes (1)
    Our code
    Self-hosted Services
    Memcached
    RabbitMQ
    Managed Services
    CLIENT SIDE
    Cognician’s Architecture
    Friday 19 April 13

    View Slide

  20. Notes: http://goo.gl/9dDnq
    Cognician app
    SERVER SIDE
    Web nodes (2)
    AWS ELB
    AWS S3
    Datomic
    Transactor
    Datomic Storage
    (Postgres)
    Worker nodes (1)
    Our code
    Self-hosted Services
    Memcached
    RabbitMQ
    Managed Services
    CLIENT SIDE
    Cognician’s Architecture
    Friday 19 April 13

    View Slide

  21. Notes: http://goo.gl/9dDnq
    Cognician app
    SERVER SIDE
    Web nodes (2)
    AWS ELB
    AWS S3
    Datomic
    Transactor
    Datomic Storage
    (Postgres)
    Worker nodes (1)
    Our code
    Self-hosted Services
    Memcached
    RabbitMQ
    Managed Services
    CLIENT SIDE
    Cognician’s Architecture
    Friday 19 April 13

    View Slide

  22. Notes: http://goo.gl/9dDnq
    Cognician app
    SERVER SIDE
    Web nodes (2)
    AWS ELB
    AWS S3
    Datomic
    Transactor
    Datomic Storage
    (Postgres)
    Worker nodes (1)
    Our code
    Self-hosted Services
    Memcached
    RabbitMQ
    Managed Services
    CLIENT SIDE
    Cognician’s Architecture
    Friday 19 April 13

    View Slide

  23. Notes: http://goo.gl/9dDnq
    Datomic
    Friday 19 April 13

    View Slide

  24. Notes: http://goo.gl/9dDnq
    Datomic is a database of
    flexible, time-based facts,
    supporting queries and joins,
    with elastic scalability,
    and ACID transactions.
    Friday 19 April 13

    View Slide

  25. Notes: http://goo.gl/9dDnq
    How Datomic models data
    • Entity-attribute-value tuples, or ‘datoms’
    • Entities are really just IDs
    • Any mix of attributes can be put on any entity
    • Schema defines what attributes can do
    • Entity relations are just a type of attribute
    Friday 19 April 13

    View Slide

  26. Notes: http://goo.gl/9dDnq
    Schema definition
    {:db/doc "User email address"
    :db/ident :user/email
    :db/valueType :db.type/string
    :db/cardinality :db.cardinality/one
    :db/unique :db.unique/value
    :db/index true}
    {:db/doc "User group references"
    :db/ident :user/groups
    :db/valueType :db.type/ref
    :db/cardinality :db.cardinality/many}
    Friday 19 April 13

    View Slide

  27. Notes: http://goo.gl/9dDnq
    Robert likes beer
    Friday 19 April 13

    View Slide

  28. Notes: http://goo.gl/9dDnq
    Robert likes beer today
    Friday 19 April 13

    View Slide

  29. Notes: http://goo.gl/9dDnq
    How Datomic models time
    • Novelty enters as transactions, of
    • Assertions and retractions of datoms
    • Every datom linked to its transaction
    • Every transaction is timestamped
    • So, it’s actually EAVT, a.k.a. facts!
    Friday 19 April 13

    View Slide

  30. Notes: http://goo.gl/9dDnq
    Query with Datalog
    ; find all entity ids that have an email address
    [:find ?entity :where
    [?entity :user/email]]
    Friday 19 April 13

    View Slide

  31. Notes: http://goo.gl/9dDnq
    Query with Datalog
    ; find all entity ids that have an email address
    [:find ?entity :where
    [?entity :user/email]]
    ; find a user id and password for a given email address
    [:find ?user ?password :where
    [?user :user/email "[email protected]"]
    [?user :user/password ?password]]
    Friday 19 April 13

    View Slide

  32. Notes: http://goo.gl/9dDnq
    Query with Datalog
    ; find all entity ids that have an email address
    [:find ?entity :where
    [?entity :user/email]]
    ; find a user id and password for a given email address
    [:find ?user ?password :where
    [?user :user/email "[email protected]"]
    [?user :user/password ?password]]
    ; find all the users in all the groups I belong to
    [:find ?user :in $ ?me
    [?me :user/groups ?group]
    [?user :user/groups ?group]]
    Friday 19 April 13

    View Slide

  33. Notes: http://goo.gl/9dDnq
    Datomic’s
    Architecture
    Friday 19 April 13

    View Slide

  34. Notes: http://goo.gl/9dDnq
    Cognician app
    SERVER SIDE
    Web nodes (2)
    AWS ELB
    AWS S3
    Datomic
    Transactor
    Datomic Storage
    (Postgres)
    Worker nodes (1)
    Our code
    Self-hosted Services
    Memcached
    RabbitMQ
    Managed Services
    CLIENT SIDE
    Cognician’s Architecture
    Friday 19 April 13

    View Slide

  35. Notes: http://goo.gl/9dDnq
    Cognician app
    SERVER SIDE
    Web nodes (2)
    AWS ELB
    AWS S3
    Datomic
    Transactor
    Datomic Storage
    (Postgres)
    Worker nodes (1)
    Our code
    Self-hosted Services
    Memcached
    RabbitMQ
    Managed Services
    CLIENT SIDE
    Cognician’s Architecture
    Friday 19 April 13

    View Slide

  36. Notes: http://goo.gl/9dDnq
    Cognician app
    SERVER SIDE
    Web nodes (2)
    AWS ELB
    AWS S3
    Datomic
    Transactor
    Datomic Storage
    (Postgres)
    Worker nodes (1)
    Our code
    Self-hosted Services
    Memcached
    RabbitMQ
    Managed Services
    CLIENT SIDE
    Cognician’s Architecture
    Friday 19 April 13

    View Slide

  37. Notes: http://goo.gl/9dDnq
    Cognician app
    SERVER SIDE
    Web nodes (2)
    AWS ELB
    AWS S3
    Datomic
    Transactor
    Datomic Storage
    (Postgres)
    Worker nodes (1)
    Our code
    Self-hosted Services
    Memcached
    RabbitMQ
    Managed Services
    CLIENT SIDE
    Cognician’s Architecture
    Friday 19 April 13

    View Slide

  38. Notes: http://goo.gl/9dDnq
    In Summary
    • Event-sourcing and immutable data are a great match
    • Clojure code is concise and easier to reason about
    (and has a lot of parenthesis)
    • Datomic cuts out a lot of the hassle in building scalable,
    distributed systems
    Friday 19 April 13

    View Slide

  39. Notes: http://goo.gl/9dDnq
    One more thing!
    Friday 19 April 13

    View Slide

  40. Notes: http://goo.gl/9dDnq
    Cape Town Clojure User Group
    •Next meet: 18 May 2013
    •Venue: Codebridge in Claremont
    •Intro to Clojure session
    •Talks on Datomic, ClojureScript, etc
    •Details in the notes:
    http://goo.gl/9dDnq
    Friday 19 April 13

    View Slide

  41. Notes: http://goo.gl/9dDnq
    Thank you!
    Friday 19 April 13

    View Slide