Clojure Redeployed

Clojure Redeployed

Slides from a talk I delivered on 28.03.2015 at LambdaCon in Bologna, Italy.

At stylefruits we've been using Clojure on production since late 2012. We've made a lot of mistakes and we've learned a lot. This talk is dedicated to the newest addition to our infrastructure: a Clojure REST-inspired API. Taught by our previous mistakes in the realm of deployment and scalability we've set our requirements high: we're aiming for continuous delivery, high availability, dynamic scaling and pauseless deployments under load. Aside from deployment matters, Clojure proves to be an excellent choice to build a modular and stateless service. In this talk we're going to share our experiences, both good and bad, gained while prototyping, deploying and maintaining our newest project.

Video: https://vimeo.com/139254964

Ae7a42fb716793697b1d222f3cc753b8?s=128

Jan Stępień

March 28, 2015
Tweet

Transcript

  1. Clojure Redeployed Jan Stępień @janstepien jan@stepien.cc

  2. © iwillbehomesoon 2012, flickr.com/photos/jp_photo_online/6964634964

  3. I work at

  4. I work at

  5. None
  6. None
  7. https://api.stylefruits.de Elasticsearch MySQL

  8. https://api.stylefruits.de Elasticsearch MySQL So how about...

  9. https://api.stylefruits.de Elasticsearch MySQL So how about... Clojure?

  10. Expectations set high ▶ short feedback loop ▶ continuous delivery

    ▶ automated and pauseless deployment ▶ ...under load ▶ high availability, dynamic scaling
  11. we’ve called it Ogrom* * plethora, immensity, enormity

  12. Outline 1. Architecture 2. Deployment and Operations 3. Lessons Learned

  13. Architecture to use a bold word

  14. sources protocols run core

  15. protocols define interfaces (defprotocol CollageSource (collage-by-id [ctx collage-id]))

  16. sources wrap data sources (reify CollageSource (collage-by-id [ctx collage-id] (fetch-from-mysql

    ctx collage-id)))
  17. sources protocols run core

  18. core defines HTTP endpoints each core endpoint ▶ is independent

    ▶ has all dependencies injected ▶ is absolute-path-agnostic ▶ is a value
  19. core uses Liberator (defresource product :exists? find-product-or-return-nil :handle-ok render-product :handle-not-found

    render-error-message)
  20. :exists? :handle-ok 200 true :handle-not-found 404 false

  21. core uses bidi (def routes ["/" {"product" product "ratings" ratings}])

  22. sources protocols run core

  23. run wires everything together products GET /products POST /comments comments

    GET /comments users
  24. /v1 /de-DE GET /products GET /comments POST /comments

  25. /v1 /de-DE GET /products GET /comments POST /comments GET /v1/de-DE/comments

  26. routing GET /comments POST /comments GET /products users third-party API

    MySQL products Elasticsearch comments
  27. run instantiates a self-contained service MySQL Elasticsearch logstash 3rd-party API

    ogrom
  28. MySQL Elasticsearch logstash 3rd-party API ogrom MySQL Elasticsearch logstash 3rd-party

    API ogrom MySQL Elasticsearch logstash 3rd-party API ogrom MySQL Elasticsearch logstash 3rd-party API ogrom
  29. /v1 /fr-FR /nl-NL /de-DE /en-GB GET /products GET /comments POST

    /comments
  30. MySQL Elasticsearch logstash 3rd-party API ogrom

  31. Deployment and Operations

  32. Deployment used to be cumbersome 1. Create an überjar 2.

    Upload to n boxes 3. Restart processes 4. Juggle load balancers
  33. sources protocols run core ▶ sources, protocols, core are sem-versioned

    ▶ run is deployed straight from master
  34. sources run protocols core CI on every push to master

    S3 depl. tool überjar commit id
  35. Elastic Beanstalk and its environments ▶ EB wraps EC2 and

    ELB ▶ environments with instances ▶ runs Docker containers
  36. deployment tool AWS API CI commit id box 1 box

    2 S3 (überjar)
  37. api.stylefruits.de Beanstalk 1 Beanstalk 2

  38. api.stylefruits.de CloudFront Beanstalk 1 Beanstalk 2

  39. Lessons Learned and Future Work

  40. Feedback loops are shorter ▶ Productive development environment ▶ Bug

    fixes and features can be deploy the same morning they were asked for
  41. Splitting wasn’t strictly necessary sources protocols run core

  42. Docker and empowered developers ▶ Developers are in control over

    deployment ▶ Reduced dependency on the ops team ▶ Production env ≈ development env
  43. There are some trade-o s ▶ Beanstalk allows opening only

    a single port when running Docker containers ▶ Deployment under load vs. JIT compilation ▶ The deployment takes a while: Clojure compile times, JVM start-up time, etc. ▶ There’s no staging environment
  44. Ogrom is an improvement ▶ Client teams are happy with

    it ▶ It performs well under load and is durable ▶ Our main web app uses the Ogrom too now
  45. Plans for the future ▶ Ring Swagger ▶ Clojure 1.7

    and its reduced compilation time ▶ Clojure(Script) client API
  46. None
  47. Clojure Redeployed Jan Stępień @janstepien jan@stepien.cc