Clojure Redeployed

Slides from a talk I delivered on 28.03.2015 at LambdaCon in Bologna, Italy.

At stylefruits we've been using Clojure on production since late 2012. We've made a lot of mistakes and we've learned a lot. This talk is dedicated to the newest addition to our infrastructure: a Clojure REST-inspired API. Taught by our previous mistakes in the realm of deployment and scalability we've set our requirements high: we're aiming for continuous delivery, high availability, dynamic scaling and pauseless deployments under load. Aside from deployment matters, Clojure proves to be an excellent choice to build a modular and stateless service. In this talk we're going to share our experiences, both good and bad, gained while prototyping, deploying and maintaining our newest project.

Video: https://vimeo.com/139254964

Transcript

1. Clojure Redeployed Jan Stępień @janstepien jan@stepien.cc
2. © iwillbehomesoon 2012, flickr.com/photos/jp_photo_online/6964634964
3. I work at
4. I work at
7. https://api.stylefruits.de Elasticsearch MySQL
8. https://api.stylefruits.de Elasticsearch MySQL So how about...
9. https://api.stylefruits.de Elasticsearch MySQL So how about... Clojure?
10. Expectations set high ▶ short feedback loop ▶ continuous delivery ▶ automated and pauseless deployment ▶ ...under load ▶ high availability, dynamic scaling
11. we’ve called it Ogrom* * plethora, immensity, enormity
12. Outline 1. Architecture 2. Deployment and Operations 3. Lessons Learned
13. Architecture to use a bold word
14. sources protocols run core
15. protocols define interfaces (defprotocol CollageSource (collage-by-id [ctx collage-id]))
16. sources wrap data sources (reify CollageSource (collage-by-id [ctx collage-id] (fetch-from-mysql ctx collage-id)))
17. sources protocols run core
18. core defines HTTP endpoints each core endpoint ▶ is independent ▶ has all dependencies injected ▶ is absolute-path-agnostic ▶ is a value
19. core uses Liberator (defresource product :exists? find-product-or-return-nil :handle-ok render-product :handle-not-found render-error-message)
20. :exists? :handle-ok 200 true :handle-not-found 404 false
21. core uses bidi (def routes ["/" {"product" product "ratings" ratings}])
22. sources protocols run core
23. run wires everything together products GET /products POST /comments comments GET /comments users
24. /v1 /de-DE GET /products GET /comments POST /comments
25. /v1 /de-DE GET /products GET /comments POST /comments GET /v1/de-DE/comments
26. routing GET /comments POST /comments GET /products users third-party API MySQL products Elasticsearch comments
27. run instantiates a self-contained service MySQL Elasticsearch logstash 3rd-party API ogrom
28. MySQL Elasticsearch logstash 3rd-party API ogrom MySQL Elasticsearch logstash 3rd-party API ogrom MySQL Elasticsearch logstash 3rd-party API ogrom MySQL Elasticsearch logstash 3rd-party API ogrom
29. /v1 /fr-FR /nl-NL /de-DE /en-GB GET /products GET /comments POST /comments
30. MySQL Elasticsearch logstash 3rd-party API ogrom
31. Deployment and Operations
32. Deployment used to be cumbersome 1. Create an überjar 2. Upload to n boxes 3. Restart processes 4. Juggle load balancers
33. sources protocols run core ▶ sources, protocols, core are sem-versioned ▶ run is deployed straight from master
34. sources run protocols core CI on every push to master S3 depl. tool überjar commit id
35. Elastic Beanstalk and its environments ▶ EB wraps EC2 and ELB ▶ environments with instances ▶ runs Docker containers
36. deployment tool AWS API CI commit id box 1 box 2 S3 (überjar)
37. api.stylefruits.de Beanstalk 1 Beanstalk 2
38. api.stylefruits.de CloudFront Beanstalk 1 Beanstalk 2
39. Lessons Learned and Future Work
40. Feedback loops are shorter ▶ Productive development environment ▶ Bug fixes and features can be deploy the same morning they were asked for
41. Splitting wasn’t strictly necessary sources protocols run core
42. Docker and empowered developers ▶ Developers are in control over deployment ▶ Reduced dependency on the ops team ▶ Production env ≈ development env
43. There are some trade-o s ▶ Beanstalk allows opening only a single port when running Docker containers ▶ Deployment under load vs. JIT compilation ▶ The deployment takes a while: Clojure compile times, JVM start-up time, etc. ▶ There’s no staging environment
44. Ogrom is an improvement ▶ Client teams are happy with it ▶ It performs well under load and is durable ▶ Our main web app uses the Ogrom too now
45. Plans for the future ▶ Ring Swagger ▶ Clojure 1.7 and its reduced compilation time ▶ Clojure(Script) client API
47. Clojure Redeployed Jan Stępień @janstepien jan@stepien.cc