Clojure Redeployed - Speaker Deck

Slide 1

Slide 1 text

Clojure Redeployed Jan Stępień @janstepien [email protected]

Slide 2

Slide 2 text

Slide 3

Slide 3 text

I work at

Slide 4

Slide 4 text

I work at

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

https://api.stylefruits.de Elasticsearch MySQL

Slide 8

Slide 8 text

https://api.stylefruits.de Elasticsearch MySQL So how about...

Slide 9

Slide 9 text

https://api.stylefruits.de Elasticsearch MySQL So how about... Clojure?

Slide 10

Slide 10 text

Expectations set high ▶ short feedback loop ▶ continuous delivery ▶ automated and pauseless deployment ▶ ...under load ▶ high availability, dynamic scaling

Slide 11

Slide 11 text

we’ve called it Ogrom* * plethora, immensity, enormity

Slide 12

Slide 12 text

Outline 1. Architecture 2. Deployment and Operations 3. Lessons Learned

Slide 13

Slide 13 text

Architecture to use a bold word

Slide 14

Slide 14 text

sources protocols run core

Slide 15

Slide 15 text

protocols deﬁne interfaces (defprotocol CollageSource (collage-by-id [ctx collage-id]))

Slide 16

Slide 16 text

sources wrap data sources (reify CollageSource (collage-by-id [ctx collage-id] (fetch-from-mysql ctx collage-id)))

Slide 17

Slide 17 text

sources protocols run core

Slide 18

Slide 18 text

core deﬁnes HTTP endpoints each core endpoint ▶ is independent ▶ has all dependencies injected ▶ is absolute-path-agnostic ▶ is a value

Slide 19

Slide 19 text

core uses Liberator (defresource product :exists? find-product-or-return-nil :handle-ok render-product :handle-not-found render-error-message)

Slide 20

Slide 20 text

:exists? :handle-ok 200 true :handle-not-found 404 false

Slide 21

Slide 21 text

core uses bidi (def routes ["/" {"product" product "ratings" ratings}])

Slide 22

Slide 22 text

sources protocols run core

Slide 23

Slide 23 text

run wires everything together products GET /products POST /comments comments GET /comments users

Slide 24

Slide 24 text

/v1 /de-DE GET /products GET /comments POST /comments

Slide 25

Slide 25 text

/v1 /de-DE GET /products GET /comments POST /comments GET /v1/de-DE/comments

Slide 26

Slide 26 text

routing GET /comments POST /comments GET /products users third-party API MySQL products Elasticsearch comments

Slide 27

Slide 27 text

run instantiates a self-contained service MySQL Elasticsearch logstash 3rd-party API ogrom

Slide 28

Slide 28 text

MySQL Elasticsearch logstash 3rd-party API ogrom MySQL Elasticsearch logstash 3rd-party API ogrom MySQL Elasticsearch logstash 3rd-party API ogrom MySQL Elasticsearch logstash 3rd-party API ogrom

Slide 29

Slide 29 text

/v1 /fr-FR /nl-NL /de-DE /en-GB GET /products GET /comments POST /comments

Slide 30

Slide 30 text

MySQL Elasticsearch logstash 3rd-party API ogrom

Slide 31

Slide 31 text

Deployment and Operations

Slide 32

Slide 32 text

Deployment used to be cumbersome 1. Create an überjar 2. Upload to n boxes 3. Restart processes 4. Juggle load balancers

Slide 33

Slide 33 text

sources protocols run core ▶ sources, protocols, core are sem-versioned ▶ run is deployed straight from master

Slide 34

Slide 34 text

sources run protocols core CI on every push to master S3 depl. tool überjar commit id

Slide 35

Slide 35 text

Elastic Beanstalk and its environments ▶ EB wraps EC2 and ELB ▶ environments with instances ▶ runs Docker containers

Slide 36

Slide 36 text

deployment tool AWS API CI commit id box 1 box 2 S3 (überjar)

Slide 37

Slide 37 text

api.stylefruits.de Beanstalk 1 Beanstalk 2

Slide 38

Slide 38 text

api.stylefruits.de CloudFront Beanstalk 1 Beanstalk 2

Slide 39

Slide 39 text

Lessons Learned and Future Work

Slide 40

Slide 40 text

Feedback loops are shorter ▶ Productive development environment ▶ Bug ﬁxes and features can be deploy the same morning they were asked for

Slide 41

Slide 41 text

Splitting wasn’t strictly necessary sources protocols run core

Slide 42

Slide 42 text

Docker and empowered developers ▶ Developers are in control over deployment ▶ Reduced dependency on the ops team ▶ Production env ≈ development env

Slide 43

Slide 43 text

There are some trade-o s ▶ Beanstalk allows opening only a single port when running Docker containers ▶ Deployment under load vs. JIT compilation ▶ The deployment takes a while: Clojure compile times, JVM start-up time, etc. ▶ There’s no staging environment

Slide 44

Slide 44 text

Ogrom is an improvement ▶ Client teams are happy with it ▶ It performs well under load and is durable ▶ Our main web app uses the Ogrom too now

Slide 45

Slide 45 text

Plans for the future ▶ Ring Swagger ▶ Clojure 1.7 and its reduced compilation time ▶ Clojure(Script) client API