Slide 1

Slide 1 text

Microservices and EDA With Clojure and Kafka (oh my...)

Slide 2

Slide 2 text

The World's leading Mobile Measurement Platform Founded in 2011 by Reshef Mann and Oren Kaniel Just completed Round B funding - total of $28M Processing 2.8B daily events (it was 1.9B just 3 months ago and 250M at the start of 2014!) 13 people in the development team (we were just 6 people 12 months ago!) AppsFlyer Who?!

Slide 3

Slide 3 text

Are we really doing microservices? Small, isolated services that communicate via Redis pub/ sub and channels Code base in Python About 10 services over 5 machines and 3 DBs 2012 2014 Small, isolated services that communicate asynchronously via Kafka and RabbitMQ The occasional synchronous http communication between services when needed Code base in Clojure About 100 services over 200 machines and 20 DBs State of The Union

Slide 4

Slide 4 text

Microservices? "... the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms…" Martin Fowler

Slide 5

Slide 5 text

The Bird's Eye View

Slide 6

Slide 6 text

Are the services isolated or dependant? Is the event in the beginning of the pipeline the same as the event at the end of the pipeline? Wait a sec, pipeline? Can EDA and Microservices co- exists? Microservices or EDA?

Slide 7

Slide 7 text

The Event Stream

Slide 8

Slide 8 text


Slide 9

Slide 9 text

Why Clojure? Sequence based processing capabilities really fit in the visualized data flow of AppsFlyer (processing the Event Stream) Enforces use of FP paradigm more strictly than Scala Repl based development Easy and common Java interop JVM!

Slide 10

Slide 10 text

Small isolated services Each service has a single business responsibility Each service encapsulates its own data (if it has any) and he exposes it over a well known interface Data objects are always POCO/POJO (simple data structures represented in JSON or EDN) Preference for queues and buffers that pass isolated data for total async processing How We Model

Slide 11

Slide 11 text

How We Test No QA team Each new service can read from the event stream to its heart content - regular Kafka consumers behavior Each new service handles real life traffic and real life load because it's connected to the event stream Test DB if needed is easy to spin up on the cloud Once deemed ready, just throw the switch to on

Slide 12

Slide 12 text

Service discovery via Consul How We Orchestrate Mesos Each service is a single uberjar in a single docker container DBs get their own dedicated machines Preference for ring based architecture for DBs Marathon

Slide 13

Slide 13 text

Jenkins build server Deployment via the in-house Santa tool Consul health checks Statsd for the JVM and application metrics Sensu End-to-end flows Deployment and Monitoring