Microservices and EDA with Clojure and Kafka

The slide deck of the talk we gave at the Reversim summit 2015

Transcript

1. Microservices and EDA With Clojure and Kafka (oh my...)
2. The World's leading Mobile Measurement Platform Founded in 2011 by Reshef Mann and Oren Kaniel Just completed Round B funding - total of $28M Processing 2.8B daily events (it was 1.9B just 3 months ago and 250M at the start of 2014!) 13 people in the development team (we were just 6 people 12 months ago!) AppsFlyer Who?!
3. Are we really doing microservices? Small, isolated services that communicate via Redis pub/ sub and channels Code base in Python About 10 services over 5 machines and 3 DBs 2012 2014 Small, isolated services that communicate asynchronously via Kafka and RabbitMQ The occasional synchronous http communication between services when needed Code base in Clojure About 100 services over 200 machines and 20 DBs State of The Union
4. Microservices? "... the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms…" Martin Fowler
5. The Bird's Eye View
6. Are the services isolated or dependant? Is the event in the beginning of the pipeline the same as the event at the end of the pipeline? Wait a sec, pipeline? Can EDA and Microservices co- exists? Microservices or EDA?
7. The Event Stream
8. WHY FUNCTIONAL?!
9. Why Clojure? Sequence based processing capabilities really fit in the visualized data flow of AppsFlyer (processing the Event Stream) Enforces use of FP paradigm more strictly than Scala Repl based development Easy and common Java interop JVM!
10. Small isolated services Each service has a single business responsibility Each service encapsulates its own data (if it has any) and he exposes it over a well known interface Data objects are always POCO/POJO (simple data structures represented in JSON or EDN) Preference for queues and buffers that pass isolated data for total async processing How We Model
11. How We Test No QA team Each new service can read from the event stream to its heart content - regular Kafka consumers behavior Each new service handles real life traffic and real life load because it's connected to the event stream Test DB if needed is easy to spin up on the cloud Once deemed ready, just throw the switch to on
12. Service discovery via Consul How We Orchestrate Mesos Each service is a single uberjar in a single docker container DBs get their own dedicated machines Preference for ring based architecture for DBs Marathon
13. Jenkins build server Deployment via the in-house Santa tool Consul health checks Statsd for the JVM and application metrics Sensu End-to-end flows Deployment and Monitoring