Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Microservices and EDA with Clojure and Kafka

Microservices and EDA with Clojure and Kafka

The slide deck of the talk we gave at the Reversim summit 2015


March 15, 2015

More Decks by AppsFlyer

Other Decks in Technology


  1. The World's leading Mobile Measurement Platform Founded in 2011 by

    Reshef Mann and Oren Kaniel Just completed Round B funding - total of $28M Processing 2.8B daily events (it was 1.9B just 3 months ago and 250M at the start of 2014!) 13 people in the development team (we were just 6 people 12 months ago!) AppsFlyer Who?!
  2. Are we really doing microservices? Small, isolated services that communicate

    via Redis pub/ sub and channels Code base in Python About 10 services over 5 machines and 3 DBs 2012 2014 Small, isolated services that communicate asynchronously via Kafka and RabbitMQ The occasional synchronous http communication between services when needed Code base in Clojure About 100 services over 200 machines and 20 DBs State of The Union
  3. Microservices? "... the microservice architectural style is an approach to

    developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms…" Martin Fowler
  4. Are the services isolated or dependant? Is the event in

    the beginning of the pipeline the same as the event at the end of the pipeline? Wait a sec, pipeline? Can EDA and Microservices co- exists? Microservices or EDA?
  5. Why Clojure? Sequence based processing capabilities really fit in the

    visualized data flow of AppsFlyer (processing the Event Stream) Enforces use of FP paradigm more strictly than Scala Repl based development Easy and common Java interop JVM!
  6. Small isolated services Each service has a single business responsibility

    Each service encapsulates its own data (if it has any) and he exposes it over a well known interface Data objects are always POCO/POJO (simple data structures represented in JSON or EDN) Preference for queues and buffers that pass isolated data for total async processing How We Model
  7. How We Test No QA team Each new service can

    read from the event stream to its heart content - regular Kafka consumers behavior Each new service handles real life traffic and real life load because it's connected to the event stream Test DB if needed is easy to spin up on the cloud Once deemed ready, just throw the switch to on
  8. Service discovery via Consul How We Orchestrate Mesos Each service

    is a single uberjar in a single docker container DBs get their own dedicated machines Preference for ring based architecture for DBs Marathon
  9. Jenkins build server Deployment via the in-house Santa tool Consul

    health checks Statsd for the JVM and application metrics Sensu End-to-end flows Deployment and Monitoring