Microservices and EDA with Clojure and Kafka

The slide deck of the talk we gave at the Reversim summit 2015

Transcript

1. Microservices and EDA
   With Clojure and Kafka
   (oh my...)
   

   View Slide

2. The World's leading Mobile
   Measurement Platform
   Founded in 2011 by Reshef Mann
   and Oren Kaniel
   Just completed Round B funding -
   total of $28M
   Processing 2.8B daily events (it was
   1.9B just 3 months ago and 250M at
   the start of 2014!)
   13 people in the development team
   (we were just 6 people 12 months ago!)
   AppsFlyer
   Who?!
   

   View Slide

3. Are we really doing microservices?
   Small, isolated services that
   communicate via Redis pub/
   sub and channels
   Code base in Python
   About 10 services over 5
   machines and 3 DBs
   2012 2014
   Small, isolated services that
   communicate asynchronously
   via Kafka and RabbitMQ
   The occasional synchronous
   http communication between
   services when needed
   Code base in Clojure
   About 100 services over 200
   machines and 20 DBs
   State of The Union
   

   View Slide

4. Microservices?
   "... the microservice architectural
   style is an approach to developing a
   single application as a suite of small
   services, each running in its own
   process and communicating with
   lightweight mechanisms…"
   Martin Fowler
   

   View Slide

5. The Bird's Eye View
   

   View Slide

6. Are the services isolated or
   dependant?
   Is the event in the beginning
   of the pipeline the same as the
   event at the end of the pipeline?
   Wait a sec, pipeline?
   Can EDA and Microservices co-
   exists?
   Microservices
   or EDA?
   

   View Slide

7. The Event
   Stream
   

   View Slide

8. WHY FUNCTIONAL?!
   

   View Slide

9. Why Clojure?
   Sequence based processing
   capabilities really fit in the
   visualized data flow of
   AppsFlyer (processing the
   Event Stream)
   Enforces use of
   FP paradigm more
   strictly than Scala
   Repl based
   development
   Easy and
   common Java
   interop
   JVM!
   

   View Slide

10. Small isolated services
    Each service has a single business
    responsibility
    Each service encapsulates its own data
    (if it has any) and he exposes it over a
    well known interface
    Data objects are always POCO/POJO
    (simple data structures represented in
    JSON or EDN)
    Preference for queues and buffers
    that pass isolated data for total async
    processing
    How We
    Model
    

    View Slide

11. How We
    Test
    No QA team
    Each new service can read from the
    event stream to its heart content -
    regular Kafka consumers behavior
    Each new service handles real life
    traffic and real life load because it's
    connected to the event stream
    Test DB if needed is easy to spin up on
    the cloud
    Once deemed ready, just throw the
    switch to on
    

    View Slide

12. Service discovery
    via Consul
    How We
    Orchestrate Mesos
    Each service is a
    single uberjar in
    a single docker
    container
    DBs get their
    own dedicated
    machines
    Preference for ring
    based architecture
    for DBs
    Marathon
    

    View Slide

13. Jenkins build server
    Deployment via the in-house
    Santa tool
    Consul health checks
    Statsd for the JVM and
    application metrics
    Sensu
    End-to-end flows
    Deployment
    and
    Monitoring
    

    View Slide