Slide 1
Slide 1 text
Microservices and EDA
With Clojure and Kafka
(oh my...)
Slide 2
Slide 2 text
The World's leading Mobile
Measurement Platform
Founded in 2011 by Reshef Mann
and Oren Kaniel
Just completed Round B funding -
total of $28M
Processing 2.8B daily events (it was
1.9B just 3 months ago and 250M at
the start of 2014!)
13 people in the development team
(we were just 6 people 12 months ago!)
AppsFlyer
Who?!
Slide 3
Slide 3 text
Are we really doing microservices?
Small, isolated services that
communicate via Redis pub/
sub and channels
Code base in Python
About 10 services over 5
machines and 3 DBs
2012 2014
Small, isolated services that
communicate asynchronously
via Kafka and RabbitMQ
The occasional synchronous
http communication between
services when needed
Code base in Clojure
About 100 services over 200
machines and 20 DBs
State of The Union
Slide 4
Slide 4 text
Microservices?
"... the microservice architectural
style is an approach to developing a
single application as a suite of small
services, each running in its own
process and communicating with
lightweight mechanisms…"
Martin Fowler
Slide 5
Slide 5 text
The Bird's Eye View
Slide 6
Slide 6 text
Are the services isolated or
dependant?
Is the event in the beginning
of the pipeline the same as the
event at the end of the pipeline?
Wait a sec, pipeline?
Can EDA and Microservices co-
exists?
Microservices
or EDA?
Slide 7
Slide 7 text
The Event
Stream
Slide 8
Slide 8 text
WHY FUNCTIONAL?!
Slide 9
Slide 9 text
Why Clojure?
Sequence based processing
capabilities really fit in the
visualized data flow of
AppsFlyer (processing the
Event Stream)
Enforces use of
FP paradigm more
strictly than Scala
Repl based
development
Easy and
common Java
interop
JVM!
Slide 10
Slide 10 text
Small isolated services
Each service has a single business
responsibility
Each service encapsulates its own data
(if it has any) and he exposes it over a
well known interface
Data objects are always POCO/POJO
(simple data structures represented in
JSON or EDN)
Preference for queues and buffers
that pass isolated data for total async
processing
How We
Model
Slide 11
Slide 11 text
How We
Test
No QA team
Each new service can read from the
event stream to its heart content -
regular Kafka consumers behavior
Each new service handles real life
traffic and real life load because it's
connected to the event stream
Test DB if needed is easy to spin up on
the cloud
Once deemed ready, just throw the
switch to on
Slide 12
Slide 12 text
Service discovery
via Consul
How We
Orchestrate Mesos
Each service is a
single uberjar in
a single docker
container
DBs get their
own dedicated
machines
Preference for ring
based architecture
for DBs
Marathon
Slide 13
Slide 13 text
Jenkins build server
Deployment via the in-house
Santa tool
Consul health checks
Statsd for the JVM and
application metrics
Sensu
End-to-end flows
Deployment
and
Monitoring