Processing 8 Billion Daily Events in Real Time!

session given in "Big Things" meetup on the 30/6/2016
Video of the session is here: (hebrew)
https://www.youtube.com/watch?v=s5Meg6j5J3A

1. Adi Belan - R&D Team Lead Processing 8 Billion events per day - in real-time! (while sleeping at night)
2. What is AppsFlyer? Mobile Attribution Measurement and Analytics
4. Tech Stack - Tools of the Trade
5. Micro Services Architecture Real Time Attr.
6. What it really feels like?
7. So, 8 Billion Events you say? • SDK installed in 10 Billion mobile devices • Every time App is launched += 1 event • Every time User engages with Ad += 1 event
8. So, Why Real Time? • Deferred Deeplinking - Opening App in correct context • Ad Networks optimize bidding in real time
9. System Structure
10. Some Numbers • 7 Different ELBs • ~90 - 160 web-handlers • 4-7 million http requests per minute • 10+ Billion Kafka messages per day • Multiple Couchbase clusters with Billions of keys Attribution decision time <= 500 Millis
11. Why RabbitMQ and Kafka? • Latency (!!) - Old(er) Kafka versions have high tail latencies • Resiliency - even if Rabbit / Kafka is down we can provide (degraded) service
12. Matching Architecture Design • Last touch wins! • Rendezvous Pattern ◦ 1st app launch could generate 25 http API calls • External APIs are not(!) reliable
13. Matching Architecture Design
14. Are Not Enough!
16. Circuit Breaking in Action
17. Developers like to sleep! • Healer - “revives” machines which stopped processing for some reason. • Monitoring takes CB into account • Monitoring ELB and scaling web handlers if needed.
18. So, you wrote a bug…. • Secor saves raw kafka data to S3 • Ad-Hoc replaying “buggy” traffic possible (in some cases)
19. We’re hiring!