Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Processing 8 Billion Daily Events in Real Time!

Processing 8 Billion Daily Events in Real Time!

session given in "Big Things" meetup on the 30/6/2016
Video of the session is here: (hebrew)
https://www.youtube.com/watch?v=s5Meg6j5J3A

AppsFlyer

June 30, 2016
Tweet

More Decks by AppsFlyer

Other Decks in Programming

Transcript

  1. Adi Belan - R&D Team Lead Processing 8 Billion events

    per day - in real-time! (while sleeping at night)
  2. What is AppsFlyer? Mobile Attribution Measurement and Analytics

  3. None
  4. Tech Stack - Tools of the Trade

  5. Micro Services Architecture Real Time Attr.

  6. What it really feels like?

  7. So, 8 Billion Events you say? • SDK installed in

    10 Billion mobile devices • Every time App is launched += 1 event • Every time User engages with Ad += 1 event
  8. So, Why Real Time? • Deferred Deeplinking - Opening App

    in correct context • Ad Networks optimize bidding in real time
  9. System Structure

  10. Some Numbers • 7 Different ELBs • ~90 - 160

    web-handlers • 4-7 million http requests per minute • 10+ Billion Kafka messages per day • Multiple Couchbase clusters with Billions of keys Attribution decision time <= 500 Millis
  11. Why RabbitMQ and Kafka? • Latency (!!) - Old(er) Kafka

    versions have high tail latencies • Resiliency - even if Rabbit / Kafka is down we can provide (degraded) service
  12. Matching Architecture Design • Last touch wins! • Rendezvous Pattern

    ◦ 1st app launch could generate 25 http API calls • External APIs are not(!) reliable
  13. Matching Architecture Design

  14. Are Not Enough!

  15. None
  16. Circuit Breaking in Action

  17. Developers like to sleep! • Healer - “revives” machines which

    stopped processing for some reason. • Monitoring takes CB into account • Monitoring ELB and scaling web handlers if needed.
  18. So, you wrote a bug…. • Secor saves raw kafka

    data to S3 • Ad-Hoc replaying “buggy” traffic possible (in some cases)
  19. We’re hiring!