Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ido Barkan

Ido Barkan

Using Druid Analyzing web access logs for 8 billion events per day

AppsFlyer

July 27, 2016
Tweet

More Decks by AppsFlyer

Other Decks in Technology

Transcript

  1. Ido Barkan Analyzing web access logs for 8 billion events

    per day
  2. 5xx Errors

  3. Appsflyer gets around 8B web events per day.

  4. Micro Services Architecture Real Time Attr.

  5. AWS Elastic load balancer Log entry

  6. A Log line 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001

    200 200 780 2 "POST https://track. appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1 $ head -1 195229424603_elasticloadbalancing_eu-west-1_appsflyer-web_.log | wc -c 331 Total: 300-1500 bytes =>sub sampling of 1/10 => 223 GB daily approx.
  7. What was missing? No transparency of incoming web requests. ?

    # error (400 / 500) responses grouped by app ? # of events grouped by app ? # of events grouped by response code
  8. What wasn’t missing? ! No single event granularity- only analytics

    ! No fancy enterprise features (role-based access, alerts etc.)
  9. Possible solutions 1. Our own ELK- will not hold the

    volume 2. SaaS based ELK (logz.io, loggly...)- expensive and gives more than we want.
  10. Data flow Log to bucket Trigger Lambda Druid sink service

  11. Druid configured naively • 3 data nodes (historical+RT) • 1

    master (coordinator) • 1 broker • No data duplication • 7d data retention • Only 5 machines
  12. Basic log processing 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001

    200 200 780 2 "POST https://track.appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1
  13. Demo! • druidquery • caravel

  14. Thank you [email protected] Questions?

  15. Thank you [email protected] We are hiring!