Ido Barkan

Using Druid Analyzing web access logs for 8 billion events per day

Transcript

1. Ido Barkan Analyzing web access logs for 8 billion events per day
2. 5xx Errors
3. Appsflyer gets around 8B web events per day.
4. Micro Services Architecture Real Time Attr.
5. AWS Elastic load balancer Log entry
6. A Log line 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001 200 200 780 2 "POST https://track. appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1 $ head -1 195229424603_elasticloadbalancing_eu-west-1_appsflyer-web_.log | wc -c 331 Total: 300-1500 bytes =>sub sampling of 1/10 => 223 GB daily approx.
7. What was missing? No transparency of incoming web requests. ? # error (400 / 500) responses grouped by app ? # of events grouped by app ? # of events grouped by response code
8. What wasn’t missing? ! No single event granularity- only analytics ! No fancy enterprise features (role-based access, alerts etc.)
9. Possible solutions 1. Our own ELK- will not hold the volume 2. SaaS based ELK (logz.io, loggly...)- expensive and gives more than we want.
10. Data flow Log to bucket Trigger Lambda Druid sink service
11. Druid configured naively • 3 data nodes (historical+RT) • 1 master (coordinator) • 1 broker • No data duplication • 7d data retention • Only 5 machines
12. Basic log processing 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001 200 200 780 2 "POST https://track.appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1
13. Demo! • druidquery • caravel
14. Thank you ido@appsflyer.com Questions?
15. Thank you ido@appsflyer.com We are hiring!