Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Ido Barkan
Search
AppsFlyer
July 27, 2016
Technology
1
140
Ido Barkan
Using Druid Analyzing web access logs for 8 billion events per day
AppsFlyer
July 27, 2016
Tweet
Share
More Decks by AppsFlyer
See All by AppsFlyer
Processing 15 Billion events a day without breaking the bank - ReversimX ILTechTalks
appsflyer
0
490
Journey to the Real-Time Analytics in Extreme Growth
appsflyer
0
290
10 Real problems & solutions in your build and deploy process
appsflyer
0
140
DevOps paradigm in R&D day-to-day
appsflyer
0
140
Building a Mobile Backend to Evolve
appsflyer
0
100
Sometimes, Druid is not the best solution for a business use case
appsflyer
1
430
Processing 8 Billion Daily Events in Real Time!
appsflyer
1
120
React Performance
appsflyer
1
210
Real-time analytics with Druid at Appsflyer
appsflyer
0
360
Other Decks in Technology
See All in Technology
2025-07-06 QGIS初級ハンズオン「はじめてのQGIS」
kou_kita
0
170
Claude Code に プロジェクト管理やらせたみた
unson
6
4k
関数型プログラミングで 「脳がバグる」を乗り越える
manabeai
1
190
開発生産性を組織全体の「生産性」へ! 部門間連携の壁を越える実践的ステップ
sudo5in5k
2
7k
ネットワーク保護はどう変わるのか?re:Inforce 2025最新アップデート解説
tokushun
0
210
20250705 Headlamp: 專注可擴展性的 Kubernetes 用戶界面
pichuang
0
270
Zero Data Loss Autonomous Recovery Service サービス概要
oracle4engineer
PRO
2
7.7k
United airlines®️ USA Contact Numbers: Complete 2025 Support Guide
unitedflyhelp
0
310
AI時代の開発生産性を加速させるアーキテクチャ設計
plaidtech
PRO
3
150
LangChain Interrupt & LangChain Ambassadors meetingレポート
os1ma
2
310
american airlines®️ USA Contact Numbers: Complete 2025 Support Guide
supportflight
1
110
CRE Camp #1 エンジニアリングを民主化するCREチームでありたい話
mntsq
1
120
Featured
See All Featured
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
970
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Building Applications with DynamoDB
mza
95
6.5k
Designing Experiences People Love
moore
142
24k
Automating Front-end Workflow
addyosmani
1370
200k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
Faster Mobile Websites
deanohume
307
31k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Designing for humans not robots
tammielis
253
25k
Transcript
Ido Barkan Analyzing web access logs for 8 billion events
per day
5xx Errors
Appsflyer gets around 8B web events per day.
Micro Services Architecture Real Time Attr.
AWS Elastic load balancer Log entry
A Log line 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001
200 200 780 2 "POST https://track. appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1 $ head -1 195229424603_elasticloadbalancing_eu-west-1_appsflyer-web_.log | wc -c 331 Total: 300-1500 bytes =>sub sampling of 1/10 => 223 GB daily approx.
What was missing? No transparency of incoming web requests. ?
# error (400 / 500) responses grouped by app ? # of events grouped by app ? # of events grouped by response code
What wasn’t missing? ! No single event granularity- only analytics
! No fancy enterprise features (role-based access, alerts etc.)
Possible solutions 1. Our own ELK- will not hold the
volume 2. SaaS based ELK (logz.io, loggly...)- expensive and gives more than we want.
Data flow Log to bucket Trigger Lambda Druid sink service
Druid configured naively • 3 data nodes (historical+RT) • 1
master (coordinator) • 1 broker • No data duplication • 7d data retention • Only 5 machines
Basic log processing 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001
200 200 780 2 "POST https://track.appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1
Demo! • druidquery • caravel
Thank you
[email protected]
Questions?
Thank you
[email protected]
We are hiring!