Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Ido Barkan
Search
AppsFlyer
July 27, 2016
Technology
1
140
Ido Barkan
Using Druid Analyzing web access logs for 8 billion events per day
AppsFlyer
July 27, 2016
Tweet
Share
More Decks by AppsFlyer
See All by AppsFlyer
Processing 15 Billion events a day without breaking the bank - ReversimX ILTechTalks
appsflyer
0
490
Journey to the Real-Time Analytics in Extreme Growth
appsflyer
0
290
10 Real problems & solutions in your build and deploy process
appsflyer
0
140
DevOps paradigm in R&D day-to-day
appsflyer
0
140
Building a Mobile Backend to Evolve
appsflyer
0
100
Sometimes, Druid is not the best solution for a business use case
appsflyer
1
420
Processing 8 Billion Daily Events in Real Time!
appsflyer
1
120
React Performance
appsflyer
1
210
Real-time analytics with Druid at Appsflyer
appsflyer
0
360
Other Decks in Technology
See All in Technology
“プロダクトを好きになれるか“も QAエンジニア転職の大事な判断基準だと思ったの
tomodakengo
0
130
OpenTelemetry Collector internals
ymotongpoo
5
530
Eight Engineering Unit 紹介資料
sansan33
PRO
0
3.4k
Snowflake Intelligenceで実現できるノーコードAI活用
takumimukaiyama
1
210
「規約、知識、オペレーション」から考える中規模以上の開発組織のCursorルールの 考え方・育て方 / Cursor Rules for Coding Styles, Domain Knowledges and Operations
yuitosato
6
1.6k
原則から考える保守しやすいComposable関数設計
moriatsushi
2
240
菸酒生在 LINE Taiwan 的後端雙刀流
line_developers_tw
PRO
0
140
Whats_new_in_Podman_and_CRI-O_2025-06
orimanabu
3
170
kubellが挑むBPaaSにおける、人とAIエージェントによるサービス開発の最前線と技術展望
kubell_hr
0
280
TODAY 看世界(?) 是我們在看扣啦!
line_developers_tw
PRO
0
160
Introduction to Sansan Meishi Maker Development Engineer
sansan33
PRO
0
280
工具人的一生: 開發很多 AI 工具讓我 慵懶過一生
line_developers_tw
PRO
0
140
Featured
See All Featured
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Designing Experiences People Love
moore
142
24k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
60k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
A Modern Web Designer's Workflow
chriscoyier
693
190k
Gamification - CAS2011
davidbonilla
81
5.3k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
900
Product Roadmaps are Hard
iamctodd
PRO
53
11k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
4
130
Transcript
Ido Barkan Analyzing web access logs for 8 billion events
per day
5xx Errors
Appsflyer gets around 8B web events per day.
Micro Services Architecture Real Time Attr.
AWS Elastic load balancer Log entry
A Log line 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001
200 200 780 2 "POST https://track. appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1 $ head -1 195229424603_elasticloadbalancing_eu-west-1_appsflyer-web_.log | wc -c 331 Total: 300-1500 bytes =>sub sampling of 1/10 => 223 GB daily approx.
What was missing? No transparency of incoming web requests. ?
# error (400 / 500) responses grouped by app ? # of events grouped by app ? # of events grouped by response code
What wasn’t missing? ! No single event granularity- only analytics
! No fancy enterprise features (role-based access, alerts etc.)
Possible solutions 1. Our own ELK- will not hold the
volume 2. SaaS based ELK (logz.io, loggly...)- expensive and gives more than we want.
Data flow Log to bucket Trigger Lambda Druid sink service
Druid configured naively • 3 data nodes (historical+RT) • 1
master (coordinator) • 1 broker • No data duplication • 7d data retention • Only 5 machines
Basic log processing 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001
200 200 780 2 "POST https://track.appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1
Demo! • druidquery • caravel
Thank you
[email protected]
Questions?
Thank you
[email protected]
We are hiring!