Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Ido Barkan
Search
AppsFlyer
July 27, 2016
Technology
1
150
Ido Barkan
Using Druid Analyzing web access logs for 8 billion events per day
AppsFlyer
July 27, 2016
Tweet
Share
More Decks by AppsFlyer
See All by AppsFlyer
Processing 15 Billion events a day without breaking the bank - ReversimX ILTechTalks
appsflyer
0
490
Journey to the Real-Time Analytics in Extreme Growth
appsflyer
0
300
10 Real problems & solutions in your build and deploy process
appsflyer
0
140
DevOps paradigm in R&D day-to-day
appsflyer
0
150
Building a Mobile Backend to Evolve
appsflyer
0
110
Sometimes, Druid is not the best solution for a business use case
appsflyer
1
430
Processing 8 Billion Daily Events in Real Time!
appsflyer
1
120
React Performance
appsflyer
1
220
Real-time analytics with Druid at Appsflyer
appsflyer
0
370
Other Decks in Technology
See All in Technology
スタートアップにおけるこれからの「データ整備」
shomaekawa
2
480
20251007: What happens when multi-agent systems become larger? (CyberAgent, Inc)
ornew
1
290
AWSでAgentic AIを開発するための前提知識の整理
nasuvitz
2
160
Git in Team
kawaguti
PRO
3
370
CoRL 2025 Survey
harukiabe
1
200
AI時代こそ求められる設計力- AWSクラウドデザインパターン3選で信頼性と拡張性を高める-
kenichirokimura
3
320
Bill One 開発エンジニア 紹介資料
sansan33
PRO
4
14k
セキュアな認可付きリモートMCPサーバーをAWSマネージドサービスでつくろう! / Let's build an OAuth protected remote MCP server based on AWS managed services
kaminashi
3
330
衛星画像超解像化によって実現する2D, 3D空間情報の即時生成と“AI as a Service”/ Real-time generation spatial data enabled_by satellite image super-resolution
lehupa
0
170
Geospatialの世界最前線を探る [2025年版]
dayjournal
1
220
Claude Code Subagents 再入門 ~cc-sddの実装で学んだこと~
gotalab555
7
11k
RDS の負荷が高い場合に AWS で取りうる具体策 N 連発/a-series-of-specific-countermeasures-available-on-aws-when-rds-is-under-high-load
emiki
2
1.2k
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
303
21k
A Tale of Four Properties
chriscoyier
161
23k
Building Adaptive Systems
keathley
44
2.8k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Statistics for Hackers
jakevdp
799
220k
Producing Creativity
orderedlist
PRO
347
40k
Writing Fast Ruby
sferik
629
62k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
16
1.7k
Imperfection Machines: The Place of Print at Facebook
scottboms
269
13k
Practical Orchestrator
shlominoach
190
11k
Speed Design
sergeychernyshev
32
1.2k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
8
910
Transcript
Ido Barkan Analyzing web access logs for 8 billion events
per day
5xx Errors
Appsflyer gets around 8B web events per day.
Micro Services Architecture Real Time Attr.
AWS Elastic load balancer Log entry
A Log line 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001
200 200 780 2 "POST https://track. appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1 $ head -1 195229424603_elasticloadbalancing_eu-west-1_appsflyer-web_.log | wc -c 331 Total: 300-1500 bytes =>sub sampling of 1/10 => 223 GB daily approx.
What was missing? No transparency of incoming web requests. ?
# error (400 / 500) responses grouped by app ? # of events grouped by app ? # of events grouped by response code
What wasn’t missing? ! No single event granularity- only analytics
! No fancy enterprise features (role-based access, alerts etc.)
Possible solutions 1. Our own ELK- will not hold the
volume 2. SaaS based ELK (logz.io, loggly...)- expensive and gives more than we want.
Data flow Log to bucket Trigger Lambda Druid sink service
Druid configured naively • 3 data nodes (historical+RT) • 1
master (coordinator) • 1 broker • No data duplication • 7d data retention • Only 5 machines
Basic log processing 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001
200 200 780 2 "POST https://track.appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1
Demo! • druidquery • caravel
Thank you
[email protected]
Questions?
Thank you
[email protected]
We are hiring!