Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Ido Barkan
Search
AppsFlyer
July 27, 2016
Technology
1
130
Ido Barkan
Using Druid Analyzing web access logs for 8 billion events per day
AppsFlyer
July 27, 2016
Tweet
Share
More Decks by AppsFlyer
See All by AppsFlyer
Processing 15 Billion events a day without breaking the bank - ReversimX ILTechTalks
appsflyer
0
460
Journey to the Real-Time Analytics in Extreme Growth
appsflyer
0
270
10 Real problems & solutions in your build and deploy process
appsflyer
0
130
DevOps paradigm in R&D day-to-day
appsflyer
0
130
Building a Mobile Backend to Evolve
appsflyer
0
78
Sometimes, Druid is not the best solution for a business use case
appsflyer
1
410
Processing 8 Billion Daily Events in Real Time!
appsflyer
1
100
React Performance
appsflyer
1
200
Real-time analytics with Druid at Appsflyer
appsflyer
0
340
Other Decks in Technology
See All in Technology
バクラクのドキュメント解析技術と実データにおける課題 / layerx-ccc-winter-2024
shimacos
2
1k
Amazon VPC Lattice 最新アップデート紹介 - PrivateLink も似たようなアップデートあったけど違いとは
bigmuramura
0
190
PHPからGoへのマイグレーション for DMMアフィリエイト
yabakokobayashi
1
160
Oracle Cloudの生成AIサービスって実際どこまで使えるの? エンジニア目線で試してみた
minorun365
PRO
4
270
日本版とグローバル版のモバイルアプリ統合の開発の裏側と今後の展望
miichan
1
120
kargoの魅力について伝える
magisystem0408
0
200
社外コミュニティで学び社内に活かす共に学ぶプロジェクトの実践/backlogworld2024
nishiuma
0
250
Amazon SageMaker Unified Studio(Preview)、Lakehouse と Amazon S3 Tables
ishikawa_satoru
0
150
ずっと昔に Star をつけたはずの思い出せない GitHub リポジトリを見つけたい!
rokuosan
0
150
新機能VPCリソースエンドポイント機能検証から得られた考察
duelist2020jp
0
210
オプトインカメラ:UWB測位を応用したオプトイン型のカメラ計測
matthewlujp
0
170
Turing × atmaCup #18 - 1st Place Solution
hakubishin3
0
470
Featured
See All Featured
Git: the NoSQL Database
bkeepers
PRO
427
64k
A Modern Web Designer's Workflow
chriscoyier
693
190k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
1.9k
Building an army of robots
kneath
302
44k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
111
49k
GraphQLとの向き合い方2022年版
quramy
44
13k
Become a Pro
speakerdeck
PRO
26
5k
Documentation Writing (for coders)
carmenintech
66
4.5k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
BBQ
matthewcrist
85
9.4k
VelocityConf: Rendering Performance Case Studies
addyosmani
326
24k
Transcript
Ido Barkan Analyzing web access logs for 8 billion events
per day
5xx Errors
Appsflyer gets around 8B web events per day.
Micro Services Architecture Real Time Attr.
AWS Elastic load balancer Log entry
A Log line 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001
200 200 780 2 "POST https://track. appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1 $ head -1 195229424603_elasticloadbalancing_eu-west-1_appsflyer-web_.log | wc -c 331 Total: 300-1500 bytes =>sub sampling of 1/10 => 223 GB daily approx.
What was missing? No transparency of incoming web requests. ?
# error (400 / 500) responses grouped by app ? # of events grouped by app ? # of events grouped by response code
What wasn’t missing? ! No single event granularity- only analytics
! No fancy enterprise features (role-based access, alerts etc.)
Possible solutions 1. Our own ELK- will not hold the
volume 2. SaaS based ELK (logz.io, loggly...)- expensive and gives more than we want.
Data flow Log to bucket Trigger Lambda Druid sink service
Druid configured naively • 3 data nodes (historical+RT) • 1
master (coordinator) • 1 broker • No data duplication • 7d data retention • Only 5 machines
Basic log processing 2016-02-06T12:51:54.201846Z Appsflyer-web 139.162.156.169:50435 10.10.8.90:6555 0.000021 0.001916 0.00001
200 200 780 2 "POST https://track.appsflyer.com:443/... HTTP/1.1" "Dalvik/1.6.0 (Linux; U; Android 4.4.4; SM-J110H Build/KTU84P)" ECDHE-RSA-AES128-SHA TLSv1
Demo! • druidquery • caravel
Thank you
[email protected]
Questions?
Thank you
[email protected]
We are hiring!