Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How to scale a Logging Infrastructure
Search
Paul Stack
June 03, 2015
Technology
0
170
How to scale a Logging Infrastructure
Logging infrastructure using ELK + Kafka
Paul Stack
June 03, 2015
Tweet
Share
More Decks by Paul Stack
See All by Paul Stack
Infrastructure as Software
stack72
0
63
Mirror, Mirror on the way, what is the vainest metric of them all?
stack72
1
2.3k
Continuously Delivering Infrastructure to the Cloud
stack72
0
170
DevOops 2016
stack72
0
120
The Quest for Infrastructure Management 2.0
stack72
0
130
The Biggest Trick Consultants Ever Pulled was Telling The World Continuous Delivery is Easy
stack72
1
110
The Transition from Product to Infrastructure
stack72
0
58
Continuous Delivery - the missing parts
stack72
0
920
Windows: Having its ass kicked by puppet and powershell
stack72
0
120
Other Decks in Technology
See All in Technology
手を動かしてレベルアップしよう!
maruto
0
210
【内製開発Summit 2025】イオンスマートテクノロジーの内製化組織の作り方/In-house-development-summit-AST
aeonpeople
2
660
4th place solution Eedi - Mining Misconceptions in Mathematics
rist
0
140
コンピュータビジョンの社会実装について考えていたらゲームを作っていた話
takmin
1
600
ABWG2024採択者が語るエンジニアとしての自分自身の見つけ方〜発信して、つながって、世界を広げていく〜
maimyyym
1
170
Active Directory攻防
cryptopeg
PRO
8
5.5k
AWS Well-Architected Frameworkで学ぶAmazon ECSのセキュリティ対策
umekou
2
140
【詳説】コンテンツ配信 システムの複数機能 基盤への拡張
hatena
0
240
急成長する企業で作った、エンジニアが輝ける制度/ 20250227 Rinto Ikenoue
shift_evolve
0
130
Share my, our lessons from the road to re:Invent
naospon
0
140
IAMのマニアックな話2025
nrinetcom
PRO
3
440
Snowflakeの開発・運用コストをApache Icebergで効率化しよう!~機能と活用例のご紹介~
sagara
1
450
Featured
See All Featured
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.2k
How STYLIGHT went responsive
nonsquared
98
5.4k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
175
52k
Making Projects Easy
brettharned
116
6k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
4
430
For a Future-Friendly Web
brad_frost
176
9.6k
Embracing the Ebb and Flow
colly
84
4.6k
Typedesign – Prime Four
hannesfritz
40
2.5k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
Music & Morning Musume
bryan
46
6.4k
Transcript
How do you scale a logging infrastructure to accept a
billion messages a day? Paul Stack http://twitter.com/stack72 mail:
[email protected]
About Me Infrastructure Engineer for a cool startup :) Reformed
ASP.NET / C# Developer DevOps Extremist Conference Junkie
Background Project was to replace the legacy ‘logging solution’
Iteration 0: A Developer created a single box with the
ELK all in 1 jar
Time to make it production ready now
None
Iteration 1: Using Redis as the input mechanism for LogStash
None
None
Enter Apache Kafka
“Kafka is a distributed publish- subscribe messaging system that is
designed to be fast, scalable, and durable” Source: Cloudera Blog
Introduction to Kafka • Kafka is made up of ‘topics’,
‘producers’, ‘consumers’ and ‘brokers’ • Communication is via TCP • Backed by Zookeeper
Kafka Topics Source: http://kafka.apache.org/documentation.html
Kafka Producers • Producers are responsible to chose what topic
to publish data to • The producer is responsible for choosing a partition to write to • Can be handled round robin or partition functions
Kafka Consumers • Consumption can be done via: • queuing
• pub-sub
Kafka Consumers • Kafka consumer group • Strong ordering
Kafka Consumers • Strong ordering
https://github.com/opentable/puppet-exhibitor
None
Iteration 2 Introduction of Kafka
None
None
Iteration 3 Further ‘Improvements’ to the cluster layout
None
The Numbers • Logs kept in ES for 30 days
then archived • 12 billion documents active in ES • ES space was about 25 - 30TB in EBS volumes • Average Doc Size ~ 1.2KB • V-Day 2015: ~750M docs collected without failure
What about metrics and monitoring?
Monitoring - Nagios • Alerts on • ES Cluster •
zK and Kafka Nodes • Logstash / Redis nodes
None
https://github.com/stack72/nagios-elasticsearch
Metrics - Kafka Offset Monitor
https://github.com/opentable/KafkaOffsetMonitor
Metrics - ElasticSearch
None
None
None
Visibility Rocks!
None
So what would I do differently?
Questions?
Paul Stack @stack72