Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
DevopsdaysIndia 2018: Loki, Prometheus but for ...
Search
Goutham Veeramachaneni
December 09, 2018
Programming
1
2.1k
DevopsdaysIndia 2018: Loki, Prometheus but for logs
Goutham Veeramachaneni
December 09, 2018
Tweet
Share
More Decks by Goutham Veeramachaneni
See All by Goutham Veeramachaneni
Storage in 2.0
gouthamve
1
120
Other Decks in Programming
See All in Programming
Datadog Workflow Automation で圧倒的価値提供
showwin
1
270
苦しいTiDBへの移行を乗り越えて快適な運用を目指す
leveragestech
0
1.1k
Djangoにおける複数ユーザー種別認証の設計アプローチ@DjangoCongress JP 2025
delhi09
PRO
4
500
推しメソッドsource_locationのしくみを探る - はじめてRubyのコードを読んでみた
nobu09
2
340
Ça bouge du côté des animations CSS !
goetter
2
150
SwiftUI移行のためのインプレッショントラッキング基盤の構築
kokihirokawa
0
160
Boost Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
940
責務と認知負荷を整える! 抽象レベルを意識した関心の分離
yahiru
8
1.5k
GoとPHPのインターフェイスの違い
shimabox
2
210
AIプログラミング雑キャッチアップ
yuheinakasaka
19
4.8k
Jasprが凄い話
hyshu
0
180
「個人開発マネタイズ大全」が教えてくれたこと
bani24884
1
270
Featured
See All Featured
How GitHub (no longer) Works
holman
314
140k
How STYLIGHT went responsive
nonsquared
99
5.4k
Navigating Team Friction
lara
183
15k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
114
51k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
134
33k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
How to Think Like a Performance Engineer
csswizardry
22
1.4k
The Power of CSS Pseudo Elements
geoffreycrofte
75
5.5k
Optimizing for Happiness
mojombo
377
70k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.7k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Transcript
Loki Like Prometheus, but for logs Goutham Veeramachaneni @gouthamve DevOpsDaysIndia
2018
putadent gouthamve
QQQ
Outline • Quick Grafana intro • Full observability • Loki
• The future
Grafana intro
Grafana From Dashboarding solution To Observability platform
Unified way to look at data from different sources Logos
of datasources
Full Observability Picture credits: Peter Bourgon
Troubleshooting journey
Instrumenting an app
App • Classic 3-tiered app • Deployed in Kubernetes •
It’s running, but how is it doing? Load balancers App servers DB servers
Add instrumentation • Prometheus for metrics • Loki for logs
• Jaeger for traces
Change app • Introduce a bug • Let’s debug this
with our new tools
Demo: http://localhost:3000/explore
Loki: Grafana logging in detail
Goal: Keeping it simple https://twitter.com/alicegoldfuss/status/981947777256079360
Logs should be cheap! Existing Solutions are hard to scale
and operate You don’t need full text search / indexing Metrics, Logs and Traces need to work together More Goals
Logging for Kubernetes {job=”app1”} {job=”app3”} {job=”app2”}
Logging for Kubernetes (2) {job=”app1”} {job=”app3”} {job=”app2”}
Service Discovery for Grafana Logging • Prometheus-style service discovery of
logging targets • Labels are indexed as metadata, e.g.: {job=”app1”} • Relabeling rules
Logging architecture {job=”app1”} {job=”app2”} Node Logging agent Logging service Logging
datasource
Logging architecture {job=”app1”} {job=”app2”} Node Promtail Loki Logging datasource
Loki: Architecture Your jobs Distributor Ingester Index Chunks Querier
Loki: Write Path Your jobs Distributor Ingester Index Chunks
Distributor Distributor Ingester Use consistent hashing to assign a timestream
to an ingester.
Loki: Write Path Ingester {job=”app1”, instance=”...”,..} {job=”app2”, instance=”...”,..} Log line
Loki: Write Path Ingester {job=”app2”, instance=”...”,..} Index Chunks
Loki: Read Path Ingester Index Chunks Querier
Loki: Scalability Index Chunks Bigtable DynamoDB Cassandra BOLTDB GCS S3
Ceph / Minio FILES Distributor Ingester Querier
Loki: Status • Alpha software • Focus on correctness, ease-of-use
first, performance later • Onboard paying customers in Q1 2019 • Beyond Kubernetes
• Cheap but not free :) • Grep + Parsing
in browser • Can’t do complicated queries Loki: Caveats
What we’re working on
Enable Explore UI (BETA: Prometheus) Behind feature flag. To enable,
edit Grafana config ini file [explore] enabled = true Set up a datasource that supports Explore, e.g., Prometheus. Will be released in 6.0 (Feb 2019)
Integrate Tracing • Associate traces with logs and metrics •
Labels FTW • Aiming for Q2 2019
Logging UI needs to be refined still behind feature flag,
feedback welcome: @davkals or david@grafana.com UX improvements on logs and metrics views Performance improvements
One last thing...
https://www.grafanacon.org/2019/
Tack for listening Questions to goutham@grafana.com @putadent