Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
DevopsdaysIndia 2018: Loki, Prometheus but for ...
Search
Goutham Veeramachaneni
December 09, 2018
Programming
1
2.1k
DevopsdaysIndia 2018: Loki, Prometheus but for logs
Goutham Veeramachaneni
December 09, 2018
Tweet
Share
More Decks by Goutham Veeramachaneni
See All by Goutham Veeramachaneni
Storage in 2.0
gouthamve
1
120
Other Decks in Programming
See All in Programming
CDKを使ったPagerDuty連携インフラのテンプレート化
shibuya_shogo
0
110
Django NinjaによるAPI開発の効率化とリプレースの実践
kashewnuts
1
250
color-scheme: light dark; を完全に理解する
uhyo
7
490
パスキーのすべて ── 導入・UX設計・実装の紹介 / 20250213 パスキー開発者の集い
kuralab
3
890
ソフトウェアエンジニアの成長
masuda220
PRO
12
2.1k
新宿駅構内を三人称視点で探索してみる
satoshi7190
2
120
Boost Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
790
メンテが命: PHPフレームワークのコンテナ化とアップグレード戦略
shunta27
0
300
Honoのおもしろいミドルウェアをみてみよう
yusukebe
1
230
お前もAI鬼にならないか?👹Bolt & Cursor & Supabase & Vercelで人間をやめるぞ、ジョジョー!👺
taishiyade
7
4.2k
データの整合性を保つ非同期処理アーキテクチャパターン / Async Architecture Patterns
mokuo
54
19k
Honoとフロントエンドの 型安全性について
yodaka
7
1.4k
Featured
See All Featured
How to Ace a Technical Interview
jacobian
276
23k
Agile that works and the tools we love
rasmusluckow
328
21k
Designing for Performance
lara
604
68k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
45
9.4k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
4
430
Rails Girls Zürich Keynote
gr2m
94
13k
The Cult of Friendly URLs
andyhume
78
6.2k
YesSQL, Process and Tooling at Scale
rocio
172
14k
Embracing the Ebb and Flow
colly
84
4.6k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.3k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Transcript
Loki Like Prometheus, but for logs Goutham Veeramachaneni @gouthamve DevOpsDaysIndia
2018
putadent gouthamve
QQQ
Outline • Quick Grafana intro • Full observability • Loki
• The future
Grafana intro
Grafana From Dashboarding solution To Observability platform
Unified way to look at data from different sources Logos
of datasources
Full Observability Picture credits: Peter Bourgon
Troubleshooting journey
Instrumenting an app
App • Classic 3-tiered app • Deployed in Kubernetes •
It’s running, but how is it doing? Load balancers App servers DB servers
Add instrumentation • Prometheus for metrics • Loki for logs
• Jaeger for traces
Change app • Introduce a bug • Let’s debug this
with our new tools
Demo: http://localhost:3000/explore
Loki: Grafana logging in detail
Goal: Keeping it simple https://twitter.com/alicegoldfuss/status/981947777256079360
Logs should be cheap! Existing Solutions are hard to scale
and operate You don’t need full text search / indexing Metrics, Logs and Traces need to work together More Goals
Logging for Kubernetes {job=”app1”} {job=”app3”} {job=”app2”}
Logging for Kubernetes (2) {job=”app1”} {job=”app3”} {job=”app2”}
Service Discovery for Grafana Logging • Prometheus-style service discovery of
logging targets • Labels are indexed as metadata, e.g.: {job=”app1”} • Relabeling rules
Logging architecture {job=”app1”} {job=”app2”} Node Logging agent Logging service Logging
datasource
Logging architecture {job=”app1”} {job=”app2”} Node Promtail Loki Logging datasource
Loki: Architecture Your jobs Distributor Ingester Index Chunks Querier
Loki: Write Path Your jobs Distributor Ingester Index Chunks
Distributor Distributor Ingester Use consistent hashing to assign a timestream
to an ingester.
Loki: Write Path Ingester {job=”app1”, instance=”...”,..} {job=”app2”, instance=”...”,..} Log line
Loki: Write Path Ingester {job=”app2”, instance=”...”,..} Index Chunks
Loki: Read Path Ingester Index Chunks Querier
Loki: Scalability Index Chunks Bigtable DynamoDB Cassandra BOLTDB GCS S3
Ceph / Minio FILES Distributor Ingester Querier
Loki: Status • Alpha software • Focus on correctness, ease-of-use
first, performance later • Onboard paying customers in Q1 2019 • Beyond Kubernetes
• Cheap but not free :) • Grep + Parsing
in browser • Can’t do complicated queries Loki: Caveats
What we’re working on
Enable Explore UI (BETA: Prometheus) Behind feature flag. To enable,
edit Grafana config ini file [explore] enabled = true Set up a datasource that supports Explore, e.g., Prometheus. Will be released in 6.0 (Feb 2019)
Integrate Tracing • Associate traces with logs and metrics •
Labels FTW • Aiming for Q2 2019
Logging UI needs to be refined still behind feature flag,
feedback welcome: @davkals or
[email protected]
UX improvements on logs and metrics views Performance improvements
One last thing...
https://www.grafanacon.org/2019/
Tack for listening Questions to
[email protected]
@putadent