Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
DevopsdaysIndia 2018: Loki, Prometheus but for logs
Search
Goutham Veeramachaneni
December 09, 2018
Programming
1
1.9k
DevopsdaysIndia 2018: Loki, Prometheus but for logs
Goutham Veeramachaneni
December 09, 2018
Tweet
Share
More Decks by Goutham Veeramachaneni
See All by Goutham Veeramachaneni
Storage in 2.0
gouthamve
1
100
Other Decks in Programming
See All in Programming
Javaエンジニアのための Nodejs/Nuxt3入門
hidekatsu_izuno
0
280
What We Can Learn From OSS
inouehi
0
400
コードレビューで学ぶ!Kotlinオブジェクト指向デザインパターン
akkie76
2
170
今の SmartHR にエンジニアで入社するとどうなるの?
daisukeshinoku
5
4.6k
コーンフレークから始める モデリング会話入門
ogurotakayuki
0
280
今、知っておきたい! 生成AIエージェントの世界
elith
3
340
StoreKit2によるiOSのアプリ内課金のリニューアル
kangnux
0
100
App Router への移行は「改善」となり得るのか?/ Can migration to App Router be an improvement
takefumiyoshii
8
2.1k
デザインシステムで Tailwind CSSとCSS in JSに分散投資をしたら良かった話
fsubal
18
4.8k
Java 22 Overview
kishida
1
170
CA.swift19 恋するAIアプリ開発の裏側
oskmr
0
330
ADRを一年運用してみた/adr_after_a_year
hanhan1978
7
2.2k
Featured
See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
72
8.2k
The Brand Is Dead. Long Live the Brand.
mthomps
48
28k
Large-scale JavaScript Application Architecture
addyosmani
503
110k
The Pragmatic Product Professional
lauravandoore
24
5.8k
Building Applications with DynamoDB
mza
88
5.6k
Build your cross-platform service in a week with App Engine
jlugia
225
17k
Adopting Sorbet at Scale
ufuk
67
8.6k
Optimising Largest Contentful Paint
csswizardry
7
2.3k
WebSockets: Embracing the real-time Web
robhawkes
59
7k
How To Stay Up To Date on Web Technology
chriscoyier
782
250k
Rails Girls Zürich Keynote
gr2m
91
13k
It's Worth the Effort
3n
180
27k
Transcript
Loki Like Prometheus, but for logs Goutham Veeramachaneni @gouthamve DevOpsDaysIndia
2018
putadent gouthamve
QQQ
Outline • Quick Grafana intro • Full observability • Loki
• The future
Grafana intro
Grafana From Dashboarding solution To Observability platform
Unified way to look at data from different sources Logos
of datasources
Full Observability Picture credits: Peter Bourgon
Troubleshooting journey
Instrumenting an app
App • Classic 3-tiered app • Deployed in Kubernetes •
It’s running, but how is it doing? Load balancers App servers DB servers
Add instrumentation • Prometheus for metrics • Loki for logs
• Jaeger for traces
Change app • Introduce a bug • Let’s debug this
with our new tools
Demo: http://localhost:3000/explore
Loki: Grafana logging in detail
Goal: Keeping it simple https://twitter.com/alicegoldfuss/status/981947777256079360
Logs should be cheap! Existing Solutions are hard to scale
and operate You don’t need full text search / indexing Metrics, Logs and Traces need to work together More Goals
Logging for Kubernetes {job=”app1”} {job=”app3”} {job=”app2”}
Logging for Kubernetes (2) {job=”app1”} {job=”app3”} {job=”app2”}
Service Discovery for Grafana Logging • Prometheus-style service discovery of
logging targets • Labels are indexed as metadata, e.g.: {job=”app1”} • Relabeling rules
Logging architecture {job=”app1”} {job=”app2”} Node Logging agent Logging service Logging
datasource
Logging architecture {job=”app1”} {job=”app2”} Node Promtail Loki Logging datasource
Loki: Architecture Your jobs Distributor Ingester Index Chunks Querier
Loki: Write Path Your jobs Distributor Ingester Index Chunks
Distributor Distributor Ingester Use consistent hashing to assign a timestream
to an ingester.
Loki: Write Path Ingester {job=”app1”, instance=”...”,..} {job=”app2”, instance=”...”,..} Log line
Loki: Write Path Ingester {job=”app2”, instance=”...”,..} Index Chunks
Loki: Read Path Ingester Index Chunks Querier
Loki: Scalability Index Chunks Bigtable DynamoDB Cassandra BOLTDB GCS S3
Ceph / Minio FILES Distributor Ingester Querier
Loki: Status • Alpha software • Focus on correctness, ease-of-use
first, performance later • Onboard paying customers in Q1 2019 • Beyond Kubernetes
• Cheap but not free :) • Grep + Parsing
in browser • Can’t do complicated queries Loki: Caveats
What we’re working on
Enable Explore UI (BETA: Prometheus) Behind feature flag. To enable,
edit Grafana config ini file [explore] enabled = true Set up a datasource that supports Explore, e.g., Prometheus. Will be released in 6.0 (Feb 2019)
Integrate Tracing • Associate traces with logs and metrics •
Labels FTW • Aiming for Q2 2019
Logging UI needs to be refined still behind feature flag,
feedback welcome: @davkals or
[email protected]
UX improvements on logs and metrics views Performance improvements
One last thing...
https://www.grafanacon.org/2019/
Tack for listening Questions to
[email protected]
@putadent