Loki
Like Prometheus, but for
logs
Goutham Veeramachaneni
@gouthamve
DevOpsDaysIndia 2018
Slide 2
Slide 2 text
putadent gouthamve
Slide 3
Slide 3 text
QQQ
Slide 4
Slide 4 text
Outline
● Quick Grafana intro
● Full observability
● Loki
● The future
Slide 5
Slide 5 text
Grafana intro
Slide 6
Slide 6 text
Grafana
From
Dashboarding solution
To
Observability platform
Slide 7
Slide 7 text
Unified way
to look at
data from
different
sources
Logos of datasources
Slide 8
Slide 8 text
Full Observability
Picture credits: Peter Bourgon
Slide 9
Slide 9 text
Troubleshooting journey
Slide 10
Slide 10 text
Instrumenting an app
Slide 11
Slide 11 text
App
● Classic 3-tiered app
● Deployed in Kubernetes
● It’s running, but how is it
doing?
Load balancers
App servers
DB servers
Slide 12
Slide 12 text
Add instrumentation
● Prometheus for metrics
● Loki for logs
● Jaeger for traces
Slide 13
Slide 13 text
Change app
● Introduce a bug
● Let’s debug this with our new tools
Slide 14
Slide 14 text
Demo: http://localhost:3000/explore
Slide 15
Slide 15 text
Loki: Grafana logging in detail
Slide 16
Slide 16 text
Goal:
Keeping it
simple
https://twitter.com/alicegoldfuss/status/981947777256079360
Slide 17
Slide 17 text
Logs should be cheap!
Existing Solutions are hard to scale and operate
You don’t need full text search / indexing
Metrics, Logs and Traces need to work together
More Goals
Slide 18
Slide 18 text
Logging for Kubernetes
{job=”app1”}
{job=”app3”}
{job=”app2”}
Slide 19
Slide 19 text
Logging for Kubernetes (2)
{job=”app1”}
{job=”app3”}
{job=”app2”}
Slide 20
Slide 20 text
Service Discovery
for Grafana Logging
● Prometheus-style service
discovery of logging targets
● Labels are indexed as
metadata, e.g.: {job=”app1”}
● Relabeling rules
Loki: Status
● Alpha software
● Focus on correctness, ease-of-use first, performance later
● Onboard paying customers in Q1 2019
● Beyond Kubernetes
Slide 31
Slide 31 text
● Cheap but not free :)
● Grep + Parsing in browser
● Can’t do complicated queries
Loki: Caveats
Slide 32
Slide 32 text
What we’re working on
Slide 33
Slide 33 text
Enable Explore UI (BETA: Prometheus)
Behind feature flag. To enable, edit Grafana config ini file
[explore]
enabled = true
Set up a datasource that supports Explore, e.g., Prometheus.
Will be released in 6.0 (Feb 2019)
Slide 34
Slide 34 text
Integrate Tracing
● Associate traces with logs and metrics
● Labels FTW
● Aiming for Q2 2019
Slide 35
Slide 35 text
Logging UI needs to be refined
still behind feature flag, feedback welcome: @davkals or david@grafana.com
UX improvements on logs and metrics views
Performance improvements
Slide 36
Slide 36 text
One last thing...
Slide 37
Slide 37 text
https://www.grafanacon.org/2019/
Slide 38
Slide 38 text
Tack for
listening
Questions to
goutham@grafana.com
@putadent