Slide 1

Slide 1 text

Loki Like Prometheus, but for logs Goutham Veeramachaneni @gouthamve DevOpsDaysIndia 2018

Slide 2

Slide 2 text

putadent gouthamve

Slide 3

Slide 3 text

QQQ

Slide 4

Slide 4 text

Outline ● Quick Grafana intro ● Full observability ● Loki ● The future

Slide 5

Slide 5 text

Grafana intro

Slide 6

Slide 6 text

Grafana From Dashboarding solution To Observability platform

Slide 7

Slide 7 text

Unified way to look at data from different sources Logos of datasources

Slide 8

Slide 8 text

Full Observability Picture credits: Peter Bourgon

Slide 9

Slide 9 text

Troubleshooting journey

Slide 10

Slide 10 text

Instrumenting an app

Slide 11

Slide 11 text

App ● Classic 3-tiered app ● Deployed in Kubernetes ● It’s running, but how is it doing? Load balancers App servers DB servers

Slide 12

Slide 12 text

Add instrumentation ● Prometheus for metrics ● Loki for logs ● Jaeger for traces

Slide 13

Slide 13 text

Change app ● Introduce a bug ● Let’s debug this with our new tools

Slide 14

Slide 14 text

Demo: http://localhost:3000/explore

Slide 15

Slide 15 text

Loki: Grafana logging in detail

Slide 16

Slide 16 text

Goal: Keeping it simple https://twitter.com/alicegoldfuss/status/981947777256079360

Slide 17

Slide 17 text

Logs should be cheap! Existing Solutions are hard to scale and operate You don’t need full text search / indexing Metrics, Logs and Traces need to work together More Goals

Slide 18

Slide 18 text

Logging for Kubernetes {job=”app1”} {job=”app3”} {job=”app2”}

Slide 19

Slide 19 text

Logging for Kubernetes (2) {job=”app1”} {job=”app3”} {job=”app2”}

Slide 20

Slide 20 text

Service Discovery for Grafana Logging ● Prometheus-style service discovery of logging targets ● Labels are indexed as metadata, e.g.: {job=”app1”} ● Relabeling rules

Slide 21

Slide 21 text

Logging architecture {job=”app1”} {job=”app2”} Node Logging agent Logging service Logging datasource

Slide 22

Slide 22 text

Logging architecture {job=”app1”} {job=”app2”} Node Promtail Loki Logging datasource

Slide 23

Slide 23 text

Loki: Architecture Your jobs Distributor Ingester Index Chunks Querier

Slide 24

Slide 24 text

Loki: Write Path Your jobs Distributor Ingester Index Chunks

Slide 25

Slide 25 text

Distributor Distributor Ingester Use consistent hashing to assign a timestream to an ingester.

Slide 26

Slide 26 text

Loki: Write Path Ingester {job=”app1”, instance=”...”,..} {job=”app2”, instance=”...”,..} Log line

Slide 27

Slide 27 text

Loki: Write Path Ingester {job=”app2”, instance=”...”,..} Index Chunks

Slide 28

Slide 28 text

Loki: Read Path Ingester Index Chunks Querier

Slide 29

Slide 29 text

Loki: Scalability Index Chunks Bigtable DynamoDB Cassandra BOLTDB GCS S3 Ceph / Minio FILES Distributor Ingester Querier

Slide 30

Slide 30 text

Loki: Status ● Alpha software ● Focus on correctness, ease-of-use first, performance later ● Onboard paying customers in Q1 2019 ● Beyond Kubernetes

Slide 31

Slide 31 text

● Cheap but not free :) ● Grep + Parsing in browser ● Can’t do complicated queries Loki: Caveats

Slide 32

Slide 32 text

What we’re working on

Slide 33

Slide 33 text

Enable Explore UI (BETA: Prometheus) Behind feature flag. To enable, edit Grafana config ini file [explore] enabled = true Set up a datasource that supports Explore, e.g., Prometheus. Will be released in 6.0 (Feb 2019)

Slide 34

Slide 34 text

Integrate Tracing ● Associate traces with logs and metrics ● Labels FTW ● Aiming for Q2 2019

Slide 35

Slide 35 text

Logging UI needs to be refined still behind feature flag, feedback welcome: @davkals or [email protected] UX improvements on logs and metrics views Performance improvements

Slide 36

Slide 36 text

One last thing...

Slide 37

Slide 37 text

https://www.grafanacon.org/2019/

Slide 38

Slide 38 text

Tack for listening Questions to [email protected] @putadent