DevopsdaysIndia 2018: Loki, Prometheus but for logs

DevopsdaysIndia 2018: Loki, Prometheus but for logs


Goutham Veeramachaneni

December 09, 2018


  1. Loki Like Prometheus, but for logs Goutham Veeramachaneni @gouthamve DevOpsDaysIndia

  2. putadent gouthamve

  3. QQQ

  4. Outline • Quick Grafana intro • Full observability • Loki

    • The future
  5. Grafana intro

  6. Grafana From Dashboarding solution To Observability platform

  7. Unified way to look at data from different sources Logos

    of datasources
  8. Full Observability Picture credits: Peter Bourgon

  9. Troubleshooting journey

  10. Instrumenting an app

  11. App • Classic 3-tiered app • Deployed in Kubernetes •

    It’s running, but how is it doing? Load balancers App servers DB servers
  12. Add instrumentation • Prometheus for metrics • Loki for logs

    • Jaeger for traces
  13. Change app • Introduce a bug • Let’s debug this

    with our new tools
  14. Demo: http://localhost:3000/explore

  15. Loki: Grafana logging in detail

  16. Goal: Keeping it simple

  17. Logs should be cheap! Existing Solutions are hard to scale

    and operate You don’t need full text search / indexing Metrics, Logs and Traces need to work together More Goals
  18. Logging for Kubernetes {job=”app1”} {job=”app3”} {job=”app2”}

  19. Logging for Kubernetes (2) {job=”app1”} {job=”app3”} {job=”app2”}

  20. Service Discovery for Grafana Logging • Prometheus-style service discovery of

    logging targets • Labels are indexed as metadata, e.g.: {job=”app1”} • Relabeling rules
  21. Logging architecture {job=”app1”} {job=”app2”} Node Logging agent Logging service Logging

  22. Logging architecture {job=”app1”} {job=”app2”} Node Promtail Loki Logging datasource

  23. Loki: Architecture Your jobs Distributor Ingester Index Chunks Querier

  24. Loki: Write Path Your jobs Distributor Ingester Index Chunks

  25. Distributor Distributor Ingester Use consistent hashing to assign a timestream

    to an ingester.
  26. Loki: Write Path Ingester {job=”app1”, instance=”...”,..} {job=”app2”, instance=”...”,..} Log line

  27. Loki: Write Path Ingester {job=”app2”, instance=”...”,..} Index Chunks

  28. Loki: Read Path Ingester Index Chunks Querier

  29. Loki: Scalability Index Chunks Bigtable DynamoDB Cassandra BOLTDB GCS S3

    Ceph / Minio FILES Distributor Ingester Querier
  30. Loki: Status • Alpha software • Focus on correctness, ease-of-use

    first, performance later • Onboard paying customers in Q1 2019 • Beyond Kubernetes
  31. • Cheap but not free :) • Grep + Parsing

    in browser • Can’t do complicated queries Loki: Caveats
  32. What we’re working on

  33. Enable Explore UI (BETA: Prometheus) Behind feature flag. To enable,

    edit Grafana config ini file [explore] enabled = true Set up a datasource that supports Explore, e.g., Prometheus. Will be released in 6.0 (Feb 2019)
  34. Integrate Tracing • Associate traces with logs and metrics •

    Labels FTW • Aiming for Q2 2019
  35. Logging UI needs to be refined still behind feature flag,

    feedback welcome: @davkals or UX improvements on logs and metrics views Performance improvements
  36. One last thing...


  38. Tack for listening Questions to @putadent