DevopsdaysIndia 2018: Loki, Prometheus but for logs

Transcript

1. Loki Like Prometheus, but for logs Goutham Veeramachaneni @gouthamve DevOpsDaysIndia

   2018

2. putadent gouthamve

3. QQQ

4. Outline • Quick Grafana intro • Full observability • Loki

   • The future

5. Grafana intro

6. Grafana From Dashboarding solution To Observability platform

7. Unified way to look at data from different sources Logos

   of datasources

8. Full Observability Picture credits: Peter Bourgon

9. Troubleshooting journey

10. Instrumenting an app

11. App • Classic 3-tiered app • Deployed in Kubernetes •

    It’s running, but how is it doing? Load balancers App servers DB servers

12. Add instrumentation • Prometheus for metrics • Loki for logs

    • Jaeger for traces

13. Change app • Introduce a bug • Let’s debug this

    with our new tools

14. Demo: http://localhost:3000/explore

15. Loki: Grafana logging in detail

16. Goal: Keeping it simple https://twitter.com/alicegoldfuss/status/981947777256079360

17. Logs should be cheap! Existing Solutions are hard to scale

    and operate You don’t need full text search / indexing Metrics, Logs and Traces need to work together More Goals

18. Logging for Kubernetes {job=”app1”} {job=”app3”} {job=”app2”}

19. Logging for Kubernetes (2) {job=”app1”} {job=”app3”} {job=”app2”}

20. Service Discovery for Grafana Logging • Prometheus-style service discovery of

    logging targets • Labels are indexed as metadata, e.g.: {job=”app1”} • Relabeling rules

21. Logging architecture {job=”app1”} {job=”app2”} Node Logging agent Logging service Logging

    datasource

22. Logging architecture {job=”app1”} {job=”app2”} Node Promtail Loki Logging datasource

23. Loki: Architecture Your jobs Distributor Ingester Index Chunks Querier

24. Loki: Write Path Your jobs Distributor Ingester Index Chunks

25. Distributor Distributor Ingester Use consistent hashing to assign a timestream

    to an ingester.

26. Loki: Write Path Ingester {job=”app1”, instance=”...”,..} {job=”app2”, instance=”...”,..} Log line

27. Loki: Write Path Ingester {job=”app2”, instance=”...”,..} Index Chunks

28. Loki: Read Path Ingester Index Chunks Querier

29. Loki: Scalability Index Chunks Bigtable DynamoDB Cassandra BOLTDB GCS S3

    Ceph / Minio FILES Distributor Ingester Querier

30. Loki: Status • Alpha software • Focus on correctness, ease-of-use

    first, performance later • Onboard paying customers in Q1 2019 • Beyond Kubernetes

31. • Cheap but not free :) • Grep + Parsing

    in browser • Can’t do complicated queries Loki: Caveats

32. What we’re working on

33. Enable Explore UI (BETA: Prometheus) Behind feature flag. To enable,

    edit Grafana config ini file [explore] enabled = true Set up a datasource that supports Explore, e.g., Prometheus. Will be released in 6.0 (Feb 2019)

34. Integrate Tracing • Associate traces with logs and metrics •

    Labels FTW • Aiming for Q2 2019

35. Logging UI needs to be refined still behind feature flag,

    feedback welcome: @davkals or david@grafana.com UX improvements on logs and metrics views Performance improvements

36. One last thing...

37. https://www.grafanacon.org/2019/

38. Tack for listening Questions to goutham@grafana.com @putadent