CNCF San Diego 2019 - Introduction To Thanos

An introduction to the Thanos project. https://thanos.io

Transcript

1. @ThanosMetrics Dominic Green, Software Engineer (Improbable) dom@improbable.io Lucas Servén, Senior Software Engineer (Red Hat) lserven@redhat.com KubeCon + CNCF San Diego - November 19, 2019 domgreen squat Introduction to Thanos
12. @ThanosMetrics Dominic Green ▪ Software Engineer @ Improbable ▪ Observability Team ▪ OSS Contributor ◦ Thanos ◦ go-grpc-middleware ◦ go-httpwares ▪ Meetup Organiser ◦ Prometheus London ◦ London Gophers
13. @ThanosMetrics Lucas Servén Marín ▪ Senior Software Engineer @ Red Hat ▪ OpenShift Monitoring Team ▪ OSS Contributor ◦ Thanos ◦ Prometheus Operator ◦ Kilo ▪ Twin
14. @ThanosMetrics
15. @ThanosMetrics Thanos Community • Fully open source from start • Started in Nov 2017 • Part of CNCF Sandbox • 4600+ Github stars • 162+ contributors • ~500 slack users • 8 maintainers, 3 triagers from 7 different companies. • Transparent Governance • Prometheus Ecosystem
16. @ThanosMetrics Production Users
17. @ThanosMetrics
18. @ThanosMetrics
19. @ThanosMetrics
20. @ThanosMetrics Collecting, processing, aggregating, and displaying real-time quantitative data about a system, such as query counts and types, error counts and types, processing times, and server lifetimes. https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ “ ” Define: Monitoring
21. @ThanosMetrics Thanos Begins
22. @ThanosMetrics Prometheus /metrics # TYPE counter app_request_total 1337 # TYPE gauge app_request_in_flight_total 3 # TYPE histogram app_request_duration_ms_bucket {le="0.005"} 500 app_request_duration_ms_bucket {le="0.01"} 213
23. @ThanosMetrics Prometheus Scrape Engine Rule + Alert Engine Query Engine Compactor Local Storage SVC 1 SVC 2 SVC 3 /metrics Alertmanager Grafana
24. @ThanosMetrics Prometheus Limitations SVC 2 SVC 3 SVC 4 NODE 1 NODE 2 SVC 1
25. @ThanosMetrics Prometheus Limitations SVC 2 SVC 3 SVC 4 NODE 1 NODE 2 SVC 1
26. @ThanosMetrics Prometheus Limitations SVC 2 SVC 3 SVC 4 NODE 1 SVC 1 NODE 2
27. @ThanosMetrics Prometheus Limitations NODE 1 Local Storage
28. @ThanosMetrics Prometheus Limitations NODE 1 Local Storage
29. @ThanosMetrics Monitoring Cluster Prometheus Limitations Cluster 1 Cluster 2 Cluster 3 /federate /federate /federate
30. @ThanosMetrics Monitoring Cluster Prometheus Limitations Cluster 1 Cluster 2 Cluster 3 /federate /federate /federate
31. @ThanosMetrics Monitoring Cluster Prometheus Limitations Cluster 1 Cluster 2 Cluster 3 /federate /federate /federate
32. @ThanosMetrics Monitoring Cluster Prometheus Limitations /federate /federate /federate /federate ... Cluster 1 Cluster 2 Cluster 3 Cluster N
33. @ThanosMetrics
34. @ThanosMetrics Project Aims Downsampling & Compaction Unlimited Retention Prometheus Compatible Global Query View
35. @ThanosMetrics Monitoring Cluster Scalability /federate /federate /federate Cluster 1 Cluster 2 Cluster 3
36. @ThanosMetrics Monitoring Cluster Scalability /query /query /query Cluster 1 Cluster 2 Cluster 3 ?
37. @ThanosMetrics Monitoring Cluster /query /query /query /query ... Cluster 1 Cluster 2 Cluster 3 Cluster N ? Scalability
38. @ThanosMetrics Monitoring Cluster Cluster 1 ... ? ? Cluster 2 ? Cluster N ? /query /query /query Scalability
39. @ThanosMetrics Monitoring Cluster Cluster 1 gRPC ... ? ? Cluster 2 gRPC ? Cluster N gRPC ? Scalability
40. @ThanosMetrics Monitoring Cluster Cluster 1 gRPC ... Cluster 2 gRPC Cluster N gRPC Scalability Query
41. @ThanosMetrics Store API Thanos Deep Dive: Inside a Distributed Monitoring System Tomorrow (5:20)
42. @ThanosMetrics Monitoring Cluster High Availability Cluster 1 gRPC ... Cluster 2 gRPC Cluster N gRPC Query Query
43. @ThanosMetrics Monitoring Cluster High Availability Cluster 1 gRPC ... Cluster 2 Cluster N gRPC gRPC gRPC gRPC gRPC Query
44. @ThanosMetrics Monitoring Cluster High Availability Cluster 1 ... Cluster 2 Cluster N StoreAPI StoreAPI StoreAPI Query Query Query Query
45. @ThanosMetrics Monitoring Cluster Improved Storage Cluster 1 ... Cluster 2 Cluster N Query Query Query Query
46. @ThanosMetrics Monitoring Cluster TSDB Uploaded to Object Storage Cluster 1 ... Cluster 2 Cluster N Object Storage Query Query Query Query TSDB blocks TSDB blocks
47. @ThanosMetrics Monitoring Cluster Store Gateway Cluster 1 ... Cluster 2 Cluster N Object Storage Query Store Query Query Query
48. @ThanosMetrics Monitoring Cluster Compactor Cluster 1 ... Query Cluster 2 Cluster N Object Storage Compact Store Query Query Query Thanos Deep Dive: Inside a Distributed Monitoring System Tomorrow (5:20)
49. @ThanosMetrics Thanos, Prometheus At Scale Query Sidecar Store Compact
50. @ThanosMetrics Scalability Cluster 1 Cluster 2 Cluster 3 Monitoring Cluster Query ?
51. @ThanosMetrics Scalability Cluster 1 Cluster 2 Cluster 3 Monitoring Cluster Query ?
52. @ThanosMetrics Scalability Monitoring Cluster ... Cluster 1 Cluster 2 Cluster 3 Cluster N Query ?
53. @ThanosMetrics Hash Ring ? ? ? ? ? ?
54. @ThanosMetrics Hash Ring ? ? ? ? ? ?
55. @ThanosMetrics ? ? ? ? ? ? Scalability Monitoring Cluster ... Cluster 1 Cluster 2 Cluster 3 Cluster N Query
56. @ThanosMetrics ? ? ? ? ? ? Scalability ... Cluster 1 Cluster 2 Cluster 3 Cluster N Monitoring Cluster Object Storage Query Store
57. @ThanosMetrics ? ? ? ? ? Scalability ... Cluster 1 Cluster 2 Cluster 3 Cluster N Monitoring Cluster Object Storage Query Store Receive Shipping Metrics From the Edge Tomorrow (11:50)
58. @ThanosMetrics ? ? ? ? ? Recording & Alerting Rules ... Cluster 1 Cluster 2 Cluster 3 Cluster N Object Storage Query Store Receive Rule
59. @ThanosMetrics Thanos, Prometheus At Scale Query Receive Store Rule
60. @ThanosMetrics Distributed Prometheus Scrape Engine Rule + Alert Engine Query Engine Compactor Local Storage Alertmanager Grafana SVC 1 SVC 2 SVC 3 /metrics
61. @ThanosMetrics Distributed Prometheus Scrape Engine Rule + Alert Engine Compactor Local Storage Alertmanager SVC 1 SVC 2 SVC 3 /metrics Grafana Query
62. @ThanosMetrics Distributed Prometheus Sidecar/Receive Compactor Object Storage Grafana Query SVC 1 SVC 2 SVC 3 /metrics Rule + Alert Engine Alertmanager
63. @ThanosMetrics Distributed Prometheus Object Storage Compact/Store SVC 1 SVC 2 SVC 3 /metrics Sidecar/Receive Grafana Query Rule + Alert Engine Alertmanager
64. @ThanosMetrics Distributed Prometheus Object Storage Compact/Store Rule Alertmanager SVC 1 SVC 2 SVC 3 /metrics Sidecar/Receive Grafana Query
65. @ThanosMetrics https://www.katacoda.com/thanos
66. @ThanosMetrics Thank You! https://thanos.io
67. @ThanosMetrics KubeCon + CloudNativeCon Shipping Metrics From the Edge ▪ Matthias Loibl, Red Hat ▪ Wednesday November 20, 2019 ▪ 11:50am (Room 11AB) Thanos Deep Dive: Inside a Distributed Monitoring System ▪ Bartłomiej Płotka & Frederic Branczyk, RedHat ▪ Wednesday November 20, 2019 ▪ 5:20pm (Room 6C)
68. @ThanosMetrics FAQ ▪ How does Thanos compare to Cortex/M3DB/X/Y/Z? ▪ When do I use the sidecar vs the receiver? ▪ Sounds too good to be true; what are the bottlenecks? ▪ Can I use Thanos with my favourite object storage provider? ▪ How do I know if I need Thanos vs a big Prometheus?