Building a Global Cross-Cloud Monitoring Platform

KubeCon + CloudNativeCon | Open Source Summit China 2019

How Improbable have leveraged Open Source to build a Global Cross-Cloud monitoring platform.

Transcript

1. Building a Global Cross-Cloud Monitoring Platform Dominic Green, Software Engineer dom@improbable.io, @domgreen 25th June 2019, Shanghai China Yifan Zhao, Improbable China, Co-founder yifan@improbable.io
2. Yifan Zhao ▪ Co-founder of Improbable China ◦ Heads up the Engineer Division ▪ Part of Improbable’s Founding Team ◦ Core contributor to the SpatialOS platform ◦ Lead the built-out of the infrastructure team in London
4. If a tree falls in a forest and no one is around to hear it, does it make a sound? 如果森林里的一棵树倒了，当时周围没有人， 它发出声音了吗？
13. Agenda Single Cluster Multi-Cluster Multi-Cloud
14. 370 Founded: Games in Development: 20+ 2012 Employees: Our Mission: Make Impossible Games Possible "Improbable’s platform, SpatialOS, is designed to let anyone build massive simulations, running in the cloud: imagine Minecraft with thousands of players in the same space…. Its ultimate goal: to create totally immersive, persistent virtual worlds." - WIRED, May 2017
15. ▪ Software Engineer @ Improbable ▪ Observability Team ▪ OSS Contributor ◦ Thanos ◦ go-grpc-middleware ◦ go-httpwares ▪ Meetup Organiser ◦ Prometheus London ◦ London Gophers Dominic Green
16. Define: Monitoring Collecting, processing, aggregating, and displaying real-time quantitative data about a system, such as query counts and types, error counts and types, processing times, and server lifetimes. https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ @domgreen “ ”
17. @domgreen
18. Thanos Begins
19. Prometheus /metrics # TYPE counter app_request_total 1337 # TYPE gauge app_request_in_flight_total 3 # TYPE histogram app_request_duration_ms_bucket {le="0.005"} 500 app_request_duration_ms_bucket {le="0.01"} 213 @domgreen
20. Prometheus @domgreen Query Engine Scrape Engine Compactor Rule & Alert Engine Prometheus Service X Service X Services /metrics every 15s HTTP Query API Grafana Alertmanager Local storage
22. Prometheus pod-a pod-b pod-c apiVersion: v1 kind: Pod metadata: annotations: prometheus.io/path: /metrics prometheus.io/port: "8080" prometheus.io/scheme: http prometheus.io/scrape: "true" scrape_configs: - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: ... Prometheus SSD blocks @domgreen
23. Single Cluster Single Cluster Prometheus Grafana @domgreen
24. ▪ Basis for workload management ▪ Kubernetes Service Discovery for discovering workloads ▪ Mature tooling and automation Single Cluster Kubernetes Prometheus ▪ Collection of data from workloads ▪ Data queried directly from Prometheus Scrapers ▪ Fast becoming industry standard for metric collection @domgreen
25. • Cons ◦ Redundancy ◦ Latency Single Cluster • Pros ◦ Simple ◦ Easy to Monitor @domgreen
26. 7ms 30ms
27. 70ms 130ms 280ms 300ms
28. 22ms 10ms 125ms 30ms
29. Multi-Cluster Single Cluster Prometheus Grafana @domgreen
30. Multi-Cluster - Prometheus US Games Cluster Prometheus EU Games Cluster Prometheus Hub Cluster (EU || US) Prometheus Grafana @domgreen
31. Multi-Cluster - Federation US Games Cluster Prometheus EU Games Cluster Prometheus Hub Cluster (EU || US) Prometheus Grafana /federate /federate @domgreen
32. Thanos Begins
33. Thanos Project ▪ Global query view of Metrics. ▪ Unlimited retention of Metrics. ▪ High availability of components, including Prometheus. ▪ Downsampling of Metrics. @domgreen
34. Cluster N Cluster 0 Multi-Cluster - Thanos Sidecar US Games Cluster SSD Prometheus EU Games Cluster SSD Prometheus ... gRPC (Store API) gRPC (Store API) @domgreen
35. Cluster N Cluster 0 Multi-Cluster - Thanos Query US Games Cluster SSD Prometheus EU Games Cluster SSD Prometheus ... Query HTTP & gRPC (Store API) @domgreen
36. Multi-Cluster - Federation US Games Cluster Prometheus EU Games Cluster Prometheus Hub Cluster (EU || US) Prometheus Grafana /federate /federate @domgreen
37. Hub Cluster (EU || US) Cluster N Cluster 0 Multi-Cluster - Thanos Query SSD Prometheus SSD Prometheus ... Query Query Hub Cluster (EU || US) Grafana SSD Prometheus Query @domgreen
38. Multi-Cluster - Thanos Query @domgreen
39. Cluster Object Storage SSD Prometheus blocks Currently Supported: - Google Cloud Storage - S3 - Azure Blob Storage - Tencent - Aliyun OSS (soon) Multi-Cluster - Thanos Sidecar @domgreen
40. Hub Cluster (EU || US) Cluster N Cluster 0 Multi-Cluster - Storage SSD Prometheus SSD Prometheus ... Query Query Hub Cluster (EU || US) Grafana SSD Prometheus Query Object Storage @domgreen
41. Hub Cluster (EU || US) Cluster N Cluster 0 Multi-Cluster - Storage SSD Prometheus SSD Prometheus ... Query Query Hub Cluster (EU || US) Grafana SSD Prometheus Query Object Storage Store gRPC (Store API) @domgreen
42. Multi-Cluster - Storage @domgreen
43. Hub Cluster (EU || US) Cluster N Multi-Cluster - High Availability SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query “replica”: thanos-0 “replica”: thanos-1 “replica”: thanos-0 “replica”: thanos-1 @domgreen
44. Multi-Cluster - High Availability @domgreen
45. Multi-Cluster - High Availability @domgreen
46. Hub Cluster (EU || US) Cluster N Multi-Cluster - Compaction SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query Compact @domgreen
47. Multi-Cluster - Compaction @domgreen
48. Hub Cluster (EU || US) Cluster N Multi-Cluster SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query Compact @domgreen
49. Multi-Cluster Thanos ▪ Global View, Retention, HA, Downsampling ▪ Pulls Metrics from Object Storage or Thanos Sidecar ▪ Builds on existing Prometheus infrastructure ▪ Consistent approach in all clusters ▪ Kubernetes Service Discovery for discovering workloads ▪ Mature tooling and automation Kubernetes Prometheus ▪ Collection of data from workloads ▪ Federation can be problematic @domgreen
50. • Cons ◦ Observability is harder ◦ Increased Complexity ◦ Automation? ◦ Tooling? Multi-Cluster • Pros ◦ Reduced Latency ◦ High Availability ▪ Cluster Level ▪ Workload level ◦ Global Query ◦ Long Term Metrics @domgreen
51. No plan survives first contact with the enemy.
52. Hub Cluster (EU || US) Cluster N Multi-Cluster - Networking SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query Compact @domgreen
53. Hub Cluster (EU || US) Cluster N Multi-Cluster - Networking SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query Compact @domgreen
54. Hub Cluster (EU || US) Cluster N Multi-Cluster - Networking SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query Compact @domgreen
55. Hub Cluster (EU || US) Cluster N Multi-Cloud - Networking SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query Compact @domgreen
57. Hub Cluster (EU || US) Cluster N Multi-Cloud - Envoy SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query Compact Envoy Envoy Envoy @domgreen
58. Hub Cluster (EU || US) Cluster N Multi-Cloud - Envoy SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query Compact Envoy Envoy Envoy https://envoy.gcp.i8e.io https://envoy.aws.i8e.io https://envoy.az.i8e.io https://envoy.ali.i8e.io @domgreen
59. Hub Cluster (EU || US) Cluster N Multi-Cloud - Persistence SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query Compact Envoy Envoy Envoy @domgreen
60. Hub Cluster (EU || US) Cluster N Multi-Cloud - Querying SSD Prometheus SSD Prometheus ... Query Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Object Storage Store Cluster N SSD Prometheus SSD Prometheus Query Compact Envoy Envoy Envoy @domgreen
61. Hub Cluster (EU || US) Multi-Cloud - Thanos Receive ... Hub Cluster (EU || US) SSD Prometheus Grafana SSD Prometheus Query Store Compact Cluster N SSD Prometheus SSD Prometheus Cluster N SSD Prometheus SSD Prometheus Receive Remote Write Object Storage @domgreen
62. Multi-Cloud Thanos ▪ Global View, Retention, HA, Downsampling ▪ Flexible StoreAPI allows for different usage scenarios ▪ Consistent approach in all clouds & clusters ▪ Kubernetes Service Discovery for discovering workloads Kubernetes Prometheus ▪ Collection of metrics from workloads ▪ TSDB Storage format Envoy ▪ Edge Proxy for same approach cross-cluster and cross-cloud communication @domgreen
63. • Cons ◦ Observability is harder ◦ Increased Complexity ◦ Automation?? ◦ Tooling?? Multi-Cloud • Pros ◦ Reduced Latency ◦ High Availability ▪ Cluster Level ▪ Workload level ◦ Global Query ◦ Long Term Metrics @domgreen
64. Summary Single Cluster Multi-Cluster Multi-Cloud
65. Learn More @domgreen • High Available + Scalable Prometheus with Thanos in Alibaba ◦ Guo'an Qin, Alibaba & Tao Li, Alibaba • Metric monitoring architecture at Improbable using Thanos ◦ Bartłomiej Płotka & Dominic Green ◦ https://improbable.io/blog/thanos-architecture-at-improbable • Autoscaling Multi-Cluster Observability with Thanos and Linkerd ◦ Andrew Seigner & Frederic Branczyk ◦ https://www.youtube.com/watch?v=qTxunwzYO0g • Thanos - Transforming Prometheus to a Global Scale in a Seven Simple Steps ◦ Bartłomiej Płotka ◦ https://www.youtube.com/watch?v=Iuo1EjCN5i4
66. We’re hiring! improbable.io