Bootes - Envoy Control Plane Kubernetes Controller -

Transcript

1. Bootes - Envoy Control Plane Kubernetes Controller - Envoy Meetup Tokyo #2 Yuki Ito
2. Merpay Architect Team Backend Engineer Yuki Ito
3. Agenda •What is Bootes? •Implementation •Use Case
4. Agenda •What is Bootes? •Implementation •Use Case
5. Bootes https://github.com/110y/bootes
6. Bootes Envoy Control Plane implemented as a... Kubernetes Custom Controller
7. Bootes: Overview Bootes Config Config Config
8. Bootes: Overview Bootes Config Config Config kube-apiserver ! kubectl
9. Motivation •Simple Control Plane •Full Envoy features •Kubernetes Custom Resource Movement
10. e.g. Service Mesh Interface kind: TrafficSplit metadata: name: canary spec: service: website backends: - service: website-v1 weight: 90 - service: website-v2 weight: 10
11. e.g. Crossplane apiVersion: database.gcp.crossplane.io/v1beta1 kind: CloudSQLInstance metadata: name: cloudsqlpostgresql spec: forProvider: databaseVersion: POSTGRES_9_6 region: us-central1 settings: tier: db-custom-1-3840 dataDiskType: PD_SSD dataDiskSizeGb: 10
12. Agenda •What is Bootes? •Implementation •Use Case
13. Agenda •What is Bootes? •Implementation •Use Case
14. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works
15. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works
16. Envoy Control Plane Control Plane Config Config Config xDS API
17. Envoy Configurations •Listener •Route •Cluster •Endpoint
18. Envoy Configurations Listener Route Cluster Endpoint Endpoint Endpoint Endpoint Cluster
19. Envoy Configurations 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster Path: /service-1 Path: /service-2
20. Static Configurations static_resources: listeners: - address: socket_address: protocol: TCP address: 0.0.0.0 port_value: 5000 #... clusters: - name: service-1 connect_timeout: 1s type: STRICT_DNS lb_policy: ROUND_ROBIN #... envoy.yaml
21. Static Configurations > envoy -c envoy.yaml
22. Problem of Static Configurations Though simplistic, fairly complicated deployments can be created using static configurations and graceful hot restarts. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/dynamic_configuration
23. Dynamic Configurations Control Plane Route Listener Cluster xDS API
24. x Discovery Service API •Listener Discovery Service •Route Discovery Service •Cluster Discovery Service •Endpoint Discovery Service
25. x Discovery Service API •Listener Discovery Service •Route Discovery Service •Cluster Discovery Service •Endpoint Discovery Service
26. x Discovery Service API Control Plane Listener Listener Listener Listener Discovery Service
27. x Discovery Service API Control Plane Cluster Cluster Cluster Cluster Discovery Service
28. e.g. Cluster Discovery Service service ClusterDiscoveryService { rpc StreamClusters(stream discovery.v3.DiscoveryRequest) returns (stream discovery.v3.DiscoveryResponse) { } rpc DeltaClusters(stream discovery.v3.DeltaDiscoveryRequest) returns (stream discovery.v3.DeltaDiscoveryResponse) { } rpc FetchClusters(discovery.v3.DiscoveryRequest) returns (discovery.v3.DiscoveryResponse) { } } https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/cluster/v3/cds.proto cds.proto
29. x Discovery Service API •Listener Discovery Service •Route Discovery Service •Cluster Discovery Service •Endpoint Discovery Service
30. Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: lds-server cds_config: #... envoy.yaml
31. Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: lds-server cds_config: #... envoy.yaml
32. Dynamic Configurations envoy.yaml static_resources: clusters: - name: lds-server connect_timeout: 1s type: STRICT_DNS http2_protocol_options: {} load_assignment: cluster_name: lds-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ... port_value: ...
33. Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: gRPC grpc_services: - envoy_grpc: cluster_name: lds-server cds_config: #... static_resources: clusters: - name: lds-server connect_timeout: 1s type: LOGICAL_DNS http2_protocol_options: {} load_assignment: cluster_name: lds-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ... port_value: ... envoy.yaml
34. x Discovery Service API Control Plane Route Listener CDS LDS RDS CDS
35. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12
36. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster
37. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster RDS CDS
38. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 ❌ RDS
39. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster CDS
40. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster CDS RDS
41. Problem of xDS API To avoid downtime... 1. Cluster 2. Endpoint 3. Listener 4. Route
42. Problem of xDS API Control Plane Route Listener CDS LDS RDS CDS
43. Aggregated Discovery Service Control Plane Listener / Route / Cluster / Endpoint Aggregated Discovery Service
44. Aggregated Discovery Service service AggregatedDiscoveryService { rpc StreamAggregatedResources(stream DiscoveryRequest) returns (stream DiscoveryResponse) { } rpc DeltaAggregatedResources(stream DeltaDiscoveryRequest) returns (stream DeltaDiscoveryResponse) { } } ads.proto https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/discovery/v3/ads.proto
45. Aggregated Discovery Service dynamic_resources: cds_config: ads: {} lds_config: ads: {} ads_config: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: ads-server envoy.yaml
46. Aggregated Discovery Service dynamic_resources: cds_config: ads: {} lds_config: ads: {} ads_config: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: ads-server envoy.yaml
47. Aggregated Discovery Service Control Plane Listener / Route / Cluster / Endpoint Aggregated Discovery Service
48. Bootes uses ADS Bootes Route Listener Cluster kube-apiserver Aggregated Discovery Service
49. go-control-plane envoyproxy/go-control-plane
50. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
51. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
52. go-control-plane Package "go-control-plane/envoy" envoy/config/cluster/v3/ ├── circuit_breaker.pb.go ├── circuit_breaker.pb.validate.go ├── cluster.pb.go ├── cluster.pb.validate.go ├── filter.pb.go ├── filter.pb.validate.go ├── outlier_detection.pb.go └── outlier_detection.pb.validate.go Go files generated from .proto
53. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
54. go-control-plane Package "go-control-plane/pkg/server" Server Implementation import ( server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" // ... ) // ... server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis)
55. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
56. go-control-plane Package "go-control-plane/pkg/cache" { "envoy-1": { "clusters": [ "service-1": {...}, "service-2": {...} ], "listeners": [...], "routes": [...], "endpoints": [...], }, "envoy-2": {...}, ... } Configurations Cache for each Envoys
57. go-control-plane Package "go-control-plane/pkg/cache" snapshot := cache.NewSnapshot( "version-xxx", endpoints, clusters, routes, listeners, //... ) snapshotCache.SetSnapshot("envoy-1", snapshot)
58. go-control-plane Config Config Config Cache (pkg/cache) Aggregated Discovery Service gRPC Server (pkg/server) Update (cache.SetSnapshot)
59. go-control-plane Config Config Config Cache (pkg/cache) Aggregated Discovery Service gRPC Server (pkg/server) Update (cache.SetSnapshot)
60. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works
61. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works
62. Kubernetes Custom Controller Bootes Config Config Config kube-apiserver Aggregated Discovery Service ! kubectl apply -f cluster.yaml
63. Custom Resource Definition (CRD) ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget ... e.g. Istio ɾVirtualService ... e.g. SMI ɾTrafficSplit ... e.g. CrossPlane ɾCloudSQLInstance ... Native Resources Custom Resources + Extends Kubernetes using CRD
64. Bootes uses CRD ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget ... ɾListener ɾRoute ɾCluster ɾEndpoint Native Resources Custom Resources + Extends Kubernetes using CRD
65. Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
66. Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
67. Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
68. Bootes uses CRD cluster.yaml Any Envoy Cluster Config apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
69. Kubernetes Custom Controller Bootes Config Config Config kube-apiserver Aggregated Discovery Service ! kubectl apply -f cluster.yaml
70. Kubernetes Custom Controller We can use any Kubernetes Ecosystem Tools $
71. e.g. kustomize kustomize base-cluster.yaml dev-cluster.yaml prod-cluster.yaml
72. e.g. kpt ! awesome-cluster.yaml kpt pkg get ... !
73. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works
74. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works
75. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml Aggregated Discovery Service
76. How Bootes Works ! kubectl apply -f example-cluster.yaml
77. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
78. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
79. How Bootes Works example-cluster.yaml config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {} # ...
80. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml JSON
81. How Bootes Works proto := &cluster.Cluster{} (&protojson.UnmarshalOptions{}).Unmarshal(json, proto) JSON (Proto) google.golang.org/protobuf/encoding/protojson
82. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml JSON Aggregated Discovery Service
83. How Bootes Works Bootes Namespace: a Namespace: b app: foo app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)
84. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
85. How Bootes Works Bootes Namespace: a Namespace: b app: foo app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)
86. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
87. How Bootes Works Bootes Namespace: a Namespace: b app: foo app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)
88. How Bootes Works Bootes Namespace: a app: foo pod-xxx-333 app: foo pod-xxx-222 app: foo pod-xxx-111
89. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml JSON Aggregated Discovery Service
90. Agenda •What is Bootes? •Implementation •Use Case
91. Agenda •What is Bootes? •Implementation •Use Case
92. Envoy Reverse Proxy Use case Pod microservice B Service Pod Service Pod Service Pod microservice A !
93. Envoy Reverse Proxy Use case Pod microservice B Service Pod Service Pod Service Pod microservice A ! CDS CDS CDS LDS RDS RDS RDS
94. Custom HTTP Filter { "microservice-A": "PR-1", "microservice-B": "PR-2" } microservice-A: PR-1 microservice-B: PR-2 Request from APP Request to Upstream JWT Payload HTTP Header ɾɾɾ ɾɾɾ
95. Envoy Reverse Proxy Use case Pod microservice B Service Pod Service Pod Service Pod microservice A ! CDS CDS CDS LDS RDS RDS RDS
96. Agenda •What is Bootes? •Implementation •Use Case