Bootes - Envoy Control Plane Kubernetes Controller -

Transcript

1. Bootes - Envoy Control Plane Kubernetes Controller - Envoy Meetup

   Tokyo #2 Yuki Ito

2. Merpay Architect Team Backend Engineer Yuki Ito

3. Agenda •What is Bootes? •Implementation •Use Case

4. Agenda •What is Bootes? •Implementation •Use Case

5. Bootes https://github.com/110y/bootes

6. Bootes Envoy Control Plane implemented as a... Kubernetes Custom Controller

7. Bootes: Overview Bootes Config Config Config

8. Bootes: Overview Bootes Config Config Config kube-apiserver ! kubectl

9. Motivation •Simple Control Plane •Full Envoy features •Kubernetes Custom Resource

   Movement

10. e.g. Service Mesh Interface kind: TrafficSplit metadata: name: canary spec:

    service: website backends: - service: website-v1 weight: 90 - service: website-v2 weight: 10

11. e.g. Crossplane apiVersion: database.gcp.crossplane.io/v1beta1 kind: CloudSQLInstance metadata: name: cloudsqlpostgresql spec:

    forProvider: databaseVersion: POSTGRES_9_6 region: us-central1 settings: tier: db-custom-1-3840 dataDiskType: PD_SSD dataDiskSizeGb: 10

12. Agenda •What is Bootes? •Implementation •Use Case

13. Agenda •What is Bootes? •Implementation •Use Case

14. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works

15. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works

16. Envoy Control Plane Control Plane Config Config Config xDS API

17. Envoy Configurations •Listener •Route •Cluster •Endpoint

18. Envoy Configurations Listener Route Cluster Endpoint Endpoint Endpoint Endpoint Cluster

19. Envoy Configurations 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13

    10.28.1.14 Service-2 Cluster Path: /service-1 Path: /service-2

20. Static Configurations static_resources: listeners: - address: socket_address: protocol: TCP address:

    0.0.0.0 port_value: 5000 #... clusters: - name: service-1 connect_timeout: 1s type: STRICT_DNS lb_policy: ROUND_ROBIN #... envoy.yaml

21. Static Configurations > envoy -c envoy.yaml

22. Problem of Static Configurations Though simplistic, fairly complicated deployments can

    be created using static configurations and graceful hot restarts. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/dynamic_configuration

23. Dynamic Configurations Control Plane Route Listener Cluster xDS API

24. x Discovery Service API •Listener Discovery Service •Route Discovery Service

    •Cluster Discovery Service •Endpoint Discovery Service

25. x Discovery Service API •Listener Discovery Service •Route Discovery Service

    •Cluster Discovery Service •Endpoint Discovery Service

26. x Discovery Service API Control Plane Listener Listener Listener Listener

    Discovery Service

27. x Discovery Service API Control Plane Cluster Cluster Cluster Cluster

    Discovery Service

28. e.g. Cluster Discovery Service service ClusterDiscoveryService { rpc StreamClusters(stream discovery.v3.DiscoveryRequest)

    returns (stream discovery.v3.DiscoveryResponse) { } rpc DeltaClusters(stream discovery.v3.DeltaDiscoveryRequest) returns (stream discovery.v3.DeltaDiscoveryResponse) { } rpc FetchClusters(discovery.v3.DiscoveryRequest) returns (discovery.v3.DiscoveryResponse) { } } https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/cluster/v3/cds.proto cds.proto

29. x Discovery Service API •Listener Discovery Service •Route Discovery Service

    •Cluster Discovery Service •Endpoint Discovery Service

30. Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc:

    cluster_name: lds-server cds_config: #... envoy.yaml

31. Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc:

    cluster_name: lds-server cds_config: #... envoy.yaml

32. Dynamic Configurations envoy.yaml static_resources: clusters: - name: lds-server connect_timeout: 1s

    type: STRICT_DNS http2_protocol_options: {} load_assignment: cluster_name: lds-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ... port_value: ...

33. Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: gRPC grpc_services: - envoy_grpc:

    cluster_name: lds-server cds_config: #... static_resources: clusters: - name: lds-server connect_timeout: 1s type: LOGICAL_DNS http2_protocol_options: {} load_assignment: cluster_name: lds-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ... port_value: ... envoy.yaml

34. x Discovery Service API Control Plane Route Listener CDS LDS

    RDS CDS

35. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12

36. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster

37. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster RDS CDS

38. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12 ❌ RDS

39. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster CDS

40. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster CDS RDS

41. Problem of xDS API To avoid downtime... 1. Cluster 2.

    Endpoint 3. Listener 4. Route

42. Problem of xDS API Control Plane Route Listener CDS LDS

    RDS CDS

43. Aggregated Discovery Service Control Plane Listener / Route / Cluster

    / Endpoint Aggregated Discovery Service

44. Aggregated Discovery Service service AggregatedDiscoveryService { rpc StreamAggregatedResources(stream DiscoveryRequest) returns

    (stream DiscoveryResponse) { } rpc DeltaAggregatedResources(stream DeltaDiscoveryRequest) returns (stream DeltaDiscoveryResponse) { } } ads.proto https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/discovery/v3/ads.proto

45. Aggregated Discovery Service dynamic_resources: cds_config: ads: {} lds_config: ads: {}

    ads_config: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: ads-server envoy.yaml

46. Aggregated Discovery Service dynamic_resources: cds_config: ads: {} lds_config: ads: {}

    ads_config: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: ads-server envoy.yaml

47. Aggregated Discovery Service Control Plane Listener / Route / Cluster

    / Endpoint Aggregated Discovery Service

48. Bootes uses ADS Bootes Route Listener Cluster kube-apiserver Aggregated Discovery

    Service

49. go-control-plane envoyproxy/go-control-plane

50. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"

    ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation

51. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"

    ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation

52. go-control-plane Package "go-control-plane/envoy" envoy/config/cluster/v3/ ├── circuit_breaker.pb.go ├── circuit_breaker.pb.validate.go ├── cluster.pb.go

    ├── cluster.pb.validate.go ├── filter.pb.go ├── filter.pb.validate.go ├── outlier_detection.pb.go └── outlier_detection.pb.validate.go Go files generated from .proto

53. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"

    ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation

54. go-control-plane Package "go-control-plane/pkg/server" Server Implementation import ( server ".../go-control-plane/pkg/server/v3" discovery

    ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" // ... ) // ... server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis)

55. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"

    ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation

56. go-control-plane Package "go-control-plane/pkg/cache" { "envoy-1": { "clusters": [ "service-1": {...},

    "service-2": {...} ], "listeners": [...], "routes": [...], "endpoints": [...], }, "envoy-2": {...}, ... } Configurations Cache for each Envoys

57. go-control-plane Package "go-control-plane/pkg/cache" snapshot := cache.NewSnapshot( "version-xxx", endpoints, clusters, routes,

    listeners, //... ) snapshotCache.SetSnapshot("envoy-1", snapshot)

58. go-control-plane Config Config Config Cache (pkg/cache) Aggregated Discovery Service gRPC

    Server (pkg/server) Update (cache.SetSnapshot)

59. go-control-plane Config Config Config Cache (pkg/cache) Aggregated Discovery Service gRPC

    Server (pkg/server) Update (cache.SetSnapshot)

60. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works

61. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works

62. Kubernetes Custom Controller Bootes Config Config Config kube-apiserver Aggregated Discovery

    Service ! kubectl apply -f cluster.yaml

63. Custom Resource Definition (CRD) ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler

    ɾPodDisruptionBudget ... e.g. Istio ɾVirtualService ... e.g. SMI ɾTrafficSplit ... e.g. CrossPlane ɾCloudSQLInstance ... Native Resources Custom Resources + Extends Kubernetes using CRD

64. Bootes uses CRD ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget

    ... ɾListener ɾRoute ɾCluster ɾEndpoint Native Resources Custom Resources + Extends Kubernetes using CRD

65. Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name:

    example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}

66. Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name:

    example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}

67. Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name:

    example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}

68. Bootes uses CRD cluster.yaml Any Envoy Cluster Config apiVersion: bootes.io/v1

    kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}

69. Kubernetes Custom Controller Bootes Config Config Config kube-apiserver Aggregated Discovery

    Service ! kubectl apply -f cluster.yaml

70. Kubernetes Custom Controller We can use any Kubernetes Ecosystem Tools

    $

71. e.g. kustomize kustomize base-cluster.yaml dev-cluster.yaml prod-cluster.yaml

72. e.g. kpt ! awesome-cluster.yaml kpt pkg get ... !

73. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works

74. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works

75. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml

    Aggregated Discovery Service

76. How Bootes Works ! kubectl apply -f example-cluster.yaml

77. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster

    namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml

78. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster

    namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml

79. How Bootes Works example-cluster.yaml config: name: example-cluster connect_timeout: 1s type:

    EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {} # ...

80. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml

    JSON

81. How Bootes Works proto := &cluster.Cluster{} (&protojson.UnmarshalOptions{}).Unmarshal(json, proto) JSON (Proto)

    google.golang.org/protobuf/encoding/protojson

82. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml

    JSON Aggregated Discovery Service

83. How Bootes Works Bootes Namespace: a Namespace: b app: foo

    app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)

84. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster

    namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml

85. How Bootes Works Bootes Namespace: a Namespace: b app: foo

    app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)

86. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster

    namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml

87. How Bootes Works Bootes Namespace: a Namespace: b app: foo

    app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)

88. How Bootes Works Bootes Namespace: a app: foo pod-xxx-333 app:

    foo pod-xxx-222 app: foo pod-xxx-111

89. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml

    JSON Aggregated Discovery Service

90. Agenda •What is Bootes? •Implementation •Use Case

91. Agenda •What is Bootes? •Implementation •Use Case

92. Envoy Reverse Proxy Use case Pod microservice B Service Pod

    Service Pod Service Pod microservice A !

93. Envoy Reverse Proxy Use case Pod microservice B Service Pod

    Service Pod Service Pod microservice A ! CDS CDS CDS LDS RDS RDS RDS

94. Custom HTTP Filter { "microservice-A": "PR-1", "microservice-B": "PR-2" } microservice-A:

    PR-1 microservice-B: PR-2 Request from APP Request to Upstream JWT Payload HTTP Header ɾɾɾ ɾɾɾ

95. Envoy Reverse Proxy Use case Pod microservice B Service Pod

    Service Pod Service Pod microservice A ! CDS CDS CDS LDS RDS RDS RDS

96. Agenda •What is Bootes? •Implementation •Use Case