Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bootes - Envoy Control Plane Kubernetes Controller -

Bootes - Envoy Control Plane Kubernetes Controller -

1bfc6e2ed04a895bb36f36b86828b689?s=128

Yuki Ito

June 20, 2020
Tweet

Transcript

  1. Bootes - Envoy Control Plane Kubernetes Controller - Envoy Meetup

    Tokyo #2 Yuki Ito
  2. Merpay Architect Team Backend Engineer Yuki Ito

  3. Agenda •What is Bootes? •Implementation •Use Case

  4. Agenda •What is Bootes? •Implementation •Use Case

  5. Bootes https://github.com/110y/bootes

  6. Bootes Envoy Control Plane implemented as a... Kubernetes Custom Controller

  7. Bootes: Overview Bootes Config Config Config

  8. Bootes: Overview Bootes Config Config Config kube-apiserver ! kubectl

  9. Motivation •Simple Control Plane •Full Envoy features •Kubernetes Custom Resource

    Movement
  10. e.g. Service Mesh Interface kind: TrafficSplit metadata: name: canary spec:

    service: website backends: - service: website-v1 weight: 90 - service: website-v2 weight: 10
  11. e.g. Crossplane apiVersion: database.gcp.crossplane.io/v1beta1 kind: CloudSQLInstance metadata: name: cloudsqlpostgresql spec:

    forProvider: databaseVersion: POSTGRES_9_6 region: us-central1 settings: tier: db-custom-1-3840 dataDiskType: PD_SSD dataDiskSizeGb: 10
  12. Agenda •What is Bootes? •Implementation •Use Case

  13. Agenda •What is Bootes? •Implementation •Use Case

  14. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works
  15. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works
  16. Envoy Control Plane Control Plane Config Config Config xDS API

  17. Envoy Configurations •Listener •Route •Cluster •Endpoint

  18. Envoy Configurations Listener Route Cluster Endpoint Endpoint Endpoint Endpoint Cluster

  19. Envoy Configurations 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13

    10.28.1.14 Service-2 Cluster Path: /service-1 Path: /service-2
  20. Static Configurations static_resources: listeners: - address: socket_address: protocol: TCP address:

    0.0.0.0 port_value: 5000 #... clusters: - name: service-1 connect_timeout: 1s type: STRICT_DNS lb_policy: ROUND_ROBIN #... envoy.yaml
  21. Static Configurations > envoy -c envoy.yaml

  22. Problem of Static Configurations Though simplistic, fairly complicated deployments can

    be created using static configurations and graceful hot restarts. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/dynamic_configuration
  23. Dynamic Configurations Control Plane Route Listener Cluster xDS API

  24. x Discovery Service API •Listener Discovery Service •Route Discovery Service

    •Cluster Discovery Service •Endpoint Discovery Service
  25. x Discovery Service API •Listener Discovery Service •Route Discovery Service

    •Cluster Discovery Service •Endpoint Discovery Service
  26. x Discovery Service API Control Plane Listener Listener Listener Listener

    Discovery Service
  27. x Discovery Service API Control Plane Cluster Cluster Cluster Cluster

    Discovery Service
  28. e.g. Cluster Discovery Service service ClusterDiscoveryService { rpc StreamClusters(stream discovery.v3.DiscoveryRequest)

    returns (stream discovery.v3.DiscoveryResponse) { } rpc DeltaClusters(stream discovery.v3.DeltaDiscoveryRequest) returns (stream discovery.v3.DeltaDiscoveryResponse) { } rpc FetchClusters(discovery.v3.DiscoveryRequest) returns (discovery.v3.DiscoveryResponse) { } } https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/cluster/v3/cds.proto cds.proto
  29. x Discovery Service API •Listener Discovery Service •Route Discovery Service

    •Cluster Discovery Service •Endpoint Discovery Service
  30. Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc:

    cluster_name: lds-server cds_config: #... envoy.yaml
  31. Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc:

    cluster_name: lds-server cds_config: #... envoy.yaml
  32. Dynamic Configurations envoy.yaml static_resources: clusters: - name: lds-server connect_timeout: 1s

    type: STRICT_DNS http2_protocol_options: {} load_assignment: cluster_name: lds-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ... port_value: ...
  33. Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: gRPC grpc_services: - envoy_grpc:

    cluster_name: lds-server cds_config: #... static_resources: clusters: - name: lds-server connect_timeout: 1s type: LOGICAL_DNS http2_protocol_options: {} load_assignment: cluster_name: lds-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ... port_value: ... envoy.yaml
  34. x Discovery Service API Control Plane Route Listener CDS LDS

    RDS CDS
  35. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12
  36. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster
  37. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster RDS CDS
  38. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12 ❌ RDS
  39. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster CDS
  40. Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11

    10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster CDS RDS
  41. Problem of xDS API To avoid downtime... 1. Cluster 2.

    Endpoint 3. Listener 4. Route
  42. Problem of xDS API Control Plane Route Listener CDS LDS

    RDS CDS
  43. Aggregated Discovery Service Control Plane Listener / Route / Cluster

    / Endpoint Aggregated Discovery Service
  44. Aggregated Discovery Service service AggregatedDiscoveryService { rpc StreamAggregatedResources(stream DiscoveryRequest) returns

    (stream DiscoveryResponse) { } rpc DeltaAggregatedResources(stream DeltaDiscoveryRequest) returns (stream DeltaDiscoveryResponse) { } } ads.proto https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/discovery/v3/ads.proto
  45. Aggregated Discovery Service dynamic_resources: cds_config: ads: {} lds_config: ads: {}

    ads_config: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: ads-server envoy.yaml
  46. Aggregated Discovery Service dynamic_resources: cds_config: ads: {} lds_config: ads: {}

    ads_config: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: ads-server envoy.yaml
  47. Aggregated Discovery Service Control Plane Listener / Route / Cluster

    / Endpoint Aggregated Discovery Service
  48. Bootes uses ADS Bootes Route Listener Cluster kube-apiserver Aggregated Discovery

    Service
  49. go-control-plane envoyproxy/go-control-plane

  50. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"

    ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
  51. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"

    ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
  52. go-control-plane Package "go-control-plane/envoy" envoy/config/cluster/v3/ ├── circuit_breaker.pb.go ├── circuit_breaker.pb.validate.go ├── cluster.pb.go

    ├── cluster.pb.validate.go ├── filter.pb.go ├── filter.pb.validate.go ├── outlier_detection.pb.go └── outlier_detection.pb.validate.go Go files generated from .proto
  53. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"

    ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
  54. go-control-plane Package "go-control-plane/pkg/server" Server Implementation import ( server ".../go-control-plane/pkg/server/v3" discovery

    ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" // ... ) // ... server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis)
  55. go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"

    ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
  56. go-control-plane Package "go-control-plane/pkg/cache" { "envoy-1": { "clusters": [ "service-1": {...},

    "service-2": {...} ], "listeners": [...], "routes": [...], "endpoints": [...], }, "envoy-2": {...}, ... } Configurations Cache for each Envoys
  57. go-control-plane Package "go-control-plane/pkg/cache" snapshot := cache.NewSnapshot( "version-xxx", endpoints, clusters, routes,

    listeners, //... ) snapshotCache.SetSnapshot("envoy-1", snapshot)
  58. go-control-plane Config Config Config Cache (pkg/cache) Aggregated Discovery Service gRPC

    Server (pkg/server) Update (cache.SetSnapshot)
  59. go-control-plane Config Config Config Cache (pkg/cache) Aggregated Discovery Service gRPC

    Server (pkg/server) Update (cache.SetSnapshot)
  60. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works
  61. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works
  62. Kubernetes Custom Controller Bootes Config Config Config kube-apiserver Aggregated Discovery

    Service ! kubectl apply -f cluster.yaml
  63. Custom Resource Definition (CRD) ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler

    ɾPodDisruptionBudget ... e.g. Istio ɾVirtualService ... e.g. SMI ɾTrafficSplit ... e.g. CrossPlane ɾCloudSQLInstance ... Native Resources Custom Resources + Extends Kubernetes using CRD
  64. Bootes uses CRD ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget

    ... ɾListener ɾRoute ɾCluster ɾEndpoint Native Resources Custom Resources + Extends Kubernetes using CRD
  65. Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name:

    example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
  66. Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name:

    example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
  67. Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name:

    example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
  68. Bootes uses CRD cluster.yaml Any Envoy Cluster Config apiVersion: bootes.io/v1

    kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
  69. Kubernetes Custom Controller Bootes Config Config Config kube-apiserver Aggregated Discovery

    Service ! kubectl apply -f cluster.yaml
  70. Kubernetes Custom Controller We can use any Kubernetes Ecosystem Tools

    $
  71. e.g. kustomize kustomize base-cluster.yaml dev-cluster.yaml prod-cluster.yaml

  72. e.g. kpt ! awesome-cluster.yaml kpt pkg get ... !

  73. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works
  74. Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes

    works
  75. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml

    Aggregated Discovery Service
  76. How Bootes Works ! kubectl apply -f example-cluster.yaml

  77. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster

    namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
  78. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster

    namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
  79. How Bootes Works example-cluster.yaml config: name: example-cluster connect_timeout: 1s type:

    EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {} # ...
  80. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml

    JSON
  81. How Bootes Works proto := &cluster.Cluster{} (&protojson.UnmarshalOptions{}).Unmarshal(json, proto) JSON (Proto)

    google.golang.org/protobuf/encoding/protojson
  82. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml

    JSON Aggregated Discovery Service
  83. How Bootes Works Bootes Namespace: a Namespace: b app: foo

    app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)
  84. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster

    namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
  85. How Bootes Works Bootes Namespace: a Namespace: b app: foo

    app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)
  86. How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster

    namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
  87. How Bootes Works Bootes Namespace: a Namespace: b app: foo

    app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)
  88. How Bootes Works Bootes Namespace: a app: foo pod-xxx-333 app:

    foo pod-xxx-222 app: foo pod-xxx-111
  89. How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml

    JSON Aggregated Discovery Service
  90. Agenda •What is Bootes? •Implementation •Use Case

  91. Agenda •What is Bootes? •Implementation •Use Case

  92. Envoy Reverse Proxy Use case Pod microservice B Service Pod

    Service Pod Service Pod microservice A !
  93. Envoy Reverse Proxy Use case Pod microservice B Service Pod

    Service Pod Service Pod microservice A ! CDS CDS CDS LDS RDS RDS RDS
  94. Custom HTTP Filter { "microservice-A": "PR-1", "microservice-B": "PR-2" } microservice-A:

    PR-1 microservice-B: PR-2 Request from APP Request to Upstream JWT Payload HTTP Header ɾɾɾ ɾɾɾ
  95. Envoy Reverse Proxy Use case Pod microservice B Service Pod

    Service Pod Service Pod microservice A ! CDS CDS CDS LDS RDS RDS RDS
  96. Agenda •What is Bootes? •Implementation •Use Case