Slide 1

Slide 1 text

Bootes - Envoy Control Plane Kubernetes Controller - Envoy Meetup Tokyo #2 Yuki Ito

Slide 2

Slide 2 text

Merpay Architect Team Backend Engineer Yuki Ito

Slide 3

Slide 3 text

Agenda •What is Bootes? •Implementation •Use Case

Slide 4

Slide 4 text

Agenda •What is Bootes? •Implementation •Use Case

Slide 5

Slide 5 text

Bootes https://github.com/110y/bootes

Slide 6

Slide 6 text

Bootes Envoy Control Plane implemented as a... Kubernetes Custom Controller

Slide 7

Slide 7 text

Bootes: Overview Bootes Config Config Config

Slide 8

Slide 8 text

Bootes: Overview Bootes Config Config Config kube-apiserver ! kubectl

Slide 9

Slide 9 text

Motivation •Simple Control Plane •Full Envoy features •Kubernetes Custom Resource Movement

Slide 10

Slide 10 text

e.g. Service Mesh Interface kind: TrafficSplit metadata: name: canary spec: service: website backends: - service: website-v1 weight: 90 - service: website-v2 weight: 10

Slide 11

Slide 11 text

e.g. Crossplane apiVersion: database.gcp.crossplane.io/v1beta1 kind: CloudSQLInstance metadata: name: cloudsqlpostgresql spec: forProvider: databaseVersion: POSTGRES_9_6 region: us-central1 settings: tier: db-custom-1-3840 dataDiskType: PD_SSD dataDiskSizeGb: 10

Slide 12

Slide 12 text

Agenda •What is Bootes? •Implementation •Use Case

Slide 13

Slide 13 text

Agenda •What is Bootes? •Implementation •Use Case

Slide 14

Slide 14 text

Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works

Slide 15

Slide 15 text

Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works

Slide 16

Slide 16 text

Envoy Control Plane Control Plane Config Config Config xDS API

Slide 17

Slide 17 text

Envoy Configurations •Listener •Route •Cluster •Endpoint

Slide 18

Slide 18 text

Envoy Configurations Listener Route Cluster Endpoint Endpoint Endpoint Endpoint Cluster

Slide 19

Slide 19 text

Envoy Configurations 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster Path: /service-1 Path: /service-2

Slide 20

Slide 20 text

Static Configurations static_resources: listeners: - address: socket_address: protocol: TCP address: 0.0.0.0 port_value: 5000 #... clusters: - name: service-1 connect_timeout: 1s type: STRICT_DNS lb_policy: ROUND_ROBIN #... envoy.yaml

Slide 21

Slide 21 text

Static Configurations > envoy -c envoy.yaml

Slide 22

Slide 22 text

Problem of Static Configurations Though simplistic, fairly complicated deployments can be created using static configurations and graceful hot restarts. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/dynamic_configuration

Slide 23

Slide 23 text

Dynamic Configurations Control Plane Route Listener Cluster xDS API

Slide 24

Slide 24 text

x Discovery Service API •Listener Discovery Service •Route Discovery Service •Cluster Discovery Service •Endpoint Discovery Service

Slide 25

Slide 25 text

x Discovery Service API •Listener Discovery Service •Route Discovery Service •Cluster Discovery Service •Endpoint Discovery Service

Slide 26

Slide 26 text

x Discovery Service API Control Plane Listener Listener Listener Listener Discovery Service

Slide 27

Slide 27 text

x Discovery Service API Control Plane Cluster Cluster Cluster Cluster Discovery Service

Slide 28

Slide 28 text

e.g. Cluster Discovery Service service ClusterDiscoveryService { rpc StreamClusters(stream discovery.v3.DiscoveryRequest) returns (stream discovery.v3.DiscoveryResponse) { } rpc DeltaClusters(stream discovery.v3.DeltaDiscoveryRequest) returns (stream discovery.v3.DeltaDiscoveryResponse) { } rpc FetchClusters(discovery.v3.DiscoveryRequest) returns (discovery.v3.DiscoveryResponse) { } } https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/cluster/v3/cds.proto cds.proto

Slide 29

Slide 29 text

x Discovery Service API •Listener Discovery Service •Route Discovery Service •Cluster Discovery Service •Endpoint Discovery Service

Slide 30

Slide 30 text

Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: lds-server cds_config: #... envoy.yaml

Slide 31

Slide 31 text

Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: lds-server cds_config: #... envoy.yaml

Slide 32

Slide 32 text

Dynamic Configurations envoy.yaml static_resources: clusters: - name: lds-server connect_timeout: 1s type: STRICT_DNS http2_protocol_options: {} load_assignment: cluster_name: lds-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ... port_value: ...

Slide 33

Slide 33 text

Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: gRPC grpc_services: - envoy_grpc: cluster_name: lds-server cds_config: #... static_resources: clusters: - name: lds-server connect_timeout: 1s type: LOGICAL_DNS http2_protocol_options: {} load_assignment: cluster_name: lds-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ... port_value: ... envoy.yaml

Slide 34

Slide 34 text

x Discovery Service API Control Plane Route Listener CDS LDS RDS CDS

Slide 35

Slide 35 text

Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12

Slide 36

Slide 36 text

Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster

Slide 37

Slide 37 text

Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster RDS CDS

Slide 38

Slide 38 text

Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 ❌ RDS

Slide 39

Slide 39 text

Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster CDS

Slide 40

Slide 40 text

Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster CDS RDS

Slide 41

Slide 41 text

Problem of xDS API To avoid downtime... 1. Cluster 2. Endpoint 3. Listener 4. Route

Slide 42

Slide 42 text

Problem of xDS API Control Plane Route Listener CDS LDS RDS CDS

Slide 43

Slide 43 text

Aggregated Discovery Service Control Plane Listener / Route / Cluster / Endpoint Aggregated Discovery Service

Slide 44

Slide 44 text

Aggregated Discovery Service service AggregatedDiscoveryService { rpc StreamAggregatedResources(stream DiscoveryRequest) returns (stream DiscoveryResponse) { } rpc DeltaAggregatedResources(stream DeltaDiscoveryRequest) returns (stream DeltaDiscoveryResponse) { } } ads.proto https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/discovery/v3/ads.proto

Slide 45

Slide 45 text

Aggregated Discovery Service dynamic_resources: cds_config: ads: {} lds_config: ads: {} ads_config: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: ads-server envoy.yaml

Slide 46

Slide 46 text

Aggregated Discovery Service dynamic_resources: cds_config: ads: {} lds_config: ads: {} ads_config: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: ads-server envoy.yaml

Slide 47

Slide 47 text

Aggregated Discovery Service Control Plane Listener / Route / Cluster / Endpoint Aggregated Discovery Service

Slide 48

Slide 48 text

Bootes uses ADS Bootes Route Listener Cluster kube-apiserver Aggregated Discovery Service

Slide 49

Slide 49 text

go-control-plane envoyproxy/go-control-plane

Slide 50

Slide 50 text

go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation

Slide 51

Slide 51 text

go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation

Slide 52

Slide 52 text

go-control-plane Package "go-control-plane/envoy" envoy/config/cluster/v3/ ├── circuit_breaker.pb.go ├── circuit_breaker.pb.validate.go ├── cluster.pb.go ├── cluster.pb.validate.go ├── filter.pb.go ├── filter.pb.validate.go ├── outlier_detection.pb.go └── outlier_detection.pb.validate.go Go files generated from .proto

Slide 53

Slide 53 text

go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation

Slide 54

Slide 54 text

go-control-plane Package "go-control-plane/pkg/server" Server Implementation import ( server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" // ... ) // ... server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis)

Slide 55

Slide 55 text

go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" ) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation

Slide 56

Slide 56 text

go-control-plane Package "go-control-plane/pkg/cache" { "envoy-1": { "clusters": [ "service-1": {...}, "service-2": {...} ], "listeners": [...], "routes": [...], "endpoints": [...], }, "envoy-2": {...}, ... } Configurations Cache for each Envoys

Slide 57

Slide 57 text

go-control-plane Package "go-control-plane/pkg/cache" snapshot := cache.NewSnapshot( "version-xxx", endpoints, clusters, routes, listeners, //... ) snapshotCache.SetSnapshot("envoy-1", snapshot)

Slide 58

Slide 58 text

go-control-plane Config Config Config Cache (pkg/cache) Aggregated Discovery Service gRPC Server (pkg/server) Update (cache.SetSnapshot)

Slide 59

Slide 59 text

go-control-plane Config Config Config Cache (pkg/cache) Aggregated Discovery Service gRPC Server (pkg/server) Update (cache.SetSnapshot)

Slide 60

Slide 60 text

Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works

Slide 61

Slide 61 text

Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works

Slide 62

Slide 62 text

Kubernetes Custom Controller Bootes Config Config Config kube-apiserver Aggregated Discovery Service ! kubectl apply -f cluster.yaml

Slide 63

Slide 63 text

Custom Resource Definition (CRD) ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget ... e.g. Istio ɾVirtualService ... e.g. SMI ɾTrafficSplit ... e.g. CrossPlane ɾCloudSQLInstance ... Native Resources Custom Resources + Extends Kubernetes using CRD

Slide 64

Slide 64 text

Bootes uses CRD ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget ... ɾListener ɾRoute ɾCluster ɾEndpoint Native Resources Custom Resources + Extends Kubernetes using CRD

Slide 65

Slide 65 text

Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}

Slide 66

Slide 66 text

Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}

Slide 67

Slide 67 text

Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}

Slide 68

Slide 68 text

Bootes uses CRD cluster.yaml Any Envoy Cluster Config apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}

Slide 69

Slide 69 text

Kubernetes Custom Controller Bootes Config Config Config kube-apiserver Aggregated Discovery Service ! kubectl apply -f cluster.yaml

Slide 70

Slide 70 text

Kubernetes Custom Controller We can use any Kubernetes Ecosystem Tools $

Slide 71

Slide 71 text

e.g. kustomize kustomize base-cluster.yaml dev-cluster.yaml prod-cluster.yaml

Slide 72

Slide 72 text

e.g. kpt ! awesome-cluster.yaml kpt pkg get ... !

Slide 73

Slide 73 text

Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works

Slide 74

Slide 74 text

Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes works

Slide 75

Slide 75 text

How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml Aggregated Discovery Service

Slide 76

Slide 76 text

How Bootes Works ! kubectl apply -f example-cluster.yaml

Slide 77

Slide 77 text

How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml

Slide 78

Slide 78 text

How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml

Slide 79

Slide 79 text

How Bootes Works example-cluster.yaml config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {} # ...

Slide 80

Slide 80 text

How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml JSON

Slide 81

Slide 81 text

How Bootes Works proto := &cluster.Cluster{} (&protojson.UnmarshalOptions{}).Unmarshal(json, proto) JSON (Proto) google.golang.org/protobuf/encoding/protojson

Slide 82

Slide 82 text

How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml JSON Aggregated Discovery Service

Slide 83

Slide 83 text

How Bootes Works Bootes Namespace: a Namespace: b app: foo app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)

Slide 84

Slide 84 text

How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml

Slide 85

Slide 85 text

How Bootes Works Bootes Namespace: a Namespace: b app: foo app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)

Slide 86

Slide 86 text

How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml

Slide 87

Slide 87 text

How Bootes Works Bootes Namespace: a Namespace: b app: foo app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)

Slide 88

Slide 88 text

How Bootes Works Bootes Namespace: a app: foo pod-xxx-333 app: foo pod-xxx-222 app: foo pod-xxx-111

Slide 89

Slide 89 text

How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml JSON Aggregated Discovery Service

Slide 90

Slide 90 text

Agenda •What is Bootes? •Implementation •Use Case

Slide 91

Slide 91 text

Agenda •What is Bootes? •Implementation •Use Case

Slide 92

Slide 92 text

Envoy Reverse Proxy Use case Pod microservice B Service Pod Service Pod Service Pod microservice A !

Slide 93

Slide 93 text

Envoy Reverse Proxy Use case Pod microservice B Service Pod Service Pod Service Pod microservice A ! CDS CDS CDS LDS RDS RDS RDS

Slide 94

Slide 94 text

Custom HTTP Filter { "microservice-A": "PR-1", "microservice-B": "PR-2" } microservice-A: PR-1 microservice-B: PR-2 Request from APP Request to Upstream JWT Payload HTTP Header ɾɾɾ ɾɾɾ

Slide 95

Slide 95 text

Envoy Reverse Proxy Use case Pod microservice B Service Pod Service Pod Service Pod microservice A ! CDS CDS CDS LDS RDS RDS RDS

Slide 96

Slide 96 text

Agenda •What is Bootes? •Implementation •Use Case