Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Bootes - Envoy Control Plane Kubernetes Controller -
Yuki Ito
June 20, 2020
Technology
0
910
Bootes - Envoy Control Plane Kubernetes Controller -
Yuki Ito
June 20, 2020
Tweet
Share
More Decks by Yuki Ito
See All by Yuki Ito
Microservices on Cloud Run @ KAUCHE
110y
0
68
KAUCHE Loves Go
110y
0
190
Evolution of Architecture @ Kauche
110y
3
240
Envoy as an API Gateway
110y
0
110
Microservices on Cloud Run + VPC Network
110y
0
230
How We Use Cloud Run and its Friends
110y
0
210
Custom Kubernetes Controllers at Mercari
110y
1
450
What Is the Go Workspace Mode
110y
4
1.1k
What Are We Doing as Merpay Architect
110y
0
1.3k
Other Decks in Technology
See All in Technology
Oracle Cloud Infrastructure:2023年1月度サービス・アップデート
oracle4engineer
PRO
0
150
スクラム導入して変わったチーム、組織のありかた
yumechi
0
190
PHPのimmutable arrayとは
hnw
1
160
私見「UNIXの考え方」/20230124-kameda-unix-phylosophy
opelab
0
160
Kaggleシミュレーションコンペの動向
nagiss
0
270
USB PD で迎える AC アダプター大統一時代
puhitaku
2
1.9k
AWS re:Invent 2022で発表された新機能を試してみた ~Cloud OperationとSecurity~ / New Cloud Operation and Security Features Announced at AWS reInvent 2022
yuj1osm
1
210
💰年度末予算消化祭💰 Large Memory Instance で 画像分類してみた
__allllllllez__
0
100
目指せCoverage100%! AutoScale環境におけるSavings Plans購入戦略 / JAWS-UG_SRE_Coverage
taishin
0
500
データベースの発表には RDBMS 以外もありますよ
maroon1st
0
240
ML PM, DS PMってどんな仕事をしているの?
line_developers
PRO
1
240
立ち止まっても、寄り道しても / even if I stop, even if I take a detour
katoaz
0
560
Featured
See All Featured
Docker and Python
trallard
30
1.9k
For a Future-Friendly Web
brad_frost
166
7.8k
How to name files
jennybc
47
73k
GitHub's CSS Performance
jonrohan
1020
430k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
182
15k
StorybookのUI Testing Handbookを読んだ
zakiyama
8
3.2k
Art Directing for the Web. Five minutes with CSS Template Areas
malarkey
196
9.8k
How GitHub (no longer) Works
holman
298
140k
Clear Off the Table
cherdarchuk
79
290k
Design by the Numbers
sachag
271
18k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
6
840
Building Adaptive Systems
keathley
27
1.3k
Transcript
Bootes - Envoy Control Plane Kubernetes Controller - Envoy Meetup
Tokyo #2 Yuki Ito
Merpay Architect Team Backend Engineer Yuki Ito
Agenda •What is Bootes? •Implementation •Use Case
Agenda •What is Bootes? •Implementation •Use Case
Bootes https://github.com/110y/bootes
Bootes Envoy Control Plane implemented as a... Kubernetes Custom Controller
Bootes: Overview Bootes Config Config Config
Bootes: Overview Bootes Config Config Config kube-apiserver ! kubectl
Motivation •Simple Control Plane •Full Envoy features •Kubernetes Custom Resource
Movement
e.g. Service Mesh Interface kind: TrafficSplit metadata: name: canary spec:
service: website backends: - service: website-v1 weight: 90 - service: website-v2 weight: 10
e.g. Crossplane apiVersion: database.gcp.crossplane.io/v1beta1 kind: CloudSQLInstance metadata: name: cloudsqlpostgresql spec:
forProvider: databaseVersion: POSTGRES_9_6 region: us-central1 settings: tier: db-custom-1-3840 dataDiskType: PD_SSD dataDiskSizeGb: 10
Agenda •What is Bootes? •Implementation •Use Case
Agenda •What is Bootes? •Implementation •Use Case
Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes
works
Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes
works
Envoy Control Plane Control Plane Config Config Config xDS API
Envoy Configurations •Listener •Route •Cluster •Endpoint
Envoy Configurations Listener Route Cluster Endpoint Endpoint Endpoint Endpoint Cluster
Envoy Configurations 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11 10.28.1.12 10.28.1.13
10.28.1.14 Service-2 Cluster Path: /service-1 Path: /service-2
Static Configurations static_resources: listeners: - address: socket_address: protocol: TCP address:
0.0.0.0 port_value: 5000 #... clusters: - name: service-1 connect_timeout: 1s type: STRICT_DNS lb_policy: ROUND_ROBIN #... envoy.yaml
Static Configurations > envoy -c envoy.yaml
Problem of Static Configurations Though simplistic, fairly complicated deployments can
be created using static configurations and graceful hot restarts. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/dynamic_configuration
Dynamic Configurations Control Plane Route Listener Cluster xDS API
x Discovery Service API •Listener Discovery Service •Route Discovery Service
•Cluster Discovery Service •Endpoint Discovery Service
x Discovery Service API •Listener Discovery Service •Route Discovery Service
•Cluster Discovery Service •Endpoint Discovery Service
x Discovery Service API Control Plane Listener Listener Listener Listener
Discovery Service
x Discovery Service API Control Plane Cluster Cluster Cluster Cluster
Discovery Service
e.g. Cluster Discovery Service service ClusterDiscoveryService { rpc StreamClusters(stream discovery.v3.DiscoveryRequest)
returns (stream discovery.v3.DiscoveryResponse) { } rpc DeltaClusters(stream discovery.v3.DeltaDiscoveryRequest) returns (stream discovery.v3.DeltaDiscoveryResponse) { } rpc FetchClusters(discovery.v3.DiscoveryRequest) returns (discovery.v3.DiscoveryResponse) { } } https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/cluster/v3/cds.proto cds.proto
x Discovery Service API •Listener Discovery Service •Route Discovery Service
•Cluster Discovery Service •Endpoint Discovery Service
Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc:
cluster_name: lds-server cds_config: #... envoy.yaml
Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc:
cluster_name: lds-server cds_config: #... envoy.yaml
Dynamic Configurations envoy.yaml static_resources: clusters: - name: lds-server connect_timeout: 1s
type: STRICT_DNS http2_protocol_options: {} load_assignment: cluster_name: lds-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ... port_value: ...
Dynamic Configurations dynamic_resources: lds_config: api_config_source: api_type: gRPC grpc_services: - envoy_grpc:
cluster_name: lds-server cds_config: #... static_resources: clusters: - name: lds-server connect_timeout: 1s type: LOGICAL_DNS http2_protocol_options: {} load_assignment: cluster_name: lds-server endpoints: - lb_endpoints: - endpoint: address: socket_address: address: ... port_value: ... envoy.yaml
x Discovery Service API Control Plane Route Listener CDS LDS
RDS CDS
Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11
10.28.1.12
Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11
10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster
Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11
10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster RDS CDS
Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11
10.28.1.12 ❌ RDS
Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11
10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster CDS
Problem of xDS API 0.0.0.0:5000 Listener Route Service-1 Cluster 10.28.1.11
10.28.1.12 10.28.1.13 10.28.1.14 Service-2 Cluster CDS RDS
Problem of xDS API To avoid downtime... 1. Cluster 2.
Endpoint 3. Listener 4. Route
Problem of xDS API Control Plane Route Listener CDS LDS
RDS CDS
Aggregated Discovery Service Control Plane Listener / Route / Cluster
/ Endpoint Aggregated Discovery Service
Aggregated Discovery Service service AggregatedDiscoveryService { rpc StreamAggregatedResources(stream DiscoveryRequest) returns
(stream DiscoveryResponse) { } rpc DeltaAggregatedResources(stream DeltaDiscoveryRequest) returns (stream DeltaDiscoveryResponse) { } } ads.proto https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/discovery/v3/ads.proto
Aggregated Discovery Service dynamic_resources: cds_config: ads: {} lds_config: ads: {}
ads_config: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: ads-server envoy.yaml
Aggregated Discovery Service dynamic_resources: cds_config: ads: {} lds_config: ads: {}
ads_config: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: ads-server envoy.yaml
Aggregated Discovery Service Control Plane Listener / Route / Cluster
/ Endpoint Aggregated Discovery Service
Bootes uses ADS Bootes Route Listener Cluster kube-apiserver Aggregated Discovery
Service
go-control-plane envoyproxy/go-control-plane
go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"
) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"
) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
go-control-plane Package "go-control-plane/envoy" envoy/config/cluster/v3/ ├── circuit_breaker.pb.go ├── circuit_breaker.pb.validate.go ├── cluster.pb.go
├── cluster.pb.validate.go ├── filter.pb.go ├── filter.pb.validate.go ├── outlier_detection.pb.go └── outlier_detection.pb.validate.go Go files generated from .proto
go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"
) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
go-control-plane Package "go-control-plane/pkg/server" Server Implementation import ( server ".../go-control-plane/pkg/server/v3" discovery
".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc" // ... ) // ... server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis)
go-control-plane import ( cache ".../go-control-plane/pkg/cache/v3" server ".../go-control-plane/pkg/server/v3" discovery ".../go-control-plane/envoy/service/discovery/v3" "google.golang.org/grpc"
) // ... snapshotCache := cache.NewSnapshotCache(...) server := server.NewServer(ctx, snapshotCache, ...) grpcServer := grpc.NewServer() lis, _ := net.Listen("tcp", ":8081") discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server) grpcServer.Serve(lis) Minimum Implementation
go-control-plane Package "go-control-plane/pkg/cache" { "envoy-1": { "clusters": [ "service-1": {...},
"service-2": {...} ], "listeners": [...], "routes": [...], "endpoints": [...], }, "envoy-2": {...}, ... } Configurations Cache for each Envoys
go-control-plane Package "go-control-plane/pkg/cache" snapshot := cache.NewSnapshot( "version-xxx", endpoints, clusters, routes,
listeners, //... ) snapshotCache.SetSnapshot("envoy-1", snapshot)
go-control-plane Config Config Config Cache (pkg/cache) Aggregated Discovery Service gRPC
Server (pkg/server) Update (cache.SetSnapshot)
go-control-plane Config Config Config Cache (pkg/cache) Aggregated Discovery Service gRPC
Server (pkg/server) Update (cache.SetSnapshot)
Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes
works
Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes
works
Kubernetes Custom Controller Bootes Config Config Config kube-apiserver Aggregated Discovery
Service ! kubectl apply -f cluster.yaml
Custom Resource Definition (CRD) ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler
ɾPodDisruptionBudget ... e.g. Istio ɾVirtualService ... e.g. SMI ɾTrafficSplit ... e.g. CrossPlane ɾCloudSQLInstance ... Native Resources Custom Resources + Extends Kubernetes using CRD
Bootes uses CRD ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget
... ɾListener ɾRoute ɾCluster ɾEndpoint Native Resources Custom Resources + Extends Kubernetes using CRD
Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name:
example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name:
example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
Bootes uses CRD cluster.yaml apiVersion: bootes.io/v1 kind: Cluster metadata: name:
example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
Bootes uses CRD cluster.yaml Any Envoy Cluster Config apiVersion: bootes.io/v1
kind: Cluster metadata: name: example-cluster namespace: foo spec: config: name: example-cluster connect_timeout: 1s type: EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {}
Kubernetes Custom Controller Bootes Config Config Config kube-apiserver Aggregated Discovery
Service ! kubectl apply -f cluster.yaml
Kubernetes Custom Controller We can use any Kubernetes Ecosystem Tools
$
e.g. kustomize kustomize base-cluster.yaml dev-cluster.yaml prod-cluster.yaml
e.g. kpt ! awesome-cluster.yaml kpt pkg get ... !
Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes
works
Implementation Envoy Control Plane Kubernetes Custom Controller " How Bootes
works
How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml
Aggregated Discovery Service
How Bootes Works ! kubectl apply -f example-cluster.yaml
How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster
namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster
namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
How Bootes Works example-cluster.yaml config: name: example-cluster connect_timeout: 1s type:
EDS lb_policy: ROUND_ROBIN http2_protocol_options: {} eds_cluster_config: eds_config: ads: {} # ...
How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml
JSON
How Bootes Works proto := &cluster.Cluster{} (&protojson.UnmarshalOptions{}).Unmarshal(json, proto) JSON (Proto)
google.golang.org/protobuf/encoding/protojson
How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml
JSON Aggregated Discovery Service
How Bootes Works Bootes Namespace: a Namespace: b app: foo
app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)
How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster
namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
How Bootes Works Bootes Namespace: a Namespace: b app: foo
app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)
How Bootes Works apiVersion: bootes.io/v1 kind: Cluster metadata: name: example-cluster
namespace: a spec: workloadSelector: labels: app: foo config: # ... example-cluster.yaml
How Bootes Works Bootes Namespace: a Namespace: b app: foo
app: bar app: foo app: bar app: foo app: bar Namespace: c Looking up target Envoy (Pod)
How Bootes Works Bootes Namespace: a app: foo pod-xxx-333 app:
foo pod-xxx-222 app: foo pod-xxx-111
How Bootes Works Bootes kube-apiserver ! kubectl apply -f example-cluster.yaml
JSON Aggregated Discovery Service
Agenda •What is Bootes? •Implementation •Use Case
Agenda •What is Bootes? •Implementation •Use Case
Envoy Reverse Proxy Use case Pod microservice B Service Pod
Service Pod Service Pod microservice A !
Envoy Reverse Proxy Use case Pod microservice B Service Pod
Service Pod Service Pod microservice A ! CDS CDS CDS LDS RDS RDS RDS
Custom HTTP Filter { "microservice-A": "PR-1", "microservice-B": "PR-2" } microservice-A:
PR-1 microservice-B: PR-2 Request from APP Request to Upstream JWT Payload HTTP Header ɾɾɾ ɾɾɾ
Envoy Reverse Proxy Use case Pod microservice B Service Pod
Service Pod Service Pod microservice A ! CDS CDS CDS LDS RDS RDS RDS
Agenda •What is Bootes? •Implementation •Use Case