How to Make Kubernetes Controller Development Happier: mercari.go #17

Slide 1

Slide 1 text

How to Make Kubernetes Controller Development Happier mercari.go #17 Yuki Ito (@mrno110)

Slide 2

Slide 2 text

Merpay / Mercoin Architect Team Mercari Microservices Platform CI/CD Yuki Ito @mrno110

Slide 3

Slide 3 text

Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

Slide 4

Slide 4 text

Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

Slide 5

Slide 5 text

What is Kubernetes Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative con fi guration and automation. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

Slide 6

Slide 6 text

Container Orchestration 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet Pod Pod Pod Node kubectl

Slide 7

Slide 7 text

Container Orchestration kubelet containerd runc Container Node runc Container Container Runtime Interface (CRI) runc Container Open Container Initiative (OCI) speci fi cation

Slide 8

Slide 8 text

Container Orchestration kubelet containerd runc Container Node runc Container Container Runtime Interface (CRI) runc Container Open Container Initiative (OCI) speci fi cation

Slide 9

Slide 9 text

Create Container (Pod) 👨💻 kube-apiserver kubectl YAML

Slide 10

Slide 10 text

Create Container (Pod) -- - apiVersion: v 1 kind: Po d metadata : name: envo y spec : containers : - name: envo y image: envoy:lates t command : - '...'

Slide 11

Slide 11 text

Create Container (Pod) kube-apiserver Pod etcd Write Read

Slide 12

Slide 12 text

Container Orchestration 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet Pod Pod Pod Node kubectl

Slide 13

Slide 13 text

Controller Pattern A controller tracks at least one Kubernetes resource type. These objects have a spec fi eld that represents the desired state. The controller(s) for that resource are responsible for making the current state come closer to that desired state. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

Slide 14

Slide 14 text

e.g. Built-in Deployment Controller -- - apiVersion: apps/v 1 kind: Deploymen t spec : replicas: 3 template : spec : containers : - image: envoy:latest

Slide 15

Slide 15 text

e.g. Built-in Deployment Controller -- - apiVersion: apps/v 1 kind: Deploymen t spec : replicas: 3 template : spec : containers : - image: envoy:latest

Slide 16

Slide 16 text

e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller Desired Replicas: 3 Watch & Reconcile

Slide 17

Slide 17 text

e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller Pod Pod Pod Desired Replicas: 3 Watch & Reconcile

Slide 18

Slide 18 text

e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller Pod Pod Pod Desired Replicas: 3 Watch & Reconcile ❌

Slide 19

Slide 19 text

e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller Pod Pod Desired Replicas: 3 Watch & Reconcile

Slide 20

Slide 20 text

e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller Pod Pod Reconcile Pod Desired Replicas: 3

Slide 21

Slide 21 text

Custom Kubernetes Controller Current Kubernetes Resource Custom Kubernetes Controller Resource Resource Resource Watch & Reconcile Desired state

Slide 22

Slide 22 text

e.g. Pull Request Replication Controller

Slide 23

Slide 23 text

e.g. Pull Request Replication Controller

Slide 24

Slide 24 text

e.g. Pull Request Replication Controller https://events.merpay.com/techfest-2021/#day-5

Slide 25

Slide 25 text

Custom Resource De fi nition (CRD) ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget ... e.g. Istio ɾVirtualService ... e.g. spanner-autoscaler ɾSpannerAutoscaler e.g. CrossPlane ɾCloudSQLInstance ... Native Resources Custom Resources + Extends Kubernetes using CRD

Slide 26

Slide 26 text

e.g. Istio VirtualService

Slide 27

Slide 27 text

e.g. spanner-autoscaler https://github.com/mercari/spanner-autoscaler

Slide 28

Slide 28 text

e.g. spanner-autoscaler https://github.com/mercari/spanner-autoscaler apiVersion: spanner.mercari.com/v1alpha 1 kind: SpannerAutoscale r metadata : name: spannerautoscale r namespace: xx x spec : scaleTargetRef : projectId: #... instanceId: #... minNodes: 5 maxNodes: 1 0 maxScaleDownNodes: 1 targetCPUUtilization : highPriority: 60 SpannerAutoscaler

Slide 29

Slide 29 text

Custom Kubernetes Controller Current Kubernetes Resource Custom Kubernetes Controller Resource Resource Resource Watch & Reconcile Desired state

Slide 30

Slide 30 text

kubernetes / client-go https://github.com/kubernetes/client-go

Slide 31

Slide 31 text

kubernetes / client-go

Slide 32

Slide 32 text

kubernetes-sigs / controller-runtime https://github.com/kubernetes-sigs/controller-runtime

Slide 33

Slide 33 text

kubernetes-sigs / controller-runtime https://github.com/kubernetes-sigs/controller-runtime

Slide 34

Slide 34 text

kubernetes-sigs / kubebuilder https://github.com/kubernetes-sigs/kubebuilder

Slide 35

Slide 35 text

kubernetes-sigs / kubebuilder > kubebuilder init --domain tutorial.kubebuilder.io > tree . . ├── confi g │ ├── defaul t │ │ ├── kustomization.yam l │ │ ├── manager_auth_proxy_patch.yam l │ │ └── manager_config_patch.yam l │ ├── manage r │ │ ├── controller_manager_config.yam l │ │ ├── kustomization.yam l │ │ └── manager.yam l │ ├── prometheu s │ │ ├── kustomization.yam l │ │ └── monitor.yam l │ └── rba c │ ├── auth_proxy_client_clusterrole.yam l │ ├── auth_proxy_role_binding.yam l │ ├── auth_proxy_role.yam l │ ├── auth_proxy_service.yam l │ ├── kustomization.yam l │ ├── leader_election_role_binding.yam l │ ├── leader_election_role.yam l │ ├── role_binding.yam l │ └── service_account.yam l ├── Dockerfil e ├── go.mo d ├── go.su m ├── hac k │ └── boilerplate.go.tx t ├── main.g o ├── Makefil e └── PROJECT

Slide 36

Slide 36 text

Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

Slide 37

Slide 37 text

Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

Slide 38

Slide 38 text

Custom Resource De fi nition -- - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML)

Slide 39

Slide 39 text

Code Generation -- - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) Go func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Repository struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec RepositorySpe c } type RepositorySpec struct { Repository string `json:"repository,omitempty"` }

Slide 40

Slide 40 text

DeepCopyObject client-go requires implementing DeepCopyObject func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Object interface { GetObjectKind() schema.ObjectKin d DeepCopyObject() Objec t }

Slide 41

Slide 41 text

kubernetes-sigs / controller-tools (controller-gen) -- - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) Go func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Repository struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec RepositorySpe c } type RepositorySpec struct { Repository string `json:"repository,omitempty"` }

Slide 42

Slide 42 text

Code Generation ɾDe fi nition should be language agnostic ɾBut, do not want to write complex CRD YAML... controller-gen is useful, however...

Slide 43

Slide 43 text

Istio de fi nes CRDs as Protocol Bu ff ers message VirtualService { repeated string hosts = 1 ; repeated string gateways = 2 ; repeated HTTPRoute http = 3 ; // ... } Protocol Bu ff ers https://github.com/istio/api/blob/1.11.4/networking/v1beta1/virtual_service.proto

Slide 44

Slide 44 text

De fi ne CRDs as Protocol Bu ff ers message Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy

Slide 45

Slide 45 text

bufbuild / buf https://buf.build/

Slide 46

Slide 46 text

bufbuild / buf -- - version: v1beta 1 plugins : - name: g o path: ./bin/protoc-gen-g o out: . opt: paths=source_relativ e - name: deepcop y path: ./bin/protoc-gen-deepcop y out: . opt: paths=source_relative buf.gen.yaml

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

istio / tools / protoc-gen-deepcopy https://github.com/istio/tools/tree/master/cmd/protoc-gen-deepcopy

Slide 50

Slide 50 text

Generic protoc-gen-deepcopy https://github.com/protobuf-tools/protoc-gen-deepcopy

Slide 51

Slide 51 text

De fi ne CRDs as Protocol Bu ff ers type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers protoc-gen-go type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen

Slide 52

Slide 52 text

istio / tools / kubetype-gen https://github.com/istio/tools/tree/master/cmd/kubetype-gen

Slide 53

Slide 53 text

Generic kubetype-gen https://github.com/110y/kubetype-gen

Slide 54

Slide 54 text

Slide 55

Slide 55 text

De fi ne CRDs as Protocol Bu ff ers -- - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen controller-gen

Slide 56

Slide 56 text

De fi ne CRDs as Protocol Bu ff ers Protocol Bu ff ers protoc-gen-go / protoc-gen-deepcopy Protocol Bu ff er Go Struct (+ DeepCopy) Kubernetes Go Types Custom Resource De fi nitions (YAML) kubetype-gen controller-gen

Slide 57

Slide 57 text

De fi ne CRDs as Protocol Bu ff ers Protocol Bu ff ers Advanced Custom Resource De fi nitions (YAML) istio /tools / cue-gen

Slide 58

Slide 58 text

Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

Slide 59

Slide 59 text

Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

Slide 60

Slide 60 text

Kubernetes Architecture 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet Pod Pod Pod Node kubectl

Slide 61

Slide 61 text

Kubernetes Architecture kube-apiserver Resource etcd Write Read

Slide 62

Slide 62 text

Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read

Slide 63

Slide 63 text

Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read Like Database

Slide 64

Slide 64 text

envtest https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest

Slide 65

Slide 65 text

setup-envtest https://github.com/kubernetes-sigs/controller-runtime/tree/master/tools/setup-envtest

Slide 66

Slide 66 text

setup-envtest > setup-envtest use Version: 1.22. 1 OS/Arch: linux/amd6 4 Path: /path/to/kubebuilder-envtest/k8s/1.22.1-linux-amd6 4 > ls /path/to/kubebuilder-envtest/k8s/1.22.1-linux-amd6 4 etcd kube-apiserver kubectl https://github.com/kubernetes-sigs/controller-runtime/tree/master/tools/setup-envtest

Slide 67

Slide 67 text

Unit Testing with envtest et := envtest.Environment { BinaryAssetsDirectory: "/path/to/envtest-binaries" , CRDDirectoryPaths: []string{"/path/to/crd-yamls"} , ControlPlaneStartTimeout: 20 * time.Second , ErrorIfCRDPathMissing: true , AttachControlPlaneOutput: false , KubeAPIServerFlags: []string { "--advertise-address=127.0.0.1" , // ... } , } cfg, err := et.Start( ) if err != nil { return nil, nil, fmt.Errorf("failed to start envtest: %w", err ) } cli, err := client.New(cfg, client.Options { // ... })

Slide 68

Slide 68 text

Unit Testing with envtest ɾSpinning up takes 10s~ envtest is useful, however...

Slide 69

Slide 69 text

Unit Testing with envtest Caching envtest instance with TestMain func TestMain(m *testing.M) { os.Exit(func() int { cli, done, err := testutil.TestK8SClient() if err != nil { fmt.Fprintf(os.Stdout, "failed to create k8s client: %s\n", err ) return 1 } defer done( ) k8sClient = cl i return m.Run( ) }() ) }

Slide 70

Slide 70 text

Unit Testing with envtest func TestXxx(t *testing.T) { t.Parallel( ) //... k8sClient.Create(... ) }

Slide 71

Slide 71 text

Unit Testing with envtest Isolating each test by creating dedicated Namespaces func TestXxx(t *testing.T) { // ... namespace := NewNamespace(t, ctx, k8sClient ) } func NewNamespace(t *testing.T, ctx context.Context, cli client.Client) string { t.Helper( ) name := uuid.New().String( ) ns := &corev1.Namespace { ObjectMeta: metav1.ObjectMeta { Name: name , } , } if err := cli.Create(ctx, ns); err != nil { t.Fatalf("failed to create namespace: %s", err ) } return nam e }

Slide 72

Slide 72 text

Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read Like Database

Slide 73

Slide 73 text

Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

Slide 74

Slide 74 text

Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

Slide 75

Slide 75 text

Using kind for Local Development https://github.com/kubernetes-sigs/kind

Slide 76

Slide 76 text

Ska ff old https://ska ff old.dev/

Slide 77

Slide 77 text

Ska ff old + kind 👨💻 kind cluster ska ff old run Controller (Image)

Slide 78

Slide 78 text

Ska ff old + kind # ... spec : containers : - args : - /usr/bin/my-controller Pod Spec

Slide 79

Slide 79 text

Ska ff old + kind 👨💻 kind cluster ska ff old run Controller (Image)

Slide 80

Slide 80 text

ska ff old debug 👨💻 kind cluster ska ff old debug Controller (Image) + Debugger (Delve)

Slide 81

Slide 81 text

ska ff old debug # ... spec : containers : - args : - /dbg/go/bin/dlv - exe c - --headles s - --listen=:56268 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # ... initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec

Slide 82

Slide 82 text

ska ff old debug # ... spec : containers : - args : - /dbg/go/bin/dl v - exe c - --headles s - --listen=:5626 8 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # ... initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec

Slide 83

Slide 83 text

Delve https://github.com/go-delve/delve

Slide 84

Slide 84 text

ska ff old debug # ... spec : containers : - args : - /dbg/go/bin/dlv - exe c - --headles s - --listen=:56268 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # .. . initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec

Slide 85

Slide 85 text

ska ff old debug 👨💻 kind cluster ska ff old debug Controller (Image) + Debugger (Delve)