How to Make Kubernetes Controller Development Happier: mercari.go #17

Transcript

1. How to Make Kubernetes Controller Development Happier mercari.go #17 Yuki Ito (@mrno110)
2. Merpay / Mercoin Architect Team Mercari Microservices Platform CI/CD Yuki Ito @mrno110
3. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging
4. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging
5. What is Kubernetes Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative con fi guration and automation. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
6. Container Orchestration 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet Pod Pod Pod Node kubectl
7. Container Orchestration kubelet containerd runc Container Node runc Container Container Runtime Interface (CRI) runc Container Open Container Initiative (OCI) speci fi cation
8. Container Orchestration kubelet containerd runc Container Node runc Container Container Runtime Interface (CRI) runc Container Open Container Initiative (OCI) speci fi cation
9. Create Container (Pod) 👨💻 kube-apiserver kubectl YAML
10. Create Container (Pod) -- - apiVersion: v 1 kind: Po d metadata : name: envo y spec : containers : - name: envo y image: envoy:lates t command : - '...'
11. Create Container (Pod) kube-apiserver Pod etcd Write Read
12. Container Orchestration 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet Pod Pod Pod Node kubectl
13. Controller Pattern A controller tracks at least one Kubernetes resource type. These objects have a spec fi eld that represents the desired state. The controller(s) for that resource are responsible for making the current state come closer to that desired state. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
14. e.g. Built-in Deployment Controller -- - apiVersion: apps/v 1 kind: Deploymen t spec : replicas: 3 template : spec : containers : - image: envoy:latest
15. e.g. Built-in Deployment Controller -- - apiVersion: apps/v 1 kind: Deploymen t spec : replicas: 3 template : spec : containers : - image: envoy:latest
16. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller Desired Replicas: 3 Watch & Reconcile
17. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller Pod Pod Pod Desired Replicas: 3 Watch & Reconcile
18. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller Pod Pod Pod Desired Replicas: 3 Watch & Reconcile ❌
19. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller Pod Pod Desired Replicas: 3 Watch & Reconcile
20. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller Pod Pod Reconcile Pod Desired Replicas: 3
21. Custom Kubernetes Controller Current Kubernetes Resource Custom Kubernetes Controller Resource Resource Resource Watch & Reconcile Desired state
22. e.g. Pull Request Replication Controller
23. e.g. Pull Request Replication Controller
24. e.g. Pull Request Replication Controller https://events.merpay.com/techfest-2021/#day-5
25. Custom Resource De fi nition (CRD) ɾPod ɾReplicaSet ɾDeployment ɾService ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget ... e.g. Istio ɾVirtualService ... e.g. spanner-autoscaler ɾSpannerAutoscaler e.g. CrossPlane ɾCloudSQLInstance ... Native Resources Custom Resources + Extends Kubernetes using CRD
26. e.g. Istio VirtualService
27. e.g. spanner-autoscaler https://github.com/mercari/spanner-autoscaler
28. e.g. spanner-autoscaler https://github.com/mercari/spanner-autoscaler apiVersion: spanner.mercari.com/v1alpha 1 kind: SpannerAutoscale r metadata : name: spannerautoscale r namespace: xx x spec : scaleTargetRef : projectId: #... instanceId: #... minNodes: 5 maxNodes: 1 0 maxScaleDownNodes: 1 targetCPUUtilization : highPriority: 60 SpannerAutoscaler
29. Custom Kubernetes Controller Current Kubernetes Resource Custom Kubernetes Controller Resource Resource Resource Watch & Reconcile Desired state
30. kubernetes / client-go https://github.com/kubernetes/client-go
31. kubernetes / client-go
32. kubernetes-sigs / controller-runtime https://github.com/kubernetes-sigs/controller-runtime
33. kubernetes-sigs / controller-runtime https://github.com/kubernetes-sigs/controller-runtime
34. kubernetes-sigs / kubebuilder https://github.com/kubernetes-sigs/kubebuilder
35. kubernetes-sigs / kubebuilder > kubebuilder init --domain tutorial.kubebuilder.io > tree . . ├── confi g │ ├── defaul t │ │ ├── kustomization.yam l │ │ ├── manager_auth_proxy_patch.yam l │ │ └── manager_config_patch.yam l │ ├── manage r │ │ ├── controller_manager_config.yam l │ │ ├── kustomization.yam l │ │ └── manager.yam l │ ├── prometheu s │ │ ├── kustomization.yam l │ │ └── monitor.yam l │ └── rba c │ ├── auth_proxy_client_clusterrole.yam l │ ├── auth_proxy_role_binding.yam l │ ├── auth_proxy_role.yam l │ ├── auth_proxy_service.yam l │ ├── kustomization.yam l │ ├── leader_election_role_binding.yam l │ ├── leader_election_role.yam l │ ├── role_binding.yam l │ └── service_account.yam l ├── Dockerfil e ├── go.mo d ├── go.su m ├── hac k │ └── boilerplate.go.tx t ├── main.g o ├── Makefil e └── PROJECT
36. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging
37. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging
38. Custom Resource De fi nition -- - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML)
39. Code Generation -- - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) Go func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Repository struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec RepositorySpe c } type RepositorySpec struct { Repository string `json:"repository,omitempty"` }
40. DeepCopyObject client-go requires implementing DeepCopyObject func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Object interface { GetObjectKind() schema.ObjectKin d DeepCopyObject() Objec t }
41. kubernetes-sigs / controller-tools (controller-gen) -- - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) Go func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Repository struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec RepositorySpe c } type RepositorySpec struct { Repository string `json:"repository,omitempty"` }
42. Code Generation ɾDe fi nition should be language agnostic ɾBut, do not want to write complex CRD YAML... controller-gen is useful, however...
43. Istio de fi nes CRDs as Protocol Bu ff ers message VirtualService { repeated string hosts = 1 ; repeated string gateways = 2 ; repeated HTTPRoute http = 3 ; // ... } Protocol Bu ff ers https://github.com/istio/api/blob/1.11.4/networking/v1beta1/virtual_service.proto
44. De fi ne CRDs as Protocol Bu ff ers message Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy
45. bufbuild / buf https://buf.build/
46. bufbuild / buf -- - version: v1beta 1 plugins : - name: g o path: ./bin/protoc-gen-g o out: . opt: paths=source_relativ e - name: deepcop y path: ./bin/protoc-gen-deepcop y out: . opt: paths=source_relative buf.gen.yaml
47. De fi ne CRDs as Protocol Bu ff ers message Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy
48. De fi ne CRDs as Protocol Bu ff ers message Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy
49. istio / tools / protoc-gen-deepcopy https://github.com/istio/tools/tree/master/cmd/protoc-gen-deepcopy
50. Generic protoc-gen-deepcopy https://github.com/protobuf-tools/protoc-gen-deepcopy
51. De fi ne CRDs as Protocol Bu ff ers type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers protoc-gen-go type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen
52. istio / tools / kubetype-gen https://github.com/istio/tools/tree/master/cmd/kubetype-gen
53. Generic kubetype-gen https://github.com/110y/kubetype-gen
54. De fi ne CRDs as Protocol Bu ff ers type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers protoc-gen-go type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen
55. De fi ne CRDs as Protocol Bu ff ers -- - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen controller-gen
56. De fi ne CRDs as Protocol Bu ff ers Protocol Bu ff ers protoc-gen-go / protoc-gen-deepcopy Protocol Bu ff er Go Struct (+ DeepCopy) Kubernetes Go Types Custom Resource De fi nitions (YAML) kubetype-gen controller-gen
57. De fi ne CRDs as Protocol Bu ff ers Protocol Bu ff ers Advanced Custom Resource De fi nitions (YAML) istio /tools / cue-gen
58. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging
59. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging
60. Kubernetes Architecture 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet Pod Pod Pod Node kubectl
61. Kubernetes Architecture kube-apiserver Resource etcd Write Read
62. Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read
63. Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read Like Database
64. envtest https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest
65. setup-envtest https://github.com/kubernetes-sigs/controller-runtime/tree/master/tools/setup-envtest
66. setup-envtest > setup-envtest use Version: 1.22. 1 OS/Arch: linux/amd6 4 Path: /path/to/kubebuilder-envtest/k8s/1.22.1-linux-amd6 4 > ls /path/to/kubebuilder-envtest/k8s/1.22.1-linux-amd6 4 etcd kube-apiserver kubectl https://github.com/kubernetes-sigs/controller-runtime/tree/master/tools/setup-envtest
67. Unit Testing with envtest et := envtest.Environment { BinaryAssetsDirectory: "/path/to/envtest-binaries" , CRDDirectoryPaths: []string{"/path/to/crd-yamls"} , ControlPlaneStartTimeout: 20 * time.Second , ErrorIfCRDPathMissing: true , AttachControlPlaneOutput: false , KubeAPIServerFlags: []string { "--advertise-address=127.0.0.1" , // ... } , } cfg, err := et.Start( ) if err != nil { return nil, nil, fmt.Errorf("failed to start envtest: %w", err ) } cli, err := client.New(cfg, client.Options { // ... })
68. Unit Testing with envtest ɾSpinning up takes 10s~ envtest is useful, however...
69. Unit Testing with envtest Caching envtest instance with TestMain func TestMain(m *testing.M) { os.Exit(func() int { cli, done, err := testutil.TestK8SClient() if err != nil { fmt.Fprintf(os.Stdout, "failed to create k8s client: %s\n", err ) return 1 } defer done( ) k8sClient = cl i return m.Run( ) }() ) }
70. Unit Testing with envtest func TestXxx(t *testing.T) { t.Parallel( ) //... k8sClient.Create(... ) }
71. Unit Testing with envtest Isolating each test by creating dedicated Namespaces func TestXxx(t *testing.T) { // ... namespace := NewNamespace(t, ctx, k8sClient ) } func NewNamespace(t *testing.T, ctx context.Context, cli client.Client) string { t.Helper( ) name := uuid.New().String( ) ns := &corev1.Namespace { ObjectMeta: metav1.ObjectMeta { Name: name , } , } if err := cli.Create(ctx, ns); err != nil { t.Fatalf("failed to create namespace: %s", err ) } return nam e }
72. Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read Like Database
73. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging
74. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging
75. Using kind for Local Development https://github.com/kubernetes-sigs/kind
76. Ska ff old https://ska ff old.dev/
77. Ska ff old + kind 👨💻 kind cluster ska ff old run Controller (Image)
78. Ska ff old + kind # ... spec : containers : - args : - /usr/bin/my-controller Pod Spec
79. Ska ff old + kind 👨💻 kind cluster ska ff old run Controller (Image)
80. ska ff old debug 👨💻 kind cluster ska ff old debug Controller (Image) + Debugger (Delve)
81. ska ff old debug # ... spec : containers : - args : - /dbg/go/bin/dlv - exe c - --headles s - --listen=:56268 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # ... initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec
82. ska ff old debug # ... spec : containers : - args : - /dbg/go/bin/dl v - exe c - --headles s - --listen=:5626 8 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # ... initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec
83. Delve https://github.com/go-delve/delve
84. ska ff old debug # ... spec : containers : - args : - /dbg/go/bin/dlv - exe c - --headles s - --listen=:56268 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # .. . initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec
85. ska ff old debug 👨💻 kind cluster ska ff old debug Controller (Image) + Debugger (Delve)
86. ska ff old debug https://gihyo.jp/magazine/SD/archive/2020/202008
87. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging