Save 37% off PRO during our Black Friday Sale! »

How to Make Kubernetes Controller Development Happier: mercari.go #17

1bfc6e2ed04a895bb36f36b86828b689?s=47 Yuki Ito
October 21, 2021

How to Make Kubernetes Controller Development Happier: mercari.go #17

1bfc6e2ed04a895bb36f36b86828b689?s=128

Yuki Ito

October 21, 2021
Tweet

Transcript

  1. How to Make Kubernetes Controller Development Happier mercari.go #17 Yuki

    Ito (@mrno110)
  2. Merpay / Mercoin Architect Team Mercari Microservices Platform CI/CD Yuki

    Ito @mrno110
  3. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

  4. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

  5. What is Kubernetes Kubernetes is a portable, extensible, open-source platform

    for managing containerized workloads and services, that facilitates both declarative con fi guration and automation. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
  6. Container Orchestration 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet

    Pod Pod Pod Node kubectl
  7. Container Orchestration kubelet containerd runc Container Node runc Container Container

    Runtime Interface (CRI) runc Container Open Container Initiative (OCI) speci fi cation
  8. Container Orchestration kubelet containerd runc Container Node runc Container Container

    Runtime Interface (CRI) runc Container Open Container Initiative (OCI) speci fi cation
  9. Create Container (Pod) 👨💻 kube-apiserver kubectl YAML

  10. Create Container (Pod) -- - apiVersion: v 1 kind: Po

    d metadata : name: envo y spec : containers : - name: envo y image: envoy:lates t command : - '...'
  11. Create Container (Pod) kube-apiserver Pod etcd Write Read

  12. Container Orchestration 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet

    Pod Pod Pod Node kubectl
  13. Controller Pattern A controller tracks at least one Kubernetes resource

    type. These objects have a spec fi eld that represents the desired state. The controller(s) for that resource are responsible for making the current state come closer to that desired state. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
  14. e.g. Built-in Deployment Controller -- - apiVersion: apps/v 1 kind:

    Deploymen t spec : replicas: 3 template : spec : containers : - image: envoy:latest
  15. e.g. Built-in Deployment Controller -- - apiVersion: apps/v 1 kind:

    Deploymen t spec : replicas: 3 template : spec : containers : - image: envoy:latest
  16. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller

    Desired Replicas: 3 Watch & Reconcile
  17. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller

    Pod Pod Pod Desired Replicas: 3 Watch & Reconcile
  18. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller

    Pod Pod Pod Desired Replicas: 3 Watch & Reconcile ❌
  19. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller

    Pod Pod Desired Replicas: 3 Watch & Reconcile
  20. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller

    Pod Pod Reconcile Pod Desired Replicas: 3
  21. Custom Kubernetes Controller Current Kubernetes Resource Custom Kubernetes Controller Resource

    Resource Resource Watch & Reconcile Desired state
  22. e.g. Pull Request Replication Controller

  23. e.g. Pull Request Replication Controller

  24. e.g. Pull Request Replication Controller https://events.merpay.com/techfest-2021/#day-5

  25. Custom Resource De fi nition (CRD) ɾPod ɾReplicaSet ɾDeployment ɾService

    ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget ... e.g. Istio ɾVirtualService ... e.g. spanner-autoscaler ɾSpannerAutoscaler e.g. CrossPlane ɾCloudSQLInstance ... Native Resources Custom Resources + Extends Kubernetes using CRD
  26. e.g. Istio VirtualService

  27. e.g. spanner-autoscaler https://github.com/mercari/spanner-autoscaler

  28. e.g. spanner-autoscaler https://github.com/mercari/spanner-autoscaler apiVersion: spanner.mercari.com/v1alpha 1 kind: SpannerAutoscale r metadata

    : name: spannerautoscale r namespace: xx x spec : scaleTargetRef : projectId: #... instanceId: #... minNodes: 5 maxNodes: 1 0 maxScaleDownNodes: 1 targetCPUUtilization : highPriority: 60 SpannerAutoscaler
  29. Custom Kubernetes Controller Current Kubernetes Resource Custom Kubernetes Controller Resource

    Resource Resource Watch & Reconcile Desired state
  30. kubernetes / client-go https://github.com/kubernetes/client-go

  31. kubernetes / client-go

  32. kubernetes-sigs / controller-runtime https://github.com/kubernetes-sigs/controller-runtime

  33. kubernetes-sigs / controller-runtime https://github.com/kubernetes-sigs/controller-runtime

  34. kubernetes-sigs / kubebuilder https://github.com/kubernetes-sigs/kubebuilder

  35. kubernetes-sigs / kubebuilder > kubebuilder init --domain tutorial.kubebuilder.io > tree

    . . ├── confi g │ ├── defaul t │ │ ├── kustomization.yam l │ │ ├── manager_auth_proxy_patch.yam l │ │ └── manager_config_patch.yam l │ ├── manage r │ │ ├── controller_manager_config.yam l │ │ ├── kustomization.yam l │ │ └── manager.yam l │ ├── prometheu s │ │ ├── kustomization.yam l │ │ └── monitor.yam l │ └── rba c │ ├── auth_proxy_client_clusterrole.yam l │ ├── auth_proxy_role_binding.yam l │ ├── auth_proxy_role.yam l │ ├── auth_proxy_service.yam l │ ├── kustomization.yam l │ ├── leader_election_role_binding.yam l │ ├── leader_election_role.yam l │ ├── role_binding.yam l │ └── service_account.yam l ├── Dockerfil e ├── go.mo d ├── go.su m ├── hac k │ └── boilerplate.go.tx t ├── main.g o ├── Makefil e └── PROJECT
  36. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

  37. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

  38. Custom Resource De fi nition -- - kind: CustomResourceDefinitio n

    spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML)
  39. Code Generation -- - kind: CustomResourceDefinitio n spec : names

    : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) Go func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Repository struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec RepositorySpe c } type RepositorySpec struct { Repository string `json:"repository,omitempty"` }
  40. DeepCopyObject client-go requires implementing DeepCopyObject func (in *Repository) DeepCopyObject() runtime.Object

    { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Object interface { GetObjectKind() schema.ObjectKin d DeepCopyObject() Objec t }
  41. kubernetes-sigs / controller-tools (controller-gen) -- - kind: CustomResourceDefinitio n spec

    : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) Go func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Repository struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec RepositorySpe c } type RepositorySpec struct { Repository string `json:"repository,omitempty"` }
  42. Code Generation ɾDe fi nition should be language agnostic ɾBut,

    do not want to write complex CRD YAML... controller-gen is useful, however...
  43. Istio de fi nes CRDs as Protocol Bu ff ers

    message VirtualService { repeated string hosts = 1 ; repeated string gateways = 2 ; repeated HTTPRoute http = 3 ; // ... } Protocol Bu ff ers https://github.com/istio/api/blob/1.11.4/networking/v1beta1/virtual_service.proto
  44. De fi ne CRDs as Protocol Bu ff ers message

    Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy
  45. bufbuild / buf https://buf.build/

  46. bufbuild / buf -- - version: v1beta 1 plugins :

    - name: g o path: ./bin/protoc-gen-g o out: . opt: paths=source_relativ e - name: deepcop y path: ./bin/protoc-gen-deepcop y out: . opt: paths=source_relative buf.gen.yaml
  47. De fi ne CRDs as Protocol Bu ff ers message

    Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy
  48. De fi ne CRDs as Protocol Bu ff ers message

    Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy
  49. istio / tools / protoc-gen-deepcopy https://github.com/istio/tools/tree/master/cmd/protoc-gen-deepcopy

  50. Generic protoc-gen-deepcopy https://github.com/protobuf-tools/protoc-gen-deepcopy

  51. De fi ne CRDs as Protocol Bu ff ers type

    Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers protoc-gen-go type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen
  52. istio / tools / kubetype-gen https://github.com/istio/tools/tree/master/cmd/kubetype-gen

  53. Generic kubetype-gen https://github.com/110y/kubetype-gen

  54. De fi ne CRDs as Protocol Bu ff ers type

    Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers protoc-gen-go type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen
  55. De fi ne CRDs as Protocol Bu ff ers --

    - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen controller-gen
  56. De fi ne CRDs as Protocol Bu ff ers Protocol

    Bu ff ers protoc-gen-go / protoc-gen-deepcopy Protocol Bu ff er Go Struct (+ DeepCopy) Kubernetes Go Types Custom Resource De fi nitions (YAML) kubetype-gen controller-gen
  57. De fi ne CRDs as Protocol Bu ff ers Protocol

    Bu ff ers Advanced Custom Resource De fi nitions (YAML) istio /tools / cue-gen
  58. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

  59. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

  60. Kubernetes Architecture 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet

    Pod Pod Pod Node kubectl
  61. Kubernetes Architecture kube-apiserver Resource etcd Write Read

  62. Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read

  63. Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read

    Like Database
  64. envtest https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest

  65. setup-envtest https://github.com/kubernetes-sigs/controller-runtime/tree/master/tools/setup-envtest

  66. setup-envtest > setup-envtest use Version: 1.22. 1 OS/Arch: linux/amd6 4

    Path: /path/to/kubebuilder-envtest/k8s/1.22.1-linux-amd6 4 > ls /path/to/kubebuilder-envtest/k8s/1.22.1-linux-amd6 4 etcd kube-apiserver kubectl https://github.com/kubernetes-sigs/controller-runtime/tree/master/tools/setup-envtest
  67. Unit Testing with envtest et := envtest.Environment { BinaryAssetsDirectory: "/path/to/envtest-binaries"

    , CRDDirectoryPaths: []string{"/path/to/crd-yamls"} , ControlPlaneStartTimeout: 20 * time.Second , ErrorIfCRDPathMissing: true , AttachControlPlaneOutput: false , KubeAPIServerFlags: []string { "--advertise-address=127.0.0.1" , // ... } , } cfg, err := et.Start( ) if err != nil { return nil, nil, fmt.Errorf("failed to start envtest: %w", err ) } cli, err := client.New(cfg, client.Options { // ... })
  68. Unit Testing with envtest ɾSpinning up takes 10s~ envtest is

    useful, however...
  69. Unit Testing with envtest Caching envtest instance with TestMain func

    TestMain(m *testing.M) { os.Exit(func() int { cli, done, err := testutil.TestK8SClient() if err != nil { fmt.Fprintf(os.Stdout, "failed to create k8s client: %s\n", err ) return 1 } defer done( ) k8sClient = cl i return m.Run( ) }() ) }
  70. Unit Testing with envtest func TestXxx(t *testing.T) { t.Parallel( )

    //... k8sClient.Create(... ) }
  71. Unit Testing with envtest Isolating each test by creating dedicated

    Namespaces func TestXxx(t *testing.T) { // ... namespace := NewNamespace(t, ctx, k8sClient ) } func NewNamespace(t *testing.T, ctx context.Context, cli client.Client) string { t.Helper( ) name := uuid.New().String( ) ns := &corev1.Namespace { ObjectMeta: metav1.ObjectMeta { Name: name , } , } if err := cli.Create(ctx, ns); err != nil { t.Fatalf("failed to create namespace: %s", err ) } return nam e }
  72. Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read

    Like Database
  73. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

  74. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

  75. Using kind for Local Development https://github.com/kubernetes-sigs/kind

  76. Ska ff old https://ska ff old.dev/

  77. Ska ff old + kind 👨💻 kind cluster ska ff

    old run Controller (Image)
  78. Ska ff old + kind # ... spec : containers

    : - args : - /usr/bin/my-controller Pod Spec
  79. Ska ff old + kind 👨💻 kind cluster ska ff

    old run Controller (Image)
  80. ska ff old debug 👨💻 kind cluster ska ff old

    debug Controller (Image) + Debugger (Delve)
  81. ska ff old debug # ... spec : containers :

    - args : - /dbg/go/bin/dlv - exe c - --headles s - --listen=:56268 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # ... initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec
  82. ska ff old debug # ... spec : containers :

    - args : - /dbg/go/bin/dl v - exe c - --headles s - --listen=:5626 8 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # ... initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec
  83. Delve https://github.com/go-delve/delve

  84. ska ff old debug # ... spec : containers :

    - args : - /dbg/go/bin/dlv - exe c - --headles s - --listen=:56268 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # .. . initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec
  85. ska ff old debug 👨💻 kind cluster ska ff old

    debug Controller (Image) + Debugger (Delve)
  86. ska ff old debug https://gihyo.jp/magazine/SD/archive/2020/202008

  87. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging