How to Make Kubernetes Controller Development Happier: mercari.go #17

Transcript

1. How to Make Kubernetes Controller Development Happier mercari.go #17 Yuki

   Ito (@mrno110)

2. Merpay / Mercoin Architect Team Mercari Microservices Platform CI/CD Yuki

   Ito @mrno110

3. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

4. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

5. What is Kubernetes Kubernetes is a portable, extensible, open-source platform

   for managing containerized workloads and services, that facilitates both declarative con fi guration and automation. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

6. Container Orchestration 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet

   Pod Pod Pod Node kubectl

7. Container Orchestration kubelet containerd runc Container Node runc Container Container

   Runtime Interface (CRI) runc Container Open Container Initiative (OCI) speci fi cation

8. Container Orchestration kubelet containerd runc Container Node runc Container Container

   Runtime Interface (CRI) runc Container Open Container Initiative (OCI) speci fi cation

9. Create Container (Pod) 👨💻 kube-apiserver kubectl YAML

10. Create Container (Pod) -- - apiVersion: v 1 kind: Po

    d metadata : name: envo y spec : containers : - name: envo y image: envoy:lates t command : - '...'

11. Create Container (Pod) kube-apiserver Pod etcd Write Read

12. Container Orchestration 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet

    Pod Pod Pod Node kubectl

13. Controller Pattern A controller tracks at least one Kubernetes resource

    type. These objects have a spec fi eld that represents the desired state. The controller(s) for that resource are responsible for making the current state come closer to that desired state. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

14. e.g. Built-in Deployment Controller -- - apiVersion: apps/v 1 kind:

    Deploymen t spec : replicas: 3 template : spec : containers : - image: envoy:latest

15. e.g. Built-in Deployment Controller -- - apiVersion: apps/v 1 kind:

    Deploymen t spec : replicas: 3 template : spec : containers : - image: envoy:latest

16. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller

    Desired Replicas: 3 Watch & Reconcile

17. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller

    Pod Pod Pod Desired Replicas: 3 Watch & Reconcile

18. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller

    Pod Pod Pod Desired Replicas: 3 Watch & Reconcile ❌

19. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller

    Pod Pod Desired Replicas: 3 Watch & Reconcile

20. e.g. Built-in Deployment Controller Current Kubernetes Resource Deployment (ReplicaSet) Controller

    Pod Pod Reconcile Pod Desired Replicas: 3

21. Custom Kubernetes Controller Current Kubernetes Resource Custom Kubernetes Controller Resource

    Resource Resource Watch & Reconcile Desired state

22. e.g. Pull Request Replication Controller

23. e.g. Pull Request Replication Controller

24. e.g. Pull Request Replication Controller https://events.merpay.com/techfest-2021/#day-5

25. Custom Resource De fi nition (CRD) ɾPod ɾReplicaSet ɾDeployment ɾService

    ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget ... e.g. Istio ɾVirtualService ... e.g. spanner-autoscaler ɾSpannerAutoscaler e.g. CrossPlane ɾCloudSQLInstance ... Native Resources Custom Resources + Extends Kubernetes using CRD

26. e.g. Istio VirtualService

27. e.g. spanner-autoscaler https://github.com/mercari/spanner-autoscaler

28. e.g. spanner-autoscaler https://github.com/mercari/spanner-autoscaler apiVersion: spanner.mercari.com/v1alpha 1 kind: SpannerAutoscale r metadata

    : name: spannerautoscale r namespace: xx x spec : scaleTargetRef : projectId: #... instanceId: #... minNodes: 5 maxNodes: 1 0 maxScaleDownNodes: 1 targetCPUUtilization : highPriority: 60 SpannerAutoscaler

29. Custom Kubernetes Controller Current Kubernetes Resource Custom Kubernetes Controller Resource

    Resource Resource Watch & Reconcile Desired state

30. kubernetes / client-go https://github.com/kubernetes/client-go

31. kubernetes / client-go

32. kubernetes-sigs / controller-runtime https://github.com/kubernetes-sigs/controller-runtime

33. kubernetes-sigs / controller-runtime https://github.com/kubernetes-sigs/controller-runtime

34. kubernetes-sigs / kubebuilder https://github.com/kubernetes-sigs/kubebuilder

35. kubernetes-sigs / kubebuilder > kubebuilder init --domain tutorial.kubebuilder.io > tree

    . . ├── confi g │ ├── defaul t │ │ ├── kustomization.yam l │ │ ├── manager_auth_proxy_patch.yam l │ │ └── manager_config_patch.yam l │ ├── manage r │ │ ├── controller_manager_config.yam l │ │ ├── kustomization.yam l │ │ └── manager.yam l │ ├── prometheu s │ │ ├── kustomization.yam l │ │ └── monitor.yam l │ └── rba c │ ├── auth_proxy_client_clusterrole.yam l │ ├── auth_proxy_role_binding.yam l │ ├── auth_proxy_role.yam l │ ├── auth_proxy_service.yam l │ ├── kustomization.yam l │ ├── leader_election_role_binding.yam l │ ├── leader_election_role.yam l │ ├── role_binding.yam l │ └── service_account.yam l ├── Dockerfil e ├── go.mo d ├── go.su m ├── hac k │ └── boilerplate.go.tx t ├── main.g o ├── Makefil e └── PROJECT

36. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

37. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

38. Custom Resource De fi nition -- - kind: CustomResourceDefinitio n

    spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML)

39. Code Generation -- - kind: CustomResourceDefinitio n spec : names

    : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) Go func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Repository struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec RepositorySpe c } type RepositorySpec struct { Repository string `json:"repository,omitempty"` }

40. DeepCopyObject client-go requires implementing DeepCopyObject func (in *Repository) DeepCopyObject() runtime.Object

    { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Object interface { GetObjectKind() schema.ObjectKin d DeepCopyObject() Objec t }

41. kubernetes-sigs / controller-tools (controller-gen) -- - kind: CustomResourceDefinitio n spec

    : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) Go func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Repository struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec RepositorySpe c } type RepositorySpec struct { Repository string `json:"repository,omitempty"` }

42. Code Generation ɾDe fi nition should be language agnostic ɾBut,

    do not want to write complex CRD YAML... controller-gen is useful, however...

43. Istio de fi nes CRDs as Protocol Bu ff ers

    message VirtualService { repeated string hosts = 1 ; repeated string gateways = 2 ; repeated HTTPRoute http = 3 ; // ... } Protocol Bu ff ers https://github.com/istio/api/blob/1.11.4/networking/v1beta1/virtual_service.proto

44. De fi ne CRDs as Protocol Bu ff ers message

    Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy

45. bufbuild / buf https://buf.build/

46. bufbuild / buf -- - version: v1beta 1 plugins :

    - name: g o path: ./bin/protoc-gen-g o out: . opt: paths=source_relativ e - name: deepcop y path: ./bin/protoc-gen-deepcop y out: . opt: paths=source_relative buf.gen.yaml

47. De fi ne CRDs as Protocol Bu ff ers message

    Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy

48. De fi ne CRDs as Protocol Bu ff ers message

    Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy

49. istio / tools / protoc-gen-deepcopy https://github.com/istio/tools/tree/master/cmd/protoc-gen-deepcopy

50. Generic protoc-gen-deepcopy https://github.com/protobuf-tools/protoc-gen-deepcopy

51. De fi ne CRDs as Protocol Bu ff ers type

    Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers protoc-gen-go type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen

52. istio / tools / kubetype-gen https://github.com/istio/tools/tree/master/cmd/kubetype-gen

53. Generic kubetype-gen https://github.com/110y/kubetype-gen

54. De fi ne CRDs as Protocol Bu ff ers type

    Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers protoc-gen-go type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen

55. De fi ne CRDs as Protocol Bu ff ers --

    - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen controller-gen

56. De fi ne CRDs as Protocol Bu ff ers Protocol

    Bu ff ers protoc-gen-go / protoc-gen-deepcopy Protocol Bu ff er Go Struct (+ DeepCopy) Kubernetes Go Types Custom Resource De fi nitions (YAML) kubetype-gen controller-gen

57. De fi ne CRDs as Protocol Bu ff ers Protocol

    Bu ff ers Advanced Custom Resource De fi nitions (YAML) istio /tools / cue-gen

58. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

59. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

60. Kubernetes Architecture 👨💻 kube-apiserver kubelet Pod Pod Pod Node kubelet

    Pod Pod Pod Node kubectl

61. Kubernetes Architecture kube-apiserver Resource etcd Write Read

62. Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read

63. Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read

    Like Database

64. envtest https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest

65. setup-envtest https://github.com/kubernetes-sigs/controller-runtime/tree/master/tools/setup-envtest

66. setup-envtest > setup-envtest use Version: 1.22. 1 OS/Arch: linux/amd6 4

    Path: /path/to/kubebuilder-envtest/k8s/1.22.1-linux-amd6 4 > ls /path/to/kubebuilder-envtest/k8s/1.22.1-linux-amd6 4 etcd kube-apiserver kubectl https://github.com/kubernetes-sigs/controller-runtime/tree/master/tools/setup-envtest

67. Unit Testing with envtest et := envtest.Environment { BinaryAssetsDirectory: "/path/to/envtest-binaries"

    , CRDDirectoryPaths: []string{"/path/to/crd-yamls"} , ControlPlaneStartTimeout: 20 * time.Second , ErrorIfCRDPathMissing: true , AttachControlPlaneOutput: false , KubeAPIServerFlags: []string { "--advertise-address=127.0.0.1" , // ... } , } cfg, err := et.Start( ) if err != nil { return nil, nil, fmt.Errorf("failed to start envtest: %w", err ) } cli, err := client.New(cfg, client.Options { // ... })

68. Unit Testing with envtest ɾSpinning up takes 10s~ envtest is

    useful, however...

69. Unit Testing with envtest Caching envtest instance with TestMain func

    TestMain(m *testing.M) { os.Exit(func() int { cli, done, err := testutil.TestK8SClient() if err != nil { fmt.Fprintf(os.Stdout, "failed to create k8s client: %s\n", err ) return 1 } defer done( ) k8sClient = cl i return m.Run( ) }() ) }

70. Unit Testing with envtest func TestXxx(t *testing.T) { t.Parallel( )

    //... k8sClient.Create(... ) }

71. Unit Testing with envtest Isolating each test by creating dedicated

    Namespaces func TestXxx(t *testing.T) { // ... namespace := NewNamespace(t, ctx, k8sClient ) } func NewNamespace(t *testing.T, ctx context.Context, cli client.Client) string { t.Helper( ) name := uuid.New().String( ) ns := &corev1.Namespace { ObjectMeta: metav1.ObjectMeta { Name: name , } , } if err := cli.Create(ctx, ns); err != nil { t.Fatalf("failed to create namespace: %s", err ) } return nam e }

72. Kubernetes Architecture kube-apiserver etcd Write Read Kubernetes Controller Write Read

    Like Database

73. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

74. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging

75. Using kind for Local Development https://github.com/kubernetes-sigs/kind

76. Ska ff old https://ska ff old.dev/

77. Ska ff old + kind 👨💻 kind cluster ska ff

    old run Controller (Image)

78. Ska ff old + kind # ... spec : containers

    : - args : - /usr/bin/my-controller Pod Spec

79. Ska ff old + kind 👨💻 kind cluster ska ff

    old run Controller (Image)

80. ska ff old debug 👨💻 kind cluster ska ff old

    debug Controller (Image) + Debugger (Delve)

81. ska ff old debug # ... spec : containers :

    - args : - /dbg/go/bin/dlv - exe c - --headles s - --listen=:56268 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # ... initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec

82. ska ff old debug # ... spec : containers :

    - args : - /dbg/go/bin/dl v - exe c - --headles s - --listen=:5626 8 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # ... initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec

83. Delve https://github.com/go-delve/delve

84. ska ff old debug # ... spec : containers :

    - args : - /dbg/go/bin/dlv - exe c - --headles s - --listen=:56268 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # .. . initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec

85. ska ff old debug 👨💻 kind cluster ska ff old

    debug Controller (Image) + Debugger (Delve)

86. ska ff old debug https://gihyo.jp/magazine/SD/archive/2020/202008

87. Agenda ɾWhat is Kubernetes Controller ɾCode Generation ɾTesting ɾDebugging