Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes基盤における運用フローのController化と継続的な改善 / kube...

Kubernetes基盤における運用フローのController化と継続的な改善 / kubernetes-controller-improvements

動画はこちらから
https://cadc.cyberagent.co.jp/2022/program/kubernetes-controller-improvements/

青山 真也
CyberAgent - CIU
ソフトウェアエンジニア / KaaS プロダクトオーナー

2016年新卒入社。Kubernetes/CloudNative領域のCA Developer Experts。CloudNative Days Tokyo のCo-chair、Kubernetes Meetup TokyoのOrganizerなどコミュニティ活動にも従事。著書に『Kubernetes完全ガイド』等。

CyberAgent group Infrastructure Unit の Dev チームでは、これまで約 5 年間 Kubernetes を拡張しながらサービスを提供するためのプラットフォームとして利用実績を積み上げてきました。

例えば、マネージド Kubernetes を提供する KaaS では、自分たちの運用・サポートの業務知識をKubernetesに組み込み、クラスタの運用を人が毎回考えずに・人を介さずに自律的に行えるように、運用に関する作業は KRM に従って Kubernetes Controller などの拡張性を活かし、ロジックをプログラムコード化しています。
一連の運用の処理がコード化されているため、運用フローや障害時対応の誤りを継続的に改善し続けることもできます。

本セッションでは、KaaS で実現している Controller の実装例とともに現在の KaaS 基盤について紹介し、皆さんがController化することができるパターンについてイメージが湧くことを目指します。

Masaya Aoyama (@amsy810)

March 23, 2022
Tweet

More Decks by Masaya Aoyama (@amsy810)

Other Decks in Programming

Transcript

  1.  ίϯςφΦʔέετϨʔλʔ ௐࠪɾٕज़બఆ։࢝    0QFO4UBDL)FBU $POTVMϕʔεͷ ,VCFSOFUFTBTB4FSWJDFΛϦϦʔε ʢ",&Wʣ

    $MVTUFS"1*ϕʔεͷ ,BB4ͷઃܭɾ࣮૷΁ $MVTUFS"1*ϕʔεͷ ,BB4ͷϦϦʔε ʢ",&Wʣ ਺ଟ͘ͷ,VCFSOFUFT$POUSPMMFSͷ࣮૷
  2. ",&WXJUI$MVTUFS"1* ؅ཧΫϥελʹσϓϩΠ͞Εͨ$MVTUFSϦιʔεΛݩʹϓϥΠϕʔτΫϥ΢υͷ"1*ͳͲΛૢ࡞ ,VCFSOFUFTΫϥελΛߏஙɾ؅ཧ͢Δ࢓૊Έ ؅ཧΫϥελ W ϢʔβʔΫϥελ W ϢʔβʔΫϥελ kind: Cluster

    metadata: name: test-cluster-01 spec: topology: version: v1.20.7 controlPlane: replicas: 3 workers: machineDeployments: - class: type1-workers name: cpu-nodepool-01 replicas: 3 kind: Cluster metadata: name: test-cluster-01 spec: topology: version: v1.20.7 controlPlane: replicas: 3 workers: machineDeployments: - class: type1-workers name: cpu-nodepool-01 replicas: 3 CVJMENBOBHF CVJMENBOBHF $VTUPN$POUSPMMFST BOE $VTUPN8FCIPPLT $VTUPN$POUSPMMFST BOE $VTUPN8FCIPPLT ϚϧνϓϥΠϕʔτΫϥ΢υ΋ରԠ
  3. ,VCFSOFUFT$POUSPMMFS ಛఆͷϦιʔεͷঢ়ଶΛએݴ͞Εͨঢ়ଶʹௐ੔͢ΔϓϩάϥϜ ,VCFSOFUFT"1*Λհͯ͠Ϧιʔεͷૢ࡞΍೚ҙͷॲཧΛߦ͏ ྫʣ3FQMJDB4FU$POUSPMMFSɿࢦఆ͞ΕͨϨϓϦΧ਺Ͱ1PEΛҡ࣋ ,VCFSOFUFTΫϥελ 3FQMJDB4FU$POUSPMMFS kind: ReplicaSet spec: replicas:

    3 template: spec: containers: - name: app image: nginx:1.19 kind: Pod spec: containers: - name: app image: nginx:1.19 kind: Pod spec: containers: - name: app image: nginx:1.19 kind: Pod spec: containers: - name: app image: nginx:1.19 XBUDI 3FQMJDB4FUͷఆٛΛݩʹ 1PEͷ࡞੒࡟আΛߦ͍ɺ ϨϓϦΧ਺Λҡ࣋ $SFBUF1PE %FMFUF1PE
  4. ,VCFSOFUFT$POUSPMMFS ಛఆͷϦιʔεͷঢ়ଶΛએݴ͞Εͨঢ়ଶʹௐ੔͢ΔϓϩάϥϜ ,VCFSOFUFT"1*Λհͯ͠Ϧιʔεͷૢ࡞΍೚ҙͷॲཧΛߦ͏ ྫʣ3FQMJDB4FU$POUSPMMFSɿࢦఆ͞ΕͨϨϓϦΧ਺Ͱ1PEΛҡ࣋ ,VCFSOFUFTΫϥελ 3FQMJDB4FU$POUSPMMFS kind: ReplicaSet spec: replicas:

    3 template: spec: containers: - name: app image: nginx:1.19 kind: Pod spec: containers: - name: app image: nginx:1.19 kind: Pod spec: containers: - name: app image: nginx:1.19 kind: Pod spec: containers: - name: app image: nginx:1.19 XBUDI $SFBUF1PE %FMFUF1PE 3FQMJDB4FUͷఆٛΛݩʹ 1PEͷ࡞੒࡟আΛߦ͍ɺ ϨϓϦΧ਺Λҡ࣋ XBUDI
  5. 3FQMJDB4FU$POUSPMMFSͷ࣮૷ RS=$(kubectl -n NAMESPACE get rs NAME -o json) FPods=$(kubectl

    get pods -l SELECTOR | wc -l) DIFF=$(( $FPods – RS.SPEC.REPLICAS )) if $FPods -lt 0; then kubectl -n NAMESPACE create pod –f RS.SPEC.TEMPLATE else if $FPods –gt 0; then kubectl -n NAMESPACE delete pods TARGET_POD fi rs, err := rsc.rsLister.ReplicaSets(namespace).Get(name) allPods, err := rsc.podLister.Pods(rs.Namespace).List(labels.Everything()) filteredPods := controller.FilterActivePods(allPods) filteredPods, err = rsc.claimPods(rs, selector, filteredPods) diff := len(filteredPods) - int(*(rs.Spec.Replicas)) if diff < 0 { rsc.podControl.CreatePodsWithControllerRef( rs.Namespace, &rs.Spec.Template, ... ) } else if diff > 0 { rsc.podControl.DeletePod(rs.Namespace, targetPod.Name, rs) } https://github.com/kubernetes/kubernetes/blob/release-1.21/pkg/controller/replicaset/replica_set.go ΑΓվมͯ͠ൈਮ 3FQMJDB4FU$POUSPMMFS XBUDI $SFBUF1PE %FMFUF1PE ,VCFSOFUFTΫϥελ (P 4IFMM 3FDPODJMJBUJPO-PPQ
  6. $VTUPN3FTPVSDF ಛఆͷϦιʔεͷఆٛʹԠͯ͡ɺ೚ҙͷॲཧΛߦ͏$POUSPMMFS ɹɹ೚ҙͷϑΟʔϧυΛ࣋ͭ৽ͨͳϦιʔεΛಠࣗʹఆٛՄೳ kind: MySQL spec: replicas: 3 version: 8.0.23

    kind: StatefulSet spec: {...(লུ)...} kind: Service spec: {...(লུ)...} ,VCFSOFUFTΫϥελ .Z42-$POUSPMMFS XBUDI .BOBHF 4UBUFGVM4FU4FSWJDF
  7. "ENJTTJPO8FCIPPL ,VCFSOFUFT"1*ʹϦΫΤετ͕ೖ͖ͬͯͨλΠϛϯάͰ ϦιʔεͷݕূɾมߋΛ8FCIPPLͰ࣮ߦ͢Δ࢓૊Έ kind: Pod spec: containers: - name: app

    image: nginx:1.19 kind: Pod spec: containers: - name: app image: nginx:1.19 - name: sidecar image: envoy:v1.21.0 .VUBUJOH kind: Pod spec: containers: - name: app image: nginx:latest kind: Pod spec: containers: - name: app image: nginx:1.19 7BMJEBUJOH
  8. ৽ͨʹఆٛͨ͠$VTUPN3FTPVSDF ʙ$MVTUFS"1*ʙ ؅ཧΫϥελʹσϓϩΠ͞Εͨ$MVTUFSϦιʔεΛݩʹ ,VCFSOFUFTΫϥελΛߏஙɾ؅ཧ͢Δ࢓૊Έ ؅ཧΫϥελ W ϢʔβʔΫϥελ W ϢʔβʔΫϥελ kind:

    Cluster metadata: name: test-cluster-01 spec: topology: version: v1.20.7 controlPlane: replicas: 3 workers: machineDeployments: - class: type1-workers name: cpu-nodepool-01 replicas: 3 kind: Cluster metadata: name: test-cluster-01 spec: topology: version: v1.20.7 controlPlane: replicas: 3 workers: machineDeployments: - class: type1-workers name: cpu-nodepool-01 replicas: 3 CVJMENBOBHF CVJMENBOBHF ˞ݫີʹ͸0QFO4UBDL$MVTUFS΍0QFO4UBDL.BDIJOFΛར༻ɻ
  9. $MVTUFS"1*͕ੜ੒͢Δೝূ৘ใΛ΋ͱʹ "SHP$%༻ͷΫϥελೝূ৘ใͷ4FDSFUϦιʔεΛੜ੒ ؅ཧΫϥελ W ϢʔβʔΫϥελ W ϢʔβʔΫϥελ NBOBHFBEEPOT NBOBHFBEEPOT "SHP$%

    kind: Cluster metadata: name: test-cluster-01 spec: {...(লུ)...} kind: Secret metadata: name: test-cluster-01 namespace: argocd labels: argocd.argoproj.io/secret-type: cluster type: Opaque stringData: name: test-cluster-01 server: https://10.0.0.1:6443 clusterResources: true config: ʢArgo CD ಛ༗ͷkubeconfig૬౰ͷϑΝΠϧʣ kind: Cluster metadata: name: test-cluster-01 spec: {...(লུ)...} ",&"EEPO.BOBHFS طଘͷ$VTUPN3FTPVSDFʹର͢Δ$POUSPMMFS ʙ"SHP$%$MVTUFS4FDSFUͷੜ੒ʙ kind: Secret metadata: name: test-cluster-01 namespace: argocd labels: argocd.argoproj.io/secret-type: cluster type: Opaque stringData: name: test-cluster-01 server: https://10.0.0.1:6443 clusterResources: true config: ʢArgo CD ಛ༗ͷkubeconfig૬౰ͷϑΝΠϧʣ
  10. #VJMEJO3FTPVSDFʹର͢Δ$POUSPMMFS ʙ*OHSFTT$POUSPMMFSʙ ʮ*OHSFTTϦιʔεͷઃఆมߋʯʮ/PEFͷঢ়ଶ΍਺ͷมߋʯ Λ΋ͱʹɺϓϥΠϕʔτΫϥ΢υ্ͷϩʔυόϥϯαΛૢ࡞ *OHSFTT$POUSPMMFS kind: Ingress metadata: name: my-http-loadbalancer

    spec: {...(লུ)...} XBUDI *OHSFTTͷઃఆ΍/PEFͷঢ়ଶΛ΋ͱʹ ϩʔυόϥϯαͷઃఆΛมߋ ϩʔυόϥϯα"1* ʢϓϥΠϕʔτΫϥ΢υ"1*ʣ ,VCFSOFUFTΫϥελ kind: Node metadata: name: node01 kind: Node metadata: name: node01 kind: Node metadata: name: node01 XBUDI
  11. ಛఆͷΞϓϦέʔγϣϯͷந৅Խ ΞϓϦέʔγϣϯͷઃఆ஋Λந৅Խͨ͠ϦιʔεΛఆٛ ɹࢀߟɿ0QFSBUPS8IJUFQBQFS FH.Z42-0QFSBUPS kind: MySQL metadata: name: sample spec:

    replicas: 3 version: 8.0.23 password: somepass kind: StatefulSet metadata: name: sample spec: replicas: 3 template: spec: containers: - name: mysql image: mysql:8.0.23 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: sample key: MYSQL_ROOT_PASSWORD kind: Service spec: {...(লུ)...} kind: Secret metadata: name: sample stringData: MYSQL_ROOT_PASSWORD : somepass ग़యɿIUUQTHJUIVCDPNDODGUBHBQQEFMJWFSZCMPCNBJOPQFSBUPSXHXIJUFQBQFS $POUSPMMFS
  12. ֎෦Ϧιʔεͷঢ়ଶఆٛ ֎෦ϦιʔεΛૢ࡞͢ΔͨΊͷઃఆΛఆٛ ɹFH"$.&Λར༻ͨ͠ূ໌ॻൃߦͷઃఆʢ$FSU.BOBHFSʣ %/4 "$.&%/4$IBMMFOHF *OHSFTT "$.&)551$IBMMFOHF $POUSPMMFS kind: Certificate

    metadata: name: tls-sample spec: dnsNames: - sample.example.com issuerRef: group: cert-manager.io kind: ClusterIssuer name: letsencrypt-production secretName: tls-sample-secret kind: ClusterIssuer metadata: name: letsencrypt-production spec: acme: server: https://acme-v02... solvers: - dns01: route53: {...}
  13. ෳࡶͳఆٛΛѻ͍΍͍͢ܗʹ·ͱΊΔ ෳࡶԽ͢ΔઃఆΛ$VTUPN3FTPVSDFͱͯ͠੾Γग़ͯ͠ར༻ ɹFH4JEFDBSͷઃఆΛ$VTUPN3FTPVSDFͰઃఆՄೳʹ͢Δ kind: SidecarProfile spec: selector: inject-envoy: true containers:

    - image: envoy env: - name: MY_KEY Value: MY_VALUE kind: Pod metadata: name: my-pod annotations: sidecar/profile: inject-envoy spec: {...(লུ)...} kind: SidecarProfile metadata: name: inject-envoy spec: containers: - image: envoy env: - name: MY_KEY Value: MY_VALUE kind: Pod metadata: name: my-pod labels: inject-envoy: true spec: {...(লུ)...} kind: Pod metadata: name: my-pod spec: containers: - {...(লུ)...} - image: envoy env: - name: MY_KEY value: MY_VALUE 03 ˞આ໌ྫͰ͢
  14. $VTUPN3FTPVSDFͷσʔλετΞར༻ $VTUPN3FTPVSDFΛεΩʔϚΛ࣋ͬͨ؆қతͳσʔλετΞͱͯ͠ར༻ *NQFSTPOBUF6TFSSFRVFTUTʹΑΓɺ,VCFSOFUFT3#"$ͷݖݶൣғͰϦΫΤετΛॲཧ 8FC$POTPMF (16BB4"1*4FSWFS *NBHF$VTUPN3FTPVSDFͷ$36%ʢ*NQFSTPOBUF6TFSSFRVFTUTʣ apiVersion: ml.cyberagent.io/v1 kind: Image

    metadata: name: asia.gcr.io-GCP_PROJECT-base-notebook-notebook-6.1.5 spec: name: asia.gcr.io/GCP_PROJECT/base-notebook tag: notebook-6.1.5 enable: true imagePullSecretRef: gcr-GCP_PROJECT apiVersion: ml.cyberagent.io/v1 kind: Image metadata: name: asia.gcr.io-GCP_PROJECT-base-notebook-notebook-6.1.5 spec: name: asia.gcr.io/GCP_PROJECT/base-notebook tag: notebook-6.1.5 enable: true imagePullSecretRef: gcr-GCP_PROJECT
  15. 3FDPODJMFؔ਺ʹΑΔௐ੔ 0QFO"1*W4DIFNB7BMJEBUJPO ه࿥ ௐ੔ ݕূɾมߋ $&-&YQSFTTJPO 7BMJEBUJOH8FCIPPL .VUBUJOH8FCIPPL $POWFSTJPO8FCIPPL ݕূɾมߋ

    $POUSPMMFSىಈ࣌ 4ZOD1FSJPEͷ࣮ߦ࣌ Ϧιʔεͷมߋ࣌ Πϕϯτൃੜ࣌ ࠶࣮ߦ࣌ ࣮ߦ੍ޚ ه࿥ ௐ੔ ࣮ߦ੍ޚ ϦΫΤετΛड͚෇͚ͯ ϦιʔεΛొ࿥͢Δ·Ͱͷલॲཧ Ϧιʔεొ࿥ޙɺ࣮ࡍʹॲཧ͢Δ಺༰ ॲཧޙͷ݁ՌΛه࿥͢Δ ͋Δ΂͖ঢ়ଶʹऩଋͤ͞ଓ͚Δ࣮ߦ੍ޚ 'JOBMJ[FST 0XOFS3FGFSFODFT ผϦιʔεͷ࡞੒ɾ࡟আ $VTUPN.FUSJDTͷߋ৽ 4UBUVTϑΟʔϧυͷมߋ "OOPUBUJPOTͷมߋ &WFOUϦιʔεͷ࡞੒ ϩάग़ྗ /PUJpDBUJPO&OHJOF ֎෦௨஌
  16. 0QFO"1*W4DIFNB7BMJEBUJPO ه࿥ ௐ੔ ݕূɾมߋ $&-&YQSFTTJPO 7BMJEBUJOH8FCIPPL .VUBUJOH8FCIPPL $POWFSTJPO8FCIPPL ݕূɾมߋ $POUSPMMFSىಈ࣌

    4ZOD1FSJPEͷ࣮ߦ࣌ Ϧιʔεͷมߋ࣌ Πϕϯτൃੜ࣌ ࠶࣮ߦ࣌ ࣮ߦ੍ޚ ه࿥ ௐ੔ ࣮ߦ੍ޚ ϦΫΤετΛड͚෇͚ͯ ϦιʔεΛొ࿥͢Δ·Ͱͷલॲཧ Ϧιʔεొ࿥ޙɺ࣮ࡍʹॲཧ͢Δ಺༰ ॲཧޙͷ݁ՌΛه࿥͢Δ ͋Δ΂͖ঢ়ଶʹऩଋͤ͞ଓ͚Δ࣮ߦ੍ޚ 3FDPODJMFؔ਺ʹΑΔௐ੔ 'JOBMJ[FST 0XOFS3FGFSFODFT ผϦιʔεͷ࡞੒ɾ࡟আ $VTUPN.FUSJDTͷߋ৽ 4UBUVTϑΟʔϧυͷมߋ "OOPUBUJPOTͷมߋ &WFOUϦιʔεͷ࡞੒ ϩάग़ྗ /PUJpDBUJPO&OHJOF ֎෦௨஌
  17. $VTUPN3FTPVSDFͷ7BMJEBUJPO $VTUPN3FTPVSDFʹର͢Δ؆қతͳ7BMJEBUJPO w 0QFO"1*Wͷ4DIFNB7BMJEBUJPO w $&-ʢ$PNNPO&YQSFTTJPO-BOHVBHFʣ7BMJEBUJPOBMQIB  ग़యɿIUUQTTXBHHFSJPTQFDJpDBUJPOTDIFNBPCKFDU ग़యɿIUUQTUPPMTJFUGPSHIUNMESBGUXSJHIUKTPOTDIFNBWBMJEBUJPO ग़యɿIUUQTPQFOTPVSDFHPPHMFQSPKFDUTDFM

    ग़యɿIUUQTHJUIVCDPNLVCFSOFUFTFOIBODFNFOUTJTTVFT openAPIV3Schema: type: object properties: spec: type: object properties: cronSpec: type: string pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$' replicas: type: integer minimum: 0 maxReplicas: type: integer minimum: 1 x-kubernetes-validation-rules: - rule: "self.replicas <= self.maxReplicas" message: "replicas should be smaller than or equal to maxReplicas." e.g.ʣ spec: cronSpec: "* * * * */5" replicas: 10 maxReplicas: 20
  18. 7BMJEBUJOH8FCIPPL γεςϜͰར༻͢Δ"OOPUBUJPOTʹ੍໿Λ͔͚Δ w FH֘౰ͷ",&7FSTJPO͕ଘࡏ͢Δ͔ w FH,VCFSOFUFT7FSTJPO͕",&7FSTJPOʹରԠ͍ͯ͠Δ͔ w FH$POUSPM1MBOFͷόʔδϣϯ8PSLFSͷόʔδϣϯҎ্͕੒ཱ͍ͯ͠Δ͔ kind: Cluster

    metadata: name: test-cluster-01 metadata: annotations: ake.cycloud.io/version: v1.23-ake.220201 spec: topology: version: v1.23.1 controlPlane: replicas: 3 workers: machineDeployments: - class: type1-workers name: cpu-nodepool-01 replicas: 3 ಛఆͷϑΟʔϧυʹରͯ͠ ೚ҙͷνΣοΫ 7BMJEBUJOH ˞(BUFLFFQFS 3FHPͰ΋୅ସՄೳ
  19. .VUBUJOH8FCIPPL طଘͷϑΟʔϧυͷઃఆ஋Λ΋ͱʹϑΟʔϧυΛઃఆ w FH,VCFSOFUFTόʔδϣϯʹରԠͨ͠ϚΠφʔϦϦʔε"OOPUBUJPOΛ෇༩ kind: Cluster metadata: name: test-cluster-01 metadata:

    annotations: ake.cycloud.io/kubernetes-release: v1.23 spec: topology: version: v1.23.1 controlPlane: replicas: 3 workers: machineDeployments: - class: type1-workers name: cpu-nodepool-01 replicas: 3 ಛఆͷϑΟʔϧυΛ௥Ճɾमਖ਼ .VUBUJOH ˞(BUFLFFQFSͰ΋͋Δఔ౓͸୅ସՄೳ
  20. $POWFSTJPO8FCIPPL $VTUPN3FTPVSDFͷ"1*7FSTJPO௥ैͷରԠ εΩʔϚͷมߋʹ൐͍૬ޓʹม׵Λߦ͏8FCIPPLTFSWFS ,VCFSOFUFT಺෦Ͱ͸ͲͪΒ͔ҰํͷܗࣜͰอ࣋ apiVersion: example.com/v1 kind: CronJob annotations: example.com/suspend:

    true spec: schedule: "*/1 * * * *" apiVersion: example.com/v2 kind: CronJob spec: schedule: minute: "*/1" hour: "*" dayOfMonth: "*" month: "*" dayOfWeek: "*" suspend: true $POWFSTJPO
  21. 0QFO"1*W4DIFNB ه࿥ ௐ੔ ݕূɾมߋ $&-&YQSFTTJPO 7BMJEBUJOH8FCIPPL .VUBUJOH8FCIPPL $POWFSTJPO8FCIPPL ݕূɾมߋ $POUSPMMFSىಈ࣌

    4ZOD1FSJPEͷ࣮ߦ࣌ Ϧιʔεͷมߋ࣌ Πϕϯτൃੜ࣌ ࠶࣮ߦ࣌ ࣮ߦ੍ޚ ه࿥ ௐ੔ ࣮ߦ੍ޚ ϦΫΤετΛड͚෇͚ͯ ϦιʔεΛొ࿥͢Δ·Ͱͷલॲཧ Ϧιʔεొ࿥ޙɺ࣮ࡍʹॲཧ͢Δ಺༰ ॲཧޙͷ݁ՌΛه࿥͢Δ ͋Δ΂͖ঢ়ଶʹऩଋͤ͞ଓ͚Δ࣮ߦ੍ޚ 3FDPODJMFؔ਺ʹΑΔௐ੔ 'JOBMJ[FST 0XOFS3FGFSFODFT ผϦιʔεͷ࡞੒ɾ࡟আ $VTUPN.FUSJDTͷߋ৽ 4UBUVTϑΟʔϧυͷมߋ "OOPUBUJPOTͷมߋ &WFOUϦιʔεͷ࡞੒ ϩάग़ྗ /PUJpDBUJPO&OHJOF ֎෦௨஌
  22. 3FDPODJMFؔ਺ͱ8PSLRVFVF 0CKFDU͝ͱʹ3FDPODJMFؔ਺Λ࣮ߦ͠ɺએݴ͞Εͨঢ়ଶ΁ௐ੔ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result,

    error) { // Reconcile ର৅ͷ Object ͷऔಘ obj := &clusterv1.Cluster{} err := r.Get(ctx, req.NamespacedName, obj) if err != nil { return ctrl.Result{}, client.IgnoreNotFound(err) } ...ʢႈ౳ʹͳΔΑ͏ͳॲཧʣ... return ctrl.Result{}, nil } 8PSLRVFVF 3FDPODJMF ग़యɿIUUQTQLHHPEFWTJHTLTJPDPOUSPMMFSSVOUJNFQLHSFDPODJMF3FRVFTU ग़యɿIUUQTHJUIVCDPNLVCFSOFUFTTBNQMFDPOUSPMMFSCMPCNBTUFSEPDTDPOUSPMMFSDMJFOUHPNE NamespacedName { Namespace: default Name: my-res }
  23. err := r.Get(ctx, req.NamespacedName, obj) if apierrors.IsNotFound(err) { // already

    deleted return ctrl.Result{}, nil } if !obj.ObjectMeta.DeletionTimestamp.IsZero() { // delete in progress } w ࡟আࡁΈɾ࡟আதͷ൑ఆ ࡟আΠϕϯτͷऔΓѻ͍ 3FDPODJMF ؔ਺ʹ͸$3&"5&61%"5&%&-&5& ͢΂ͯͷΠϕϯτ͕౉ͬͯ͘Δ ˞ࢠϦιʔε࡞੒࣌ʹ$SFBUF͢Δ͔6QEBUF͢Δ͔ͷ ൑ผ࣌ʹ΋ར༻Մೳͳύλʔϯ ˞ ݫີʹ͸'JMUFSJOHͯ͠%&-&5&ΠϕϯτΛࣺͯΔ͜ͱ΋Մೳ
  24. 0XOFS3FGFSFODFT ಛఆͷϦιʔεʹґଘ͢ΔϦιʔεͷ࡞੒࣌ʹ਌ࢠؔ܎Λఆٛ ɹ਌Ϧιʔε͕࡟আ͞Εͨ৔߹ɺࢠϦιʔε͸࡟আ w FH3FQMJDB4FUϦιʔεͱ1PEϦιʔε w FH$MVTUFSϦιʔεͱͦΕΛݩʹ࡞੒ͨ͠4FDSFU "SHP$%޲͚$MVTUFS઀ଓ༻4FDSFU kind: Cluster

    metadata: name: test-cluster-01 spec: {...(লུ)...} 0XOFS kind: Secret metadata: name: test-cluster-01 namespace: argocd labels: argocd.argoproj.io/secret-type: cluster ownerReferences: - kind: Cluster name: test-cluster-01 blockOwnerDeletion: true controller: true type: Opaque stringData: name: test-cluster-01 server: https://10.0.0.1:6443 clusterResources: true config: ʢArgo CD ಛ༗ͷkubeconfig૬౰ͷϑΝΠϧʣ $IJME
  25. 0XOFS3FGFSFODFTͷ෇༩ؔ਺ setControllerReference(owner, child, scheme) w $IJME͸୯Ұͷ0XOFSΛ࣋ͭΑ͏ʹ੍ݶ w DPOUSPMMFSUSVF w 0XOFSϦιʔε͸$IJMEϦιʔεͷ࡟আ׬ྃΛ଴ػ

    w CMPDL0XOFS%FMFUJPOGBMTF 0XOFS $IJME setOwnerReference(owner, child, scheme) w શ0XOFSϦιʔε࡟আޙʹ$IJMEϦιʔε͸ඇಉظʹ࡟আ w CMPDL0XOFS%FMFUJPOGBMTF 0XOFS 0XOFS $IJME 0XOFS 0XOFS $IJME 0XOFS $IJME ग़యɿIUUQTQLHHPEFWTJHTLTJPDPOUSPMMFSSVOUJNFQLHDPOUSPMMFSDPOUSPMMFSVUJM metadata: ownerReferences: - kind: Cluster name: test-cluster-01 blockOwnerDeletion: true controller: true 0XOFS $IJME
  26. 'JOBMJ[FST Ϧιʔεͷ࡟আલʹɺಛఆͷॲཧͷ׬ྃ·Ͱ଴ػ͢Δ࢓૊Έ w 4FSWJDF࡟আ࣌ʹ֎෦ͷϩʔυόϥϯαʔΛ࡟আ͢Δ w $MVTUFSϦιʔε࡟আ࣌ʹ"SHP$%ͷ"QQ1SPKFDU͔ΒΫϥελϨίʔυΛ࡟আ͢Δ "QQ1SPKFDU ࡟আࡁ $MVTUFS" $MVTUFS#

    $MVTUFS" "QQ1SPKFDU ࡟আલ $MVTUFS" $MVTUFS# ࡟আΠϕϯτ 8PSLRVFVF $MVTUFS" "QQ1SPKFDU ࡟আࡁ $MVTUFS" $MVTUFS# 3FDPODJMF $MVTUFS" ࡟আॲཧ "QQ1SPKFDU ࡟আࡁ $MVTUFS" $MVTUFS# $MVTUFS" 'BJMBOE3FRVFVF $POUSPMMFS࠶ىಈʢϝϞϦ্ͷ8PSLRVFVF͸ΫϦΞʣ ࡟আ͞ΕͨΠϕϯτ͸ফࣦ 3FDPODJMFؔ਺͸ೋ౓ͱ࣮ߦ͞Εͳ͍ ॲཧͷྲྀΕ
  27. 'JOBMJ[FST Ϧιʔεͷ࡟আલʹɺಛఆͷॲཧͷ׬ྃ·Ͱ଴ػ͢Δ࢓૊Έ w 4FSWJDF࡟আ࣌ʹ֎෦ͷϩʔυόϥϯαʔΛ࡟আ͢Δ w $MVTUFSϦιʔε࡟আ࣌ʹ"SHP$%ͷ"QQ1SPKFDU͔ΒΫϥελϨίʔυΛ࡟আ͢Δ ॲཧͷྲྀΕ "QQ1SPKFDU ࡟আ଴ػ $MVTUFS"

    $MVTUFS# $MVTUFS" "QQ1SPKFDU ࡟আલ $MVTUFS" $MVTUFS# ࡟আΠϕϯτ 8PSLRVFVF $MVTUFS" "QQ1SPKFDU ࡟আ଴ػ $MVTUFS" $MVTUFS# 3FDPODJMF $MVTUFS" ࡟আॲཧ "QQ1SPKFDU ࡟আ଴ػ $MVTUFS" $MVTUFS# $MVTUFS" 'BJMBOE3FRVFVF $POUSPMMFS࠶ىಈʢϝϞϦ্ͷ8PSLRVFVF͸ΫϦΞʣ ࠶౓࡟আΠϕϯτΛੜ੒ ࡟আΠϕϯτ
  28. ʢࢀߟʣ'JOBMJ[FSTͷ࣮૷ྫ Ϧιʔε͕࡟আத͔͸NFUBEBUBEFMFUJPO5JNFTUBNQΛ΋ͱʹ൑அ if obj.ObjectMeta.DeletionTimestamp.IsZero() { if !controllerutil.ContainsFinalizer(obj, "ake.cycloud.io/unregister-argocd"){ controllerutil.AddFinalizer(obj, "ake.cycloud.io/unregister-argocd")

    if err := r.Update(ctx, obj); err != nil { return ctrl.Result{}. err } } } else { if controllerutil.ContainsFinalizer(obj, "ake.cycloud.io/unregister-argocd"){ if err :=ʢ࡟আલͷґଘॲཧʣ; err != nil { return ctrl.Result{}, err } controllerutil.RemoveFinalizer(obj, "ake.cycloud.io/unregister-argocd") if err := r.Update(ctx, obj); err != nil { return ctrl.Result{}. err } } return ctrl.Result{}, nil } ग़యɿIUUQTQLHHPEFWTJHTLTJPDPOUSPMMFSSVOUJNFQLHDPOUSPMMFSDPOUSPMMFSVUJM
  29. 0QFO"1*W4DIFNB ه࿥ ผϦιʔεͷ࡞੒ɾ࡟আ $VTUPN.FUSJDTͷߋ৽ 4UBUVTϑΟʔϧυͷมߋ "OOPUBUJPOTͷมߋ &WFOUϦιʔεͷ࡞੒ ϩάग़ྗ ௐ੔ ݕূɾมߋ

    $&-&YQSFTTJPO 7BMJEBUJOH8FCIPPL .VUBUJOH8FCIPPL $POWFSTJPO8FCIPPL ݕূɾมߋ $POUSPMMFSىಈ࣌ 4ZOD1FSJPEͷ࣮ߦ࣌ Ϧιʔεͷมߋ࣌ Πϕϯτൃੜ࣌ ࠶࣮ߦ࣌ ࣮ߦ੍ޚ ه࿥ ௐ੔ ࣮ߦ੍ޚ ϦΫΤετΛड͚෇͚ͯ ϦιʔεΛొ࿥͢Δ·Ͱͷલॲཧ Ϧιʔεొ࿥ޙɺ࣮ࡍʹॲཧ͢Δ಺༰ ॲཧޙͷ݁ՌΛه࿥͢Δ ͋Δ΂͖ঢ়ଶʹऩଋͤ͞ଓ͚Δ࣮ߦ੍ޚ 3FDPODJMFؔ਺ʹΑΔௐ੔ 'JOBMJ[FST 0XOFS3FGFSFODFT /PUJpDBUJPO&OHJOF ֎෦௨஌
  30. ผϦιʔεͷ࡞੒ɾ࡟আ $POUSPMMFS͕3FDPODJMFதʹผϦιʔεΛ࡞੒ ɹ㱺࡞੒͞ΕͨϦιʔεͷঢ়ଶࣗମ͕ه࿥ kind: MySQL metadata: name: sample spec: replicas:

    3 version: 8.0.23 password: somepass kind: StatefulSet metadata: name: sample spec: replicas: 3 template: spec: containers: - name: mysql image: mysql:8.0.23 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: sample key: MYSQL_ROOT_PASSWORD kind: Service spec: {...(লུ)...} kind: Secret metadata: name: sample stringData: MYSQL_ROOT_PASSWORD : somepass $POUSPMMFS
  31. 4UBUVTϑΟʔϧυͷมߋ 3FTPVSDFͷTUBUVTϑΟʔϧυͷߋ৽ʢ#VJMEJOSFTPVSDFؚΉʣ ɹશମͷঢ়گɺ઀ଓ৘ใɺFUD kind: MySQL metadata: name: sample spec: replicas:

    3 version: 8.0.23 password: somepass status: primaryNode: sample-mysql-0 updatedNodes: 2 updateStatus: - node: sample-mysql-0 version: 8.0.11 - node: sample-mysql-1 version: 8.0.23 - node: sample-mysql-2 version: 8.0.23 connectionInfo: ipAddress: x.x.x.x username: mysql Password: somepass conditions: [] obj.Status = api.MySQLStatus{...} err := r.Status().Update(context.TODO(), obj) // +kubebuilder:rbac:groups=db.example.com, resources=mysql/status, verbs=get;update;patch w ϚʔΧʔͷઃஔ w 4UBUVTϑΟʔϧυͷߋ৽
  32. "OOPUBUJPOTͷมߋ 4UBUVTϑΟʔϧυͷ୅ΘΓʹ"OOPUBUJPOTΛར༻ w ؔ࿈͢Δ༷ʑͳϦιʔεʹ෇༩Մೳ kind: MySQL metadata: name: sample annotations:

    hoge: fuga spec: replicas: 3 version: 8.0.23 password: somepass kind: StatefulSet metadata: name: sample annotations: hoge: fuga spec: replicas: 3 template: spec: containers: - name: mysql image: mysql:8.0.23 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: sample key: MYSQL_ROOT_PASSWORD kind: Secret metadata: name: sample annotations: hoge: fuga stringData: MYSQL_ROOT_PASSWORD : somepass $POUSPMMFS
  33. &WFOUϦιʔεͷ࡞੒ ಛఆͷΠϕϯτ͝ͱʹ&WFOUϦιʔεΛൃߦ  LVCFDUMίϚϯυͰ֬ೝ  PQTHFOJFLVCFSOFUFTFWFOUFYQPSUFSͰ௨஌ Recorder.Eventf( &obj, # Object

    corev1.EventTypeWarning, # EventType "SomethingFailed", # Reason ”resource %s has error", # message obj.Name. # message args ) &WFOU 8BSOJOH ௚ۙ࣌ؒͷ&WFOUϦιʔεͷΈอ࣋ʢσϑΥϧτʣ var Recorder = mgr.GetEventRecorderFor("my-cont") ग़యɿIUUQTQLHHPEFWLTJPDMJFOUHPUPPMTSFDPSE&WFOU3FDPSEFS ग़యɿIUUQTQLHHPEFWLTJPBQJDPSFWQLHDPOTUBOUT w .BOBHFS͔Β&WFOU3FDPSEFSͷऔಘ w &WFOUϦιʔεͷ࡞੒ kubectl get events kubectl describe RESOURCE &WFOU 8BSOJOH 3FTPVSDF &WFOU /PSNBM &WFOU /PSNBM ʜ &WFOU 8BSOJOH &WFOUϦιʔε͸ͭͷϦιʔεʹඥ෇͚
  34. $VTUPN.FUSJDTͷߋ৽ DPOUSPMMFSSVOUJNFͰ͸.BOBHFS͕NFUSJDTFOEQPJOUΛอ࣋ɾެ։ ొ࿥ͨ͠ϝτϦΫε͸3FDPODJMF ؔ਺͔Βߋ৽ ग़యɿIUUQTQLHHPEFWHJUIVCDPNQSPNFUIFVTDMJFOU@HPMBOHQSPNFUIFVT ग़యɿIUUQTQLHHPEFWTJHTLTJPDPOUSPMMFSSVOUJNF!WQLHNFUSJDT .BOBHFS NFUSJDT 4DSBQF SFHJTUFS

    SFHJTUFS .Z$VTUPN.FUSJDT" .Z$VTUPN.FUSJDT# 6QEBUFGSPN3FDPODJMF var MyCustomMetricsA = prometheus.NewGaugeVec( prometheus.GaugeOpts{ Name: "my_custom_metrics_a", Help: "my custom metrics related A", }, []string{ "resource_name", "resource_namespace" }, ) w ϝτϦΫεͷఆٛ MyCustomMetricsA.With( prometheus.Labels{ "resource_name": obj.Name, "resource_namespace": obj.Namespace, }).Set(0.5) w 3FDPODJMF ؔ਺Ͱͷߋ৽ func init() { metrics.Registry.MustRegister(MyCustomMetricsA) } w .BOBHFS΁ͷొ࿥
  35. ֎෦௨஌ 3FTPVSDFʹ௨஌ઃఆ༻ͷϑΟʔϧυΛ༻ҙ͠ɺͦͷ৘ใΛ༻͍ͯ֎෦௨஌Λߦ͏ w 4QFDϑΟʔϧυɺ"OOPUBUJPOT w ࣮ࡍʹ͸5PLFO͸4FDSFUͱͯ࣋ͪ͠ɺ0CKFDU3FGFSFODFͳͲͰࢀর͕ਪ঑ w IUUQTHJUIVCDPNLVCFSOFUFTLVCFSOFUFTCMPCSFMFBTFQLHBQJTDPSFUZQFTHP-- ॲཧ͕࣌ؒ௕͍৔߹͸ɺඇಉظԽͳͲ΋ݕ౼ func

    (r *MyReconciler) Reconcile(...) (...) { ... api := slack.New(obj.Spec.Notification.Slack.Token) _, _, err := api.PostMessage( obj.Spec.Notification.Slack.Channel, slack.MsgOptionText("Some message", false) ) ... } kind: MyCustomResource metadata: name: sample spec: notification: slack: token: xxx channel: my-channel ... ग़యɿIUUQTQLHHPEFWHJUIVCDPNTMBDLHPTMBDL
  36. /PUJpDBUJPO&OHJOF Ϧιʔεͷঢ়ଶʹԠͯ͡௨஌Λߦ͏"SHP1SPKFDUൃ঵ͷϥΠϒϥϦʢ4MBDLɺ5FBNTɺ&NBJMɺFUDʣ ग़యɿIUUQTHJUIVCDPNBSHPQSPKOPUJpDBUJPOTFOHJOF ग़యɿIUUQT[FOOEFWLTQFSBSUJDMFTCFGCC !,,BXBCF kind: ConfigMap metadata: name: cert-manager-notifications-cm

    data: trigger.on-cert-ready: | - when: any(cert.status.conditions, {.reason == 'Ready' && .status == 'True'}) send: [cert-ready] template.cert-ready: | message: | Certificate {{.cert.metadata.name}} is ready! ௨஌ઃఆͷ࡞੒ w ௨஌৚݅ʢUSJHHFS ʣ w ௨஌ϑΥʔϚοτʢUFNQMBUF ʣ w ௨஌ઌʢTFSWJDF ʣ ௨஌ର৅ͷઃఆ ར༻ kind: Certificate metadata: annotations: notifications.argoproj.io/subscribe.on-cert-ready.slack: my-channel spec: {...} ৄ͘͠͸
  37. ϩάग़ྗ ىಈ࣌ʹ ctrl.SetLogger()Ͱొ࿥ 3FDPODJMFؔ਺಺Ͱ͸log.FromContext()Ͱऔಘͯ͠ར༻ ˞LVCFCVJMEFSར༻࣌͸ొ࿥ࡁΈ import "sigs.k8s.io/controller-runtime/pkg/log" func (r *Reconciler)

    Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { log := log.FromContext(ctx) log.Error(err, "something went wrong..." "req", req.NamespacedName ) } ग़యɿIUUQTQLHHPEFWTJHTLTJPDPOUSPMMFSSVOUJNFQLHMPH
  38. 0QFO"1*W4DIFNB ه࿥ ௐ੔ ݕূɾมߋ $&-&YQSFTTJPO 7BMJEBUJOH8FCIPPL .VUBUJOH8FCIPPL $POWFSTJPO8FCIPPL ݕূɾมߋ $POUSPMMFSىಈ࣌

    4ZOD1FSJPEͷ࣮ߦ࣌ Ϧιʔεͷมߋ࣌ Πϕϯτൃੜ࣌ ࠶࣮ߦ࣌ ࣮ߦ੍ޚ ه࿥ ௐ੔ ࣮ߦ੍ޚ ϦΫΤετΛड͚෇͚ͯ ϦιʔεΛొ࿥͢Δ·Ͱͷલॲཧ Ϧιʔεొ࿥ޙɺ࣮ࡍʹॲཧ͢Δ಺༰ ॲཧޙͷ݁ՌΛه࿥͢Δ ͋Δ΂͖ঢ়ଶʹऩଋͤ͞ଓ͚Δ࣮ߦ੍ޚ 3FDPODJMFؔ਺ʹΑΔௐ੔ 'JOBMJ[FST 0XOFS3FGFSFODFT ผϦιʔεͷ࡞੒ɾ࡟আ $VTUPN.FUSJDTͷߋ৽ 4UBUVTϑΟʔϧυͷมߋ "OOPUBUJPOTͷมߋ &WFOUϦιʔεͷ࡞੒ ϩάग़ྗ /PUJpDBUJPO&OHJOF ֎෦௨஌
  39. Ϧιʔεͷมߋ࣌ 3FDPODJMF ؔ਺Λݺͼग़͢৚݅ΛࢦఆՄೳ 'PS  $POUSPMMFSͷ3FDPODJMFର৅ͷϦιʔεΛࢦఆ 0XOT  Ϧιʔε͕࡞੒͢ΔࢠϦιʔεΛࢦఆ 

     ࢠϦιʔεͷঢ়ଶมԽ࣌ʹ΋3FDPODJMF͞ΕΔ   ࣄલʹ0XOFS3FGFSFODFͷ෇༩͕ඞཁ 8JUI&WFOU'JMUFS 3FDPODJMFର৅ͷ&WFOUΛ੍ޚ func (r *MyDeploymentReconciler) SetupWithManager(...) error { return ctrl.NewControllerManagedBy(mgr). For(&myappsv1.MyDeployment{}). Owns(&appsv1.ReplicaSet{}). Complete(r) } ग़యɿIUUQTQLHHPEFWTJHTLTJPDPOUSPMMFSSVOUJNFQLHCVJMEFS
  40. Ϧιʔεͷมߋ࣌ 3FDPODJMFؔ਺ΛൃՐͤ͞ΔϦιʔεΛࢦఆ 'PS .Z%FQMPZNFOU"͕มߋ͞Εͨ৔߹  .Z%FQMPZNFOU"ʹରͯ͠ൃՐ 0XOT 3FQMJDB4FU#͕มߋ͞Εͨ৔߹  .Z%FQMPZNFOU#ʹରͯ͠ൃՐ

    func (r *MyDeploymentReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&myappsv1.MyDeployment{}). Owns(&appsv1.ReplicaSet{}). Complete(r) } 3FQMJDB4FU" 3FQMJDB4FU# .Z%FQMPZNFOU" .Z%FQMPZNFOU# PXOFS3FGFSFODF PXOFS3FGFSFODF 'PS 0XOT
  41. Ϧιʔεͷมߋ࣌ ,JOEɿ ಛఆͷLJOEͷϦιʔεͷมߋΠϕϯτ $IBOOFMɿ (PDIBOOFM΁ͷ௨஌Πϕϯτʢ֎෦ΠϕϯτͳͲʣ *OGPSNFSɿJOGPSNFSܦ༝ͰͷΠϕϯτ 4PVSDF )BOEMFS 8PSL2VFVFʹ௥Ճ͠ɺ3FDPODJMFͷ࣮ߦ଴ػ΁ &ORVFVF3FRVFTU'PS0CKFDUɿ

    ड͚औͬͨϦιʔεͷΠϕϯτΛൃՐ &ORVFVF3FRVFTU'PS0XOFSɿ ड͚औͬͨϦιʔεͷPXOFSͷΠϕϯτΛൃՐ &ORVFVF3FRVFTUT'SPN.BQ'VODɿ ड͚औͬͨϦιʔεͷΠϕϯτΛݩʹ    Ϛοϐϯά͢Δؔ਺ʹ͔͚ͯΠϕϯτΛൃՐ 8BUDI ؔ਺Λར༻͠ɺݺͼग़͢৚݅Λ͞ΒʹৄࡉࢦఆՄೳ ग़యɿIUUQTQLHHPEFWTJHTLTJPDPOUSPMMFSSVOUJNF!WQLHTPVSDF ग़యɿIUUQTQLHHPEFWTJHTLTJPDPOUSPMMFSSVOUJNF!WQLHIBOEMFS