Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Goでkubernetes operatorを実装してアプリのプレビュー環境を作る/go-kubernetes-operator

Goでkubernetes operatorを実装してアプリのプレビュー環境を作る/go-kubernetes-operator

fukuoka.go#17

22522e2bc35255ab75308c399ff782f6?s=128

Takuma Kume

June 10, 2021
Tweet

Transcript

  1. GVLVPLBHP (PͰLVCFSOFUFTPQFSBUPSΛ࣮૷ͯ͠ ΞϓϦͷϓϨϏϡʔ؀ڥΛ࡞Δ

  2. (.0ϖύϘגࣜձࣾ ϗεςΟϯάࣄۀ෦ 43&νʔϜ Ϋϥ΢υωΠςΟϒԽͷਪਐ ٱถ୓അ!UBLVNBLVNF

  3. ໨࣍ wlLVCFSOFUFTPQFSBUPSΛ࡞Δzͱ͸ wࠓճ࣮૷ͨ͠ιϑτ΢ΣΞͷ঺հ w։ൃܦҢ w(PʹΑΔ0QFSBUPS࣮૷ wॴײ

  4. zLVCFSOFUFTPQFSBUPSΛ࡞Δzͱ͸

  5. zLVCFSOFUFTPQFSBUPSΛ࡞Δzͱ͸ apiserver kubectl apply Control plane Data plane controller manager

    ReplicaSet Pod Pod ࢦఆ͞ΕͨPodͷ਺Λ อͱ͏ͱ͢Δ ReplicaSetͷྫ replicas:2 PodΛ1ͭ࡟আͯ͠΋ ίϯτϩʔϥʔ͕ݕ஌ͯ͠ ࠶࡞੒͞ΕΔ
  6. zLVCFSOFUFTPQFSBUPSΛ࡞Δzͱ͸ apiserver kubectl apply Control plane Data plane controller manager

    ReplicaSet Pod Pod ࢦఆ͞ΕͨPodͷ਺Λ อͱ͏ͱ͢Δ ReplicaSetͷྫ replicas:2 PodΛ1ͭ࡟আͯ͠΋ ίϯτϩʔϥʔ͕ݕ஌ͯ͠ ࠶࡞੒͞ΕΔ kubernetes͸ ͜ͷಈ͖Λ֦ுͰ͖Δ kubernetesΛ֦ு͢Δख๏ͷͻͱͭʹOperator͕͋Δ
  7. zLVCFSOFUFTPQFSBUPSΛ࡞Δzͱ͸ apiserver kubectl apply Control plane Data plane Custom Controller

    Custom Resource Custom Resource Definition (CRD) + ಠࣗͷϦιʔεఆٛ CRDͷఆٛʹج͍ͮͨ Ϧιʔε CRΛίϯτϩʔϧͯ͠ ఆٛ͞Εͨঢ়ଶʹอͭ kubernetes operator ͷ࣮ମ
  8. ࠓճ࣮૷ͨ͠΋ͷ

  9. apiVersion: service-expose.../v1alpha1 kind: ServiceExpose metadata: name: example namespace: ns1 spec:

    backend: service: name: example-svc port: number: 8080 domain: example.com path: / pathType: Prefix tlsEnable: true tlsSecretName: example-tls annotations: cert-manager.io/cluster-issuer: letsencrypt apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example namespace: ns1 annotations: cert-manager.io/cluster-issuer: letsencrypt spec: rules: - host: example-svc.ns1.example.com http: paths: - backend: service: name: example-svc port: number: 8080 path: / pathType: Prefix tls: - hosts: - example-svc.ns1.example.com secretName: example-tls backend: service: name: example-svc port: number: 8080 example-svc ns1 example.com example-svc ns1 example.com example-svc ns1 example.com ࢦఆͨ͠ServiceΛ IngressͰެ։͢Δ ެ։͢Δϗετ໊ͱͯ͠ αʔϏεσΟεΧόϦతʹ ࣗಈͰ෇༩͢Δ IUUQTHJUIVCDPNUBLVNBLVNFTFSWJDFFYQPTFPQFSBUPS αʔϏε໊ namespace υϝΠϯ Custom Resource
  10. ։ൃܦҢ

  11. ։ൃܦҢ w 8FCΞϓϦέʔγϣϯΛෳ਺ӡ༻͍ͯͯ͠ɺ։ൃؔ܎ऀ͸໊Ҏ্͍ Δ w 1VMM3FRVFTUຖͷϓϨϏϡʔ؀ڥΛLVCFSOFUFT্Ͱ࣮ߦ͍ͨ͠ w 1VMM3FRVFTU͕࡞੒͞ΕͨΒɺઐ༻ͷ؀ڥ্ཱ͕͕ͪΔ w ݱࡏ͸ͭͷTUBHJOH؀ڥΛ։ൃऀͰڞ༗͍ͯ͠Δ

    w σϓϩΠͷखؒ w ར༻ऀͷڝ߹  എܠ
  12. kubernetes cluster app repo system manifests repo Pull Request Github

    Actions ArgoCD Config ArgoCD Github Actions ArgoCD Config Pod Ingress Namespace: app-pr-XXX 1.PRͷ࡞੒ 2.PRͷϒϥϯνΛ kubernetesΫϥελʹ σϓϩΠ͢ΔͨΊͷ ArgoCDͷઃఆΛੜ੒ commit Service 3.ArgoCDͷ ઃఆ௥ՃΛݕ஌ 4.ArgoCDͷઃఆΛ σϓϩΠ 5.PRͷϒϥϯνͷ σϓϩΠΛ։࢝ dispatch argocd-util ίϚϯυͰੜ੒ ʲ1VMM3FRVFTUຖͷϓϨϏϡʔ؀ڥʳ ArgoCD΍Github ActionsΛ׆༻ͯ͠Pull ReqτϦΨʔͰGitOpsͰϓϨϏϡʔ؀ڥΛੜ੒͍ͯ͠Δ 6.PRͷϒϥϯνͷ ϓϨϏϡʔ؀ڥ͕࡞ΒΕΔ 7. external-dnsͰAϨίʔυ, cert-managerͰTLSূ໌ॻΛ ࣗಈઃఆ
  13. !"" main.go #"" manifests !"" base $ !"" kustomization.yaml $

    !"" app.deployment.yaml $ #"" app.service.yaml #"" overlays !"" production $ !"" kustomization.yaml $ #"" app.ingress.yaml #"" staging !"" kustomization.yaml #"" app.ingress.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: app-pr-XXX spec: destination: namespace: app-pr-XXX server: https://kubernetes.default.svc source: path: manifests/overlays/staging repoURL: https://github.com/takumakume/app targetRevision: future-branch syncPolicy: syncOptions: - CreateNamespace=true ֤؀ڥͷmanifestsΛkustomizeͰ؅ཧ͍ͯ͠Δ #"" staging !"" kustomization.yaml #"" app.ingress.yaml Pull RequestຖʹNamespaceΛ੾ͬͯ staging؀ڥͷෳ੡Λ࡞͍ͬͯΔ app-pr-XXX ʲ1VMM3FRVFTUຖͷϓϨϏϡʔ؀ڥʳ app repo
  14. ։ൃܦҢ  ٕज़త՝୊ apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: app-ingress

    namespace: app spec: rules: - host: staging-app.example.com http: paths: - backend: service: name: app-svc port: number: 8080 path: / pathType: Prefix tls: - hosts: - staging-app.example.com secretName: app-tls staging-app.example.com staging-app.example.com w ؀ڥͷෳ੡͸࡞Εͯ΋ɺ*OHSFTTͷϗετ ໊෦෼Λม͑Δ͜ͱ͕Ͱ͖ͳ͍ɻ w LVTUPNJ[Fͷ+40/1BUDIػೳ΍ɺZRίϚ ϯυͳͲͰஔ׵ͭͭ͠ద༻͢Δ͜ͱ͸Ͱ ͖Δ͕؅ཧ͕൥ࡶʹͳΔɻʢܦݧࡁʣ  w ద༻௚લͰNBOJGFTUTΛॻ͖׵࣮͑ͭͭ ߦ͢Δͱ(JU0QTʹΑΔԸܙ͕ബΕΔɻ ίί ίί
  15. ։ൃܦҢ  ࣮૷ํ਑ w (JU0QT͕Ͱ͖Δ͜ͱ w એݴతͰ͋Δ͜ͱ w ϓϨϏϡʔ؀ڥʹΞΫηε͢ΔͨΊͷϗετ໊ΛͲ͏͢Δ͔ʁ w

    LVCFSOFUFTͷ4FSWJDF%JTDPWFSZ w 4&37*$&@/".&/".&"1"$&TWDDMVTUFSMPDBM w *OHSFTTͰ࣮ݱͰ͖ΔͱศརͰ͸ͳ͍͔
  16. apiVersion: service-expose.../v1alpha1 kind: ServiceExpose metadata: name: example namespace: ns1 spec:

    backend: service: name: example-svc port: number: 8080 domain: example.com path: / pathType: Prefix tlsEnable: true tlsSecretName: example-tls annotations: cert-manager.io/cluster-issuer: letsencrypt apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example namespace: ns1 annotations: cert-manager.io/cluster-issuer: letsencrypt spec: rules: - host: example-svc.ns1.example.com http: paths: - backend: service: name: example-svc port: number: 8080 path: / pathType: Prefix tls: - hosts: - example-svc.ns1.example.com secretName: example-tls backend: service: name: example-svc port: number: 8080 example-svc ns1 example.com example-svc ns1 example.com example-svc ns1 example.com ެ։͢Δ Service αʔϏεσΟεΧόϦతͳ ϗετ໊ΛࣗಈͰ෇༩ αʔϏε໊ namespace υϝΠϯ ࠶ܝ
  17. kubernetes cluster app repo system manifests repo Pull Request Github

    Actions ArgoCD Config ArgoCD Github Actions ArgoCD Config Pod Namespace: app-pr-XXX 1.PRͷ࡞੒ 2.PRͷϒϥϯνΛ kubernetesΫϥελʹ σϓϩΠ͢ΔͨΊͷ ArgoCDͷઃఆΛੜ੒ commit Service 3.ArgoCDͷ ઃఆ௥ՃΛݕ஌ 4.ArgoCDͷઃఆΛ σϓϩΠ 5.PRͷϒϥϯνͷ σϓϩΠΛ։࢝ 6.PRͷϒϥϯνͷ ϓϨϏϡʔ؀ڥ͕࡞ΒΕΔ dispatch argocd-util ίϚϯυͰੜ੒ Service Expose ੜ੒ app.app-pr-XXX.example.com Ingress 7. external-dnsͰAϨίʔυ, cert-managerͰTLSূ໌ॻΛ ࣗಈઃఆ ServiceExposeͷΈσϓϩΠ͢Δ͜ͱͰIngress͸ࣗಈੜ੒͞ΕΔ
  18. (PʹΑΔ0QFSBUPSͷ࣮૷

  19. (PʹΑΔ0QFSBUPS࣮૷  w ࣮૷खஈ w IUUQTLVCFSOFUFTJPEPDTDPODFQUTFYUFOELVCFSOFUFTPQFSBUPS w 0QFSBUPS'SBNFXPSL w $/$'*ODVCBUJOH1SPKFDU

    w (PΛ༻͍࣮ͨ૷ʹ͓͍ͯ͸಺෦ͰLVCFCVJMEFSΛར༻͍ͯ͠Δ
  20. 0QFSBUPS'SBNFXPSL  w 0QFSBUPS4%, w LVCFSOFUFT"1*ʹਂ͍஌͕ࣝͳͯ͘΋ɺϩδοΫʹूதͰ͖ΔΑ͏ ʹӅณͯ͘͠Ε͍ͯΔ w ίʔυδΣωϨʔλʔ w

    ςετ w ύοέʔδϯά
  21. 0QFSBUPS'SBNFXPSL  w ࣮૷ʹ͋ͨͬͯ΍ͬͨ͜ͱ w 0QFSBUPS'SBNFXPSLͷެࣜυΩϡϝϯτ͕ॆ࣮͍ͯ͠ΔͷͰɺج ຊతʹ͸ͦ͜Λࢀর͢Δ IUUQTTELPQFSBUPSGSBNFXPSLJPEPDTCVJMEJOHPQFSBUPSTHPMBOH  w

    ͨ·ʹࡌ͍ͬͯͳ͍৔߹͕͋ΔͷͰɺLVCFCVJMEFSͷެࣜυΩϡϝ ϯτͰิ׬͢Δ IUUQTCPPLLVCFCVJMEFSJP  w ϕετϓϥΫςΟεͷ࣮ફ IUUQTTELPQFSBUPSGSBNFXPSLJPEPDTCFTUQSBDUJDFTCFTUQSBDUJDFT 
  22.  w 3FDPODJMFS-PPQͷ࣮૷͕ϝΠϯ w ྫɿʮ1PEΛݸ࣮ߦ͢Δʯͱఆٛ͢Ε͹ͦͷঢ়ଶʹऩଋ͢Δಈ͖ (PʹΑΔ0QFSBUPS࣮૷ Observe Diff Action ঢ়ଶΛऔಘ

    ࠩ෼Λݕग़ ࠩ෼ͷमਖ਼
  23.  w TFSWJDFFYQPTFPQFSBUPSͷ3FDPODJMFS-PPQ (PʹΑΔ0QFSBUPS࣮૷ Observe Diff Action - ੜ੒͢΂͖IngressͷSpec͸ʁ -

    ੜ੒͢΂͖Ingressͱݱࡏͷ Ingressͷࠩ෼͸ͳʹ͔ʁ - ࠩ෼ΛຒΊΔͨΊʹIngressͷ Create/Update/DeleteΛ࣮ߦ
  24.  w ςετ (PʹΑΔ0QFSBUPS࣮૷ ServiceExpose ঢ়ଶऔಘ Ingress͸ ଘࡏ͢Δ͔ʁ Ingressͷ Ξοϓσʔτ͸

    ඞཁ͔ʁ Ingress ੜ੒ Ingress Ξοϓσʔτ Y Y N N Reconciler Loop w ্هͷΑ͏ʹ0QFSBUPS͸ঢ়ଶભҠ͕ൃੜ͢Δɻ w ͋ΒΏΔύλʔϯͰ3FDPODJMFS-PPQ͕ႈ౳ʹͳΔΑ͏ʹςετ͠ ͍ͨɻ
  25.  w 0QFSBUPS4%,͕ҎԼͷπʔϧΛ༻͍ͯৼΔ෣͍ςετͷ࣮ߦ؀ڥΛ ఏڙ͍ͯ͠Δ w FOWUFTUDPOUSPMMFSSVOUJNFͷύοέʔδͰɺςετ༻ͷ LVCFSOFUFTDPOUSPMQMBOFΛఏڙ͢Δ w HJOLHP(PMBOHͷ#%%ςετϑϨʔϜϫʔΫ w

    HPNFHB(PMBOHͷ.BUDIFS-JCSBSZ HJOLHPͱηοτͰ࢖͏ (PʹΑΔ0QFSBUPS࣮૷
  26.  w ྫ͑͹ɺ4FSWJDF&YQPTF$VTUPN3FTPVSDF͕σϓϩΠ͞Εͨ͋ͱʹɺ 4UBUVT͕3FBEZʹભҠ͢ΔͷΛ଴ͪɺ*OHSFTT͕ੜ੒Ͱ͖͍ͯΔ͔ͱ͍ ͏ςετ͕ॻ͚Δɻ (PʹΑΔ0QFSBUPS࣮૷

  27. ॴײ

  28.  w (JU0QTͰΧόʔͰ͖ͳ͔ͬͨҰ෦ͷ໰୊Λ੾Γग़ͯ͠0QFSBUPSͱ͍ ͏ख๏ͰղܾͰ͖ͨɻ w ϓϨϏϡʔ؀ڥͷੜ੒શମΛ0QFSBUPSͱ࣮ͯ͠૷͢Δ͜ͱ΋ߟ͑ ͕ͨɺιϑτ΢ΣΞΛγϯϓϧʹอͭ΄͏͕ྑ͍ͱߟ͑ͨɻ w (PͰ0QFSBUPSΛ࣮૷͢Δ্Ͱ0QFSBUPS4%,Λ࢖͕ͬͨɺϩδοΫ ʹूதͰ͖ͯศརͩͬͨɻ

    w (Pͱ͍͑͹ςʔϒϧۦಈςετΛΑ͘࢖͏͕ɺঢ়ଶભҠΛςετ͢ Δ্Ͱ#%%͸ศརͩͬͨɻ ॴײ