ゼロから始めるKubernetes Controller / Under the Kubernetes Controller

C174e1ef0c746f53d989b1902b4e674e?s=47 go_vargo
September 27, 2019

ゼロから始めるKubernetes Controller / Under the Kubernetes Controller

C174e1ef0c746f53d989b1902b4e674e?s=128

go_vargo

September 27, 2019
Tweet

Transcript

  1. 2.

    ࣗݾ঺հ ໊લ: ү ݡେ(@go_vargo) ॴଐ: גࣜձࣾίϩϓϥ ɹɹ Πϯϑϥάϧʔϓ ୈ2νʔϜ ৬຿಺༰:

    Kubernetes؀ڥΛྑ͍ײ͡ʹ͢Δ #21 CI/CD ʮίϩϓϥ͕࣮ફ͍ͯ͠ΔSpinnakerΛ༻͍ͨσϓϩΠઓུʯ https://speakerdeck.com/govargo/deploy-strategy-with-spinnaker-at-colopl 2
  2. 4.

    ࿩͞ͳ͍͜ͱ ɾKubernetes Custom Controller + CRDͷৄࡉ ※ ͨͩ͠ɺ७ਖ਼Controllerͷ࢓૊ΈΛ஌Δ͜ͱ͸ɺ Custom ControllerΛ࣮૷͢Δ্Ͱ΋໾ཱͪ·͢

    ɾKubernetes Custom ControllerΛ࣮૷͢ΔͨΊͷϑϨʔϜϫʔΫ KubebuilderɺOperator SDKͷৄࡉʹ͍ͭͯ͸࿩͠·ͤΜ ಺෦SDKͰ͋Δcontroller-runtime΍controller-toolsʹ͍ͭͯ΋ ຊൃදͰ͸৮Ε·ͤΜ ࿩͞ͳ͍͜ͱ controller-runtimeࢀߟ: https://www.slideshare.net/pfi/kubernete-meetup-tokyo-18-kubebuildercontrollerruntime
  3. 8.

    KubernetesϋΠϨϕϧΞʔΩςΫνϟ .BTUFS 8PSLFS FUDE DMPVEDPOUSPMMFS NBOBHFS /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ DPOUSPMMFS NBOBHFS BQJTFSWFS TDIFEVMFS /PEF $POUBJOFS 3VOUJNF LVCFMFU /PEF $POUBJOFS 3VOUJNF LVCFMFU DMPVE LVCF QSPYZ LVCF QSPYZ
  4. 10.

    KubernetesϋΠϨϕϧΞʔΩςΫνϟ .BTUFS 8PSLFS FUDE DMPVEDPOUSPMMFS NBOBHFS /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ DPOUSPMMFS NBOBHFS BQJTFSWFS TDIFEVMFS /PEF $POUBJOFS 3VOUJNF LVCFMFU /PEF $POUBJOFS 3VOUJNF LVCFMFU DMPVE LVCF QSPYZ LVCF QSPYZ
  5. 13.

    3FQMJDB4FU $POUSPMMFS LJOE3FQMJDB4FU NFUBEBUB OBNFYYYYYY TQFD ʜ 3FTPVSDF  3FTPVSDF

     3FTPVSDF ʜ 3FTPVSDF / ؅ཧ $POUSPM-PPQ .BOJGFTU ControllerͱResource Controller͕Resourceͷ؅ཧΛߦ͏ 3FQMJDB4FU$POUSPMMFSͷྫ
  6. 14.

    3FQMJDB4FU $POUSPMMFS ControllerͱResource Controller͸ɺͻͱͭͷResourceΛ؅ཧ͢Δ 3FQMJDB4FU 1PE %FQMPZNFOU $POUSPMMFS %FQMPZNFOU NBOBHF

    DSFBUF NBOBHF DSFBUF ࢀߟ: OwnerReference https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/ ্ҐResource͕ԼҐResourceΛ؅ཧ͢Δ࢓૊Έͱͯ͠ʮOwnerReferenceʯ͕͋Δ ্ҐResourceΛ࡟আ͢ΔͱɺԼҐResource͸ΨϕʔδίϨΫγϣϯͰ࡟আ͞ΕΔ
  7. 17.

    Control Loop(Reconciliation Loop) Controller Loop͸ɺControllerͷجຊݪཧ ※Reconciliation Loopͱ΋ݺ͹Ε͍ͯΔ Controller LoopͷྲྀΕ: 1.

    Resourceͷݱࡏͷঢ়ଶΛಡΈࠐΉ 2. ResourceΛ๬Μͩঢ়ଶʹมߋ͢Δ 3. Resourceͷঢ়ଶ(Status)Λߋ৽͢Δ Controller LoopʹΑͬͯɺ๬Μͩঢ়ଶΛએݴ͢Δ͜ͱͰ ؀ڥ͕ߏங͞ΕΔɻ = Immutable Infrastructure͕࣮ݱͰ͖Δ ϧʔϓ
  8. 19.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ LJOE3FQMJDB4FU NFUBEBUB OBNFYYYYYY TQFD ʜ .BOJGFTU 6TFS

    .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU LVCF QSPYZ BQJTFSWFS TDIFEVMFS ,VCFDUMBQQMZGNBOJGFTUZBNM †3FTPVSDFΛ"QQMZ͢Δ 3FQMJDB4FU $POUSPMMFS ᶃ3FQMJDB4FU͕ੜ੒͞ΕΔ
  9. 20.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS $POUSPM-PPQ ᶄ3FQMJDB4FU͕ੜ੒͞Εͨ͜ͱΛɺ3FQMJDB4FU$POUSPMMFS͕ݕ஌ ᶄ`ϝΠϯϩδοΫʹΑͬͯɺ TQFDOPEF/BNF͕ۭͷ1PEΛ࡞੒͢Δ DSFBUF ˞8PSLFSʹ͸·ͩσϦόϦ͞Εͳ͍
  10. 21.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS ᶅ4DIFEVMFS͕1PEͷ࡞੒Λݕ஌ queue ᶅ`1PEͷTQFDOPEF/BNF͕ۭͳͷͰɺ TDIEVMFS͕RVFVFʹ௥Ճ͢Δ
  11. 22.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS ᶆLVCFMFU͕1PEͷ࡞੒Λݕ஌ queue ᶆ`1PEͷTQFDOPEF/BNF͕ۭͳͷͰɺ LVCFMFU͸ॲཧΛεΩοϓ skip
  12. 23.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS ᶇ4DIFEVMFS͕1PEΛRVFVF͔ΒऔΓग़͢ queue ᶇ`ΞαΠϯՄೳͳ/PEFʹ1PEΛ εέδϡʔϦϯά͢Δ schedule ᶇ`1PEͷTQFDOPEF/BNFΛߋ৽ 6QEBUF TQFDOPEF/BNF
  13. 24.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue ᶈLVCFMFU͕1PEͷߋ৽Λݕ஌ ᶈ`LVCFMFU͕ίϯςφΛىಈ
  14. 25.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue ᶉLVCFMFU͕BQJTFSWFSʹϦΫΤετ͠ɺ 1PEͷ4UBUVTΛߋ৽ TUBUVTDPOEJUJPOT
  15. 26.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue ͜͜Ͱɺίϯςφ͕ͳΜΒ͔ͷཧ༝Ͱࢮ๢ͨ͠ͱ͠·͢
  16. 27.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue 1PEͷ5FSNJOBUJOH TUBUVTDPOEJUJPOT ᶊLVCFMFU͕BQJTFSWFSʹϦΫΤετ͠ɺ 1PEͷ4UBUVTΛߋ৽
  17. 28.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue 1PEͷ5FSNJOBUJOH ᶋ3FQMJDB4FU$POUSPMMFS͕1PEͷߋ৽Λ ݕ஌ EFMFUF ᶋ`3FQMJDB4FU$POUSPMMFS͕1PEΛ࡟আ
  18. 29.

    ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU

    LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue 3FDPODJMF ᶌ3FQMJDB4FU$POUSPMMFS͕3FDPODJMF͠ɺ ๬Μͩঢ়ଶ 3FQMJDBT ʹ෮چ͠ɺ 1PE͕৽͘͠࡞੒͞ΕΔ EFMFUF Ҏ߱ɺϧʔϓʜ ᶄʙᶌͷॲཧΛ܁Γฦ͢
  19. 30.

    Appendix) ιʔε ᶄ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L487 ᶅ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/eventhandlers.go#L436 ᶆ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/kubelet/config/apiserver.go#L33 ᶇ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/scheduler.go#L535

    ᶈ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L803 ᶉ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kubelet.go#L1527 ᶊ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kubelet.go#L2006 ᶋ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/controller/replicaset/replica_set.go#L535 ᶌ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L487
  20. 33.

    Kubernetes is jazz improv OrchestrationͰ͸ͳ͘ɺ Player(Controller)͕֤ʑͷԋ૗ (Control Loop)ʹूத͢Δ͜ͱͰ ଈڵͷԻָͱͯ͠ɺશମ͕੒Γཱ͍ͬͯΔ Kubernetes

    is more jazz improv than orchestration. Joe Beda https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca Co-founder Core Kubernetes: Jazz Improv over Orchestration
  21. 34.

    ͳͥControl LoopΛߦ͏ͷ͔ʁ Control LoopΛߦ͏ཧ༝Λߟ͑Δ࣌ʹɺॏཁʹͳΔ΋ͷ͕ ʮEventʯ ※ e.g. Added, Modified, Deleted,

    Error… ෼ࢄίϯϙʔωϯτͰ੒ΓཱͭKubernetesʹͱͬͯɺ Event͸ɺͦΕͧΕͷίϯϙʔωϯτΛͭͳ͙ॏཁͳ΋ͷ ࢀߟ: Events, the DNA of Kubernetes EventΛͲͷΑ͏ʹѻ͏͔ɺ伴ͱͳΔߟ͑ํ͕ ɾEdge-driven Triggers ɾLevel-driven Triggers https://www.mgasch.com/post/k8sevents/
  22. 35.

    Appendix) Edge vs. Level https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d ࢀߟ: Level Triggering and Reconciliation

    in Kubernetes Edge-driven Triggers Level-driven Triggers EventͷൃੜʹԠͯ͡Trigger ಛఆͷঢ়ଶʹͳΔͱTrigger up down low high low
  23. 37.

    ΋͠΋Controller͕खଓ͖ܕͩͬͨΒ ݱࡏͷ3FQMJDBT਺   ˢ  ࡞ ੒  ˢ

     ࡟ ݮ खଓ͖తʹಈͨ͘ΊɺEvent͕ൃੜ͢Δͨͼʹ Edge-driven Triggerͱͯ͠ಈ͘ Ұݟ໰୊ͳ͍Α͏ʹݟ͑Δ͕ɺऑ఺΋͋Δ &WFOU
  24. 38.

    ΋͠΋Reconcile͕ͳ͔ͬͨΒ ݱࡏͷ3FQMJDBT਺   ˢ  ࡞ ੒ ˢ 

    ࡟ ݮ Ұ࣌తͳωοτϫʔΫͷෆௐ΍όάͳͲͷো֐͕͋ͬͨ࣌ʹ Eventͷ৘ใ͕ࣦΘΕͯ͠·͏ ো ֐ &WFOU
  25. 39.

    Resync IntervalͱReconcile ݱࡏͷ3FQMJDBT਺   ˢ  ࡞ ੒ 

    ˢ  ࡟ ݮ Resync Interval͝ͱʹReconcile(ௐ੔)Λ͢Δ͜ͱͰɺ Event͕ࣦΘΕͯ΋ཧ૝తͳঢ়ଶʹ͚ۙͮΔ ো ֐ SFTZODJOUFSWBM &WFOU Kubernetes = Edge-driven Trigger + Level-driven Trigger
  26. 41.

    ༻ޠͷ੔ཧ Kind: API Objectͷछྨ(Deployment, ServiceͳͲ) 
 Resource: Kindͱಉ͡ҙຯ߹͍Ͱ༻͍ΒΕΔɻ HTTP Endpointͱͯ͠΋ར༻͞ΕΔɻ

    খจࣈ͔ͭෳ਺ܗͰදݱ͞ΕΔ(pods, servicesͳͲ) Object: ࡞੒͞ΕͨAPI Object ͷ࣮ମ Metadata΍Spec, StatusͳͲͷσʔλΛอ࣋͢Δ
  27. 43.

    Appendix) Custom Controller SDK Kubebuilder Informer ϑϨʔϜϫʔΫ ίϯϙʔωϯτ Lister Operator

    SDK ϥΠϒϥϦ (ϋΠϨϕϧ) controller-runtime controller-tools ϥΠϒϥϦ (ϩʔϨϕϧ) client-go api-machinery etc… Scheme runtime.Object WorkQueue etc…
  28. 44.

    ControllerΛࢧ͑ΔϥΠϒϥϦ client-go: KubernetesͷެࣜΫϥΠΞϯτϥΠϒϥϦ Kubernetesຊମͷ։ൃʹ΋࢖ΘΕ͍ͯΔ Controller࡞੒ʹ͸͔ܽͤͳ͍ϥΠϒϥϦ api-machinery: Kubernetes API Object &

    Kubernetes API like Object ʹඞཁͳػೳΛඋ͑ͨϥΠϒϥϦ e.g. conversion, decode, encode, etc… Controller͸API ObjectΛѻ͏ͷͰɺඞཁʹͳΔ code-generator: Informer, Lister, clientset, DeepCopyͳͲͷίʔυΛੜ੒
  29. 49.

    Appendix) InformerΛ࢖࣮ͬͨ૷ func main() { ... clientset, err := kubernetes.NewForConfig(config)

    // Create InformerFactory informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30) // Create pod informer by informerFactory podInformer := informerFactory.Core().V1().Pods() // Add EventHandler to informer podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(new interface{}) { log.Println("Added") }, UpdateFunc: func(old, new interface{}) { log.Println("Updated") }, DeleteFunc: func(old interface{}) { log.Println("Deleted") }, }) // Start Go routines informerFactory.Start(wait.NeverStop) // Wait until finish caching with List API informerFactory.WaitForCacheSync(wait.NeverStop) // Create Pod Lister podLister := podInformer.Lister() // Get List of pods _, err = podLister.List(labels.Nothing()) … } https://github.com/govargo/kubecontorller-book-sample-snippet/blob/master/02/podinformer/podinformer.go ᶃ kubeconfig͔ΒconfigΛੜ੒ ᶄ config͔ΒclientsetΛੜ੒ ᶅ informerFactoryੜ੒ ᶆ ᶅΛ࢖ͬͯinformerੜ੒ ᶇ EventHandlerΛ௥Ճ Add, Update, Delete࣌ʹൃՐ ᶈ InformerΛىಈ ᶉ Lister(Getter)Λੜ੒ ᶊ ListerΛ࢖ͬͯGet΍List
  30. 52.

    3FqFDUPS %FMUB'*'0 RVFVF -JTU8BUDI JONFNPSZDBDIF 4UPSF *OEFYFS -JTUFS (FUPS-JTU FH

    DPSFW1PE (FUPS-JTU 3FRVFTU *OGPSNFS )BOEMF%FMUBT ৄࡉInformer
  31. 60.

    3FqFDUPS %FMUB'*'0 RVFVF JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI JOEFYFS(FU#Z,FZ

    PS JOEFYFS-JTU ৄࡉInformer ~σʔλΛऔΓग़͢~ -JTUFS (FUPS-JTU 3FRVFTU
  32. 66.

    Appendix) WorkQueueΛ࢖࣮ͬͨ૷ func main() { ... clientset, err := kubernetes.NewForConfig(config)

    // Create InformerFactory informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30) // Create pod informer by informerFactory podInformer := informerFactory.Core().V1().Pods() // Create RateLimitQueue queue := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter()) // shutdown when process ends defer queue.ShutDown() // Add EventHandler to informer podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(old interface{}) { var key string var err error if key, err = cache.MetaNamespaceKeyFunc(old); err != nil { runtime.HandleError(err) return } queue.Add(key) log.Println("Added: " + key) }, UpdateFunc: func(old, new interface{}) { … }, DeleteFunc: func(old interface{}) { … }, }) … } https://github.com/govargo/kubecontorller-book-sample-snippet/blob/master/02/workqueue/enqueuePod.go ᶃ kubeconfig͔ΒconfigΛੜ੒ ᶄ config͔ΒclientsetΛੜ੒ ᶅ informerFactoryੜ੒ ᶆ ᶅΛ࢖ͬͯinformerੜ੒ ᶇ EventHandlerΛ௥Ճ Add, Update, Delete࣌ʹൃՐ ᶈ InformerΛىಈ ᶉ Lister(Getter)Λੜ੒ ᶊ ListerΛ࢖ͬͯGet΍List ᶇ WorkQueueੜ੒
  33. 67.

    3FqFDUPS %FMUB'*'0 RVFVF -JTU8BUDI 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS

    8PSL2VFVF 1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue
  34. 68.

    3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF

    1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ &WFOU -JTU8BUDI
  35. 69.

    3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF

    1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ᶃ8BUDI ৄࡉWorkQueue ~Enqueue~
  36. 70.

    3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF

    1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶄ1PQ
  37. 71.

    3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF

    1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶅ "EE'VOD 6QEBUF'VOD %FMFUF'VOD
  38. 72.

    3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF

    1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI namesapce/name e.g. default/nginx ᶆXPSLRVFVF"EE
  39. 73.

    3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF

    1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶅ "EE'VOD 6QEBUF'VOD %FMFUF'VOD
  40. 74.

    3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF

    1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶆXPSLRVFVF"EE
  41. 75.

    3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF

    1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Dequeue~ 8BUDI ᶇXPSLRVFVF(FU
  42. 76.

    Appendix) Informer enqueue ιʔε ᶃ https://github.com/kubernetes/client-go/blob/master/tools/cache/reflector.go#L267 ᶄ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L153 ᶅ
 https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L198

    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L153 ※ ReplicaSet Controllerͷ৔߹ ᶆ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L417 ※ ReplicaSet Controllerͷ৔߹ ᶇ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L438 ※ ReplicaSet Controllerͷ৔߹
  43. 79.

    *OGPSNFS 1SPDFTT *UFN 3FTPVSDF &WFOU )BOEMFS 8PSL2VFVF ControllerͷCycle Informer +

    WorkQueue + Controller ͷશମͷCycleΛ֬ೝ $POUSPMMFS ϝΠϯϩδοΫ
  44. 88.

    *OGPSNFS 1SPDFTT *UFN 3FTPVSDF &WFOU )BOEMFS ControllerͷCycle XBUDI ᶈXPSL2VFVF'PSHFU ᶉXPSL2VFVF%POF

    Reconcile͕ਖ਼ৗऴྃͨ͠Β ΞΠςϜΛ8PSL2VFVF͔Βআ֎ 8PSL2VFVF
  45. 92.

    Controllerͷجຊઓུ Read͸In-memory-cache͔ΒɺWrite͸api-server΁ ※cacheΛ͍͡ΔͱҰ؏ੑΛอͭͷ͕೉͍͠ͷͰɺ Write͢Δͱ͖͸ɺҰճDeepCopy͔ͯ͠Βߋ৽͢Δ e.g. kubernetes/pkg/controller/replicaset/replica_set.go rs = rs.DeepCopy() newStatus

    := calculateStatus(rs, filteredPods, manageReplicasErr) // Always updates status as pods come up or die. updatedRS, err := updateReplicaSetStatus(rsc.kubeClient.AppsV1(). ὎ ReplicaSets(rs.Namespace), rs, newStatus) https://github.com/kubernetes/kubernetes/blob/release-1.15/pkg/controller/replicaset/replica_set.go#L611
  46. 93.

    XPSLFS XPSLFS QSPDFTT/FYU8PSL*UFN TZOD)BOEMFS ControllerͷMain Logic worker: processNextWorkItemΛແݶϧʔϓ processNextWorkItem: WorkQueueΛૢ࡞(Get,

    Add)ͯ͠ ReconcileΛݺͼग़͢ syncHandler: Reconcileʹ͋ͨΔϝΠϯϩδοΫ Add TZOD)BOEMFS Update Delete Event Reconcile Eventͷछྨʹ͔͔ΘΒͣɺ Reconcile͢Δ
  47. 94.

    Appendix) ReplicaSet Controller worker: processNextWorkItemΛແݶϧʔϓ processNextWorkItem: WorkQueueΛૢ࡞(Get, Add)ͯ͠ ReconcileΛݺͼग़͢ syncReplicaSet:

    Reconcileʹ͋ͨΔϝΠϯϩδοΫ XPSLFS XPSLFS QSPDFTT/FYU8PSL*UFN TZOD3FQMJDB4FU https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L432 https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L437 https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L562 ReplicaSet Controllerͷιʔε v1.16 ※Deployment΋ྨࣅͷߏ଄Λ͍ͯ͠Δ
  48. 98.

    Controllerͷ·ͱΊ ɾController͸ɺControl Loop(Reconciliation Loop)ʹΑͬͯ એݴతͳAPIΛ࣮ݱ ɾ෼ࢄίϯϙʔωϯτ͸ͦΕͧΕಠཱ͓ͯ͠Γɺ Event͕ίϯϙʔωϯτؒΛ݁ͼ͚ͭΔ ɾclient-go, apimachinery, code-generator౳͕ControllerΛࢧ͑Δ

    ɾInformer͸ೋͭͷ໨తΛ࣋ͭ ᶃ in-memory-cacheʹObjectͷσʔλΛ֨ೲ͢Δ ᶄ EventHandlerܦ༝ͰWorkQueueʹΞΠςϜΛ֨ೲ͢Δ ɾWorkQueueʹஷΊΒΕͨΞΠςϜ͸ReconcileͰॲཧ͞ΕΔ ※ ͜ΕΒΛ஌Βͳͯ͘΋ɺController͸ར༻Ͱ͖·͢
  49. 99.

    Deep Dive΁ͷಓ ʙ ΋ͬͱਂ͘஌Γ͍ͨਓ޲͚ʹ ʙ ɾSample ControllerΛಈ͔͢ ɾKubernetesຊମͷ࣮૷ΛݟͯΈΔ - Deployment

    Controller - ReplicaSet Controller ɾCustom ControllerΛ࡞ͬͯΈΔ(+ CRD) ϦϯΫ: https://github.com/kubernetes/sample-controller mkdir -p $GOPATH/src/k8s.io && cd $GOPATH/src/k8s.io && git clone https://github.com/kubernetes/sample-controller.git export GO111MODULE=on go build -o sample-controller . ./sample-controller -kubeconfig $HOME/.kube/config https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/deployment/deployment_controller.go https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go
  50. 100.

    Programming Kubernetes https://programming-kubernetes.info/ ঺հϒϩάࢀߟ: https://go-vargo.hatenablog.com/entry/2019/08/05/201546 O’Reilly Media, Inc.ΑΓग़൛ ʮProgramming Kubernetesʯ

    Stefan Schimanski, Michael Hausenblasஶ 1~2ষ: Kubernetesͷجຊࢥ૝, API Object 3ষ: client-go 4ষ: CRD 5ষ: code-generator 6~7ষ: Custom Controller 8~9ষ: Custom API Server, CRDԠ༻ػೳ
  51. 101.

    ΧελϜίϯτϩʔϥ΁ͷಓ https://go-vargo.booth.pm/items/1566979 Custom ControllerΛ࡞ΔͨΊͷΨΠυ CRD + Custom Controller(=Operator)Λ ΍Γ͍ͨਓ޲͚ 1ষ:

    Controller + CRD 2ষ: client-go, api-machinery 3ষ: Sample Controller 4ষ: controller-runtime, controller-tools 5ষ: Kubebuilder v2Ͱαϯϓϧ࣮૷ 6ষ: Operator SDKͰαϯϓϧ࣮૷
  52. 103.
  53. 104.

    Reference ɾWeb Article - https://kubernetes.io/docs/concepts/architecture/nodes/ - https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/ - https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md -

    A deep dive into Kubernetes controllers (https://engineering.bitnami.com/articles/a-deep-dive-into-kubernetes-controllers.html) - Core Kubernetes: Jazz Improv over Orchestration (https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca) - Events, the DNA of Kubernetes(https://www.mgasch.com/post/k8sevents/) ɾPresentation - Kubernete Meetup Tokyo #18 - Kubebuilder/controller-runtime ೖ໳ (https://www.slideshare.net/pfi/kubernete-meetup-tokyo-18-kubebuildercontrollerruntime) - KubernetesͷιʔείʔυϦʔσΟϯάೖ໳ (https://speakerdeck.com/smatsuzaki/kubernetesfalsesosukodorideinguru-men) ɾBook - Programming Kubernetes (https://programming-kubernetes.info/)
  54. 105.

    Reference ɾRepository - Kubernetes(https://github.com/kubernetes/kubernetes) - Sample Controller(https://github.com/kubernetes/sample-controller) - client-go(https://github.com/kubernetes/client-go) -

    apimachinery(https://github.com/kubernetes/apimachinery) - codegenerator(https://github.com/kubernetes/code-generator) - what-happens-when-k8s(https://github.com/jamiehannaford/what-happens-when-k8s)