Share
Embed
Transcript
103 slides
-
ࠓͷඪɾ͢͜ͱ ɾKubernetesͷResourceΛ͋Δఔ͍ͬͯΔਓ͕ɺɹ ɹControllerͷࣝθϩͷঢ়ଶ͔ΒͰDeep DiveʹΑͬͯɺ ɹControllerͷΈɾ෦࣮ʹ͍ͭͯཧղͰ͖Δ͜ͱ ɾKubernetes Controllerͷ֓ཁɾࢥ ɾKubernetes ControllerͷΈ, ෦࣮
ɾControllerΛࢧ͑Δίϯϙʔωϯτ = ControllerͷϋΠϨϕϧʙϩʔϨϕϧ·Ͱ ͢͜ͱ ඪ -
͞ͳ͍͜ͱ ɾKubernetes Custom Controller + CRDͷৄࡉ ※ ͨͩ͠ɺ७ਖ਼ControllerͷΈΛΔ͜ͱɺ Custom ControllerΛ࣮͢Δ্Ͱཱͪ·͢
ɾKubernetes Custom ControllerΛ࣮͢ΔͨΊͷϑϨʔϜϫʔΫ KubebuilderɺOperator SDKͷৄࡉʹ͍ͭͯ͠·ͤΜ ෦SDKͰ͋Δcontroller-runtimecontroller-toolsʹ͍ͭͯ ຊൃදͰ৮Ε·ͤΜ ͞ͳ͍͜ͱ controller-runtimeࢀߟ: https://www.slideshare.net/pfi/kubernete-meetup-tokyo-18-kubebuildercontrollerruntime -
ΞδΣϯμ ɾ Kubernetes Controllerͱʁ ɾ Control Loop(Reconciliation Loop) ɾControllerΛࢧ͑Δɺ͓͖ͬͯ͘ίϯϙʔωϯτ ɾClient-Go
- Informer - WorkQueue ɾControllerͷCycle, Main Logic ɾControllerͷ·ͱΊ - Reference -
KubernetesϋΠϨϕϧΞʔΩςΫνϟ .BTUFS 8PSLFS FUDE DMPVEDPOUSPMMFS NBOBHFS /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ DPOUSPMMFS NBOBHFS BQJTFSWFS TDIFEVMFS /PEF $POUBJOFS 3VOUJNF LVCFMFU /PEF $POUBJOFS 3VOUJNF LVCFMFU DMPVE LVCF QSPYZ LVCF QSPYZ -
KubernetesϋΠϨϕϧΞʔΩςΫνϟ .BTUFS 8PSLFS FUDE DMPVEDPOUSPMMFS NBOBHFS /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ DPOUSPMMFS NBOBHFS BQJTFSWFS TDIFEVMFS /PEF $POUBJOFS 3VOUJNF LVCFMFU /PEF $POUBJOFS 3VOUJNF LVCFMFU DMPVE LVCF QSPYZ LVCF QSPYZ -
3FQMJDB4FU $POUSPMMFS LJOE3FQMJDB4FU NFUBEBUB OBNFYYYYYY TQFD ʜ 3FTPVSDF 3FTPVSDF
3FTPVSDF ʜ 3FTPVSDF / ཧ $POUSPM-PPQ .BOJGFTU ControllerͱResource Controller͕ResourceͷཧΛߦ͏ 3FQMJDB4FU$POUSPMMFSͷྫ -
3FQMJDB4FU $POUSPMMFS ControllerͱResource ControllerɺͻͱͭͷResourceΛཧ͢Δ 3FQMJDB4FU 1PE %FQMPZNFOU $POUSPMMFS %FQMPZNFOU NBOBHF
DSFBUF NBOBHF DSFBUF ࢀߟ: OwnerReference https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/ ্ҐResource͕ԼҐResourceΛཧ͢ΔΈͱͯ͠ʮOwnerReferenceʯ͕͋Δ ্ҐResourceΛআ͢ΔͱɺԼҐResourceΨϕʔδίϨΫγϣϯͰআ͞ΕΔ -
"DUVBM4UBUF $POUSPM-PPQ %FTJSFE4UBUF 3FBESFBMSFTPVSDFT $IBOHFJOUFSOBMFYUFSOBMSFTPVSDFT PCTFSWF BOBMZ[F "DU Controllerͷجຊݪཧ جຊݪཧ:
Control Loop(Reconciliation Loop) -
Control Loop(Reconciliation Loop) Controller LoopɺControllerͷجຊݪཧ ※Reconciliation LoopͱݺΕ͍ͯΔ Controller LoopͷྲྀΕ: 1.
Resourceͷݱࡏͷঢ়ଶΛಡΈࠐΉ 2. ResourceΛΜͩঢ়ଶʹมߋ͢Δ 3. Resourceͷঢ়ଶ(Status)Λߋ৽͢Δ Controller LoopʹΑͬͯɺΜͩঢ়ଶΛએݴ͢Δ͜ͱͰ ڥ͕ߏங͞ΕΔɻ = Immutable Infrastructure͕࣮ݱͰ͖Δ ϧʔϓ -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ LJOE3FQMJDB4FU NFUBEBUB OBNFYYYYYY TQFD ʜ .BOJGFTU 6TFS
.BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU LVCF QSPYZ BQJTFSWFS TDIFEVMFS ,VCFDUMBQQMZGNBOJGFTUZBNM †3FTPVSDFΛ"QQMZ͢Δ 3FQMJDB4FU $POUSPMMFS ᶃ3FQMJDB4FU͕ੜ͞ΕΔ -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS $POUSPM-PPQ ᶄ3FQMJDB4FU͕ੜ͞Εͨ͜ͱΛɺ3FQMJDB4FU$POUSPMMFS͕ݕ ᶄ`ϝΠϯϩδοΫʹΑͬͯɺ TQFDOPEF/BNF͕ۭͷ1PEΛ࡞͢Δ DSFBUF ˞8PSLFSʹ·ͩσϦόϦ͞Εͳ͍ -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS ᶅ4DIFEVMFS͕1PEͷ࡞Λݕ queue ᶅ`1PEͷTQFDOPEF/BNF͕ۭͳͷͰɺ TDIEVMFS͕RVFVFʹՃ͢Δ -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS ᶆLVCFMFU͕1PEͷ࡞Λݕ queue ᶆ`1PEͷTQFDOPEF/BNF͕ۭͳͷͰɺ LVCFMFUॲཧΛεΩοϓ skip -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS ᶇ4DIFEVMFS͕1PEΛRVFVF͔ΒऔΓग़͢ queue ᶇ`ΞαΠϯՄೳͳ/PEFʹ1PEΛ εέδϡʔϦϯά͢Δ schedule ᶇ`1PEͷTQFDOPEF/BNFΛߋ৽ 6QEBUF TQFDOPEF/BNF -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue ᶈLVCFMFU͕1PEͷߋ৽Λݕ ᶈ`LVCFMFU͕ίϯςφΛىಈ -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue ᶉLVCFMFU͕BQJTFSWFSʹϦΫΤετ͠ɺ 1PEͷ4UBUVTΛߋ৽ TUBUVTDPOEJUJPOT -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue ͜͜Ͱɺίϯςφ͕ͳΜΒ͔ͷཧ༝Ͱࢮͨ͠ͱ͠·͢ -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue 1PEͷ5FSNJOBUJOH TUBUVTDPOEJUJPOT ᶊLVCFMFU͕BQJTFSWFSʹϦΫΤετ͠ɺ 1PEͷ4UBUVTΛߋ৽ -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue 1PEͷ5FSNJOBUJOH ᶋ3FQMJDB4FU$POUSPMMFS͕1PEͷߋ৽Λ ݕ EFMFUF ᶋ`3FQMJDB4FU$POUSPMMFS͕1PEΛআ -
ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS 3VOUJNF LVCFMFU
LVCF QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU $POUSPMMFS queue 3FDPODJMF ᶌ3FQMJDB4FU$POUSPMMFS͕3FDPODJMF͠ɺ Μͩঢ়ଶ 3FQMJDBT ʹ෮چ͠ɺ 1PE͕৽͘͠࡞͞ΕΔ EFMFUF Ҏ߱ɺϧʔϓʜ ᶄʙᶌͷॲཧΛ܁Γฦ͢ -
Appendix) ιʔε ᶄ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L487 ᶅ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/eventhandlers.go#L436 ᶆ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/kubelet/config/apiserver.go#L33 ᶇ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/scheduler.go#L535
ᶈ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L803 ᶉ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kubelet.go#L1527 ᶊ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kubelet.go#L2006 ᶋ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/controller/replicaset/replica_set.go#L535 ᶌ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L487 -
Controllerͱίϯϙʔωϯτͨͪ ֤ίϯϙʔωϯτͷಇ͖ ɾapi-server: ResourceͷCRUD ɾscheduler: ResourceͷεέδϡʔϦϯά ɾkubelet: ίϯςφͷىಈ ɾcontroller: ResourceͷReconcile(ௐ)
֤ίϯϙʔωϯτͦΕͧΕͷʹूத = શମΛ౷ɾ౷੍͢Δࢦشऀ͍ͳ͍ -
Kubernetes is jazz improv OrchestrationͰͳ͘ɺ Player(Controller)͕֤ʑͷԋ (Control Loop)ʹूத͢Δ͜ͱͰ ଈڵͷԻָͱͯ͠ɺશମ͕Γཱ͍ͬͯΔ Kubernetes
is more jazz improv than orchestration. Joe Beda https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca Co-founder Core Kubernetes: Jazz Improv over Orchestration -
ͳͥControl LoopΛߦ͏ͷ͔ʁ Control LoopΛߦ͏ཧ༝Λߟ͑Δ࣌ʹɺॏཁʹͳΔͷ͕ ʮEventʯ ※ e.g. Added, Modified, Deleted,
Error… ࢄίϯϙʔωϯτͰΓཱͭKubernetesʹͱͬͯɺ EventɺͦΕͧΕͷίϯϙʔωϯτΛͭͳ͙ॏཁͳͷ ࢀߟ: Events, the DNA of Kubernetes EventΛͲͷΑ͏ʹѻ͏͔ɺ伴ͱͳΔߟ͑ํ͕ ɾEdge-driven Triggers ɾLevel-driven Triggers https://www.mgasch.com/post/k8sevents/ -
Appendix) Edge vs. Level https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d ࢀߟ: Level Triggering and Reconciliation
in Kubernetes Edge-driven Triggers Level-driven Triggers EventͷൃੜʹԠͯ͡Trigger ಛఆͷঢ়ଶʹͳΔͱTrigger up down low high low -
͠Controller͕खଓ͖ܕͩͬͨΒ ݱࡏͷ3FQMJDBT ˢ ࡞ ˢ
ݮ खଓ͖తʹಈͨ͘ΊɺEvent͕ൃੜ͢Δͨͼʹ Edge-driven Triggerͱͯ͠ಈ͘ Ұݟͳ͍Α͏ʹݟ͑Δ͕ɺऑ͋Δ &WFOU -
͠Reconcile͕ͳ͔ͬͨΒ ݱࡏͷ3FQMJDBT ˢ ࡞ ˢ
ݮ Ұ࣌తͳωοτϫʔΫͷෆௐόάͳͲͷো͕͋ͬͨ࣌ʹ Eventͷใ͕ࣦΘΕͯ͠·͏ ো &WFOU -
Resync IntervalͱReconcile ݱࡏͷ3FQMJDBT ˢ ࡞
ˢ ݮ Resync Interval͝ͱʹReconcile(ௐ)Λ͢Δ͜ͱͰɺ Event͕ࣦΘΕͯཧతͳঢ়ଶʹ͚ۙͮΔ ো SFTZODJOUFSWBM &WFOU Kubernetes = Edge-driven Trigger + Level-driven Trigger -
༻ޠͷཧ Kind: API Objectͷछྨ(Deployment, ServiceͳͲ) Resource: Kindͱಉ͡ҙຯ߹͍Ͱ༻͍ΒΕΔɻ HTTP Endpointͱͯ͠ར༻͞ΕΔɻ
খจࣈ͔ͭෳܗͰදݱ͞ΕΔ(pods, servicesͳͲ) Object: ࡞͞ΕͨAPI Object ͷ࣮ମ MetadataSpec, StatusͳͲͷσʔλΛอ࣋͢Δ -
Appendix) Custom Controller SDK Kubebuilder Informer ϑϨʔϜϫʔΫ ίϯϙʔωϯτ Lister Operator
SDK ϥΠϒϥϦ (ϋΠϨϕϧ) controller-runtime controller-tools ϥΠϒϥϦ (ϩʔϨϕϧ) client-go api-machinery etc… Scheme runtime.Object WorkQueue etc… -
ControllerΛࢧ͑ΔϥΠϒϥϦ client-go: KubernetesͷެࣜΫϥΠΞϯτϥΠϒϥϦ Kubernetesຊମͷ։ൃʹΘΕ͍ͯΔ Controller࡞ʹ͔ܽͤͳ͍ϥΠϒϥϦ api-machinery: Kubernetes API Object &
Kubernetes API like Object ʹඞཁͳػೳΛඋ͑ͨϥΠϒϥϦ e.g. conversion, decode, encode, etc… ControllerAPI ObjectΛѻ͏ͷͰɺඞཁʹͳΔ code-generator: Informer, Lister, clientset, DeepCopyͳͲͷίʔυΛੜ -
ControllerΛࢧ͑Δίϯϙʔωϯτ ͦΕͧΕͷৄࡉɺControllerͷΈΛࢀরͭͭ͠ޙड़͠·͢ɻ Informer: ObjectͷEventΛࢹ͠ɺin-memory-cacheʹσʔλΛ֨ೲ͢Δ Lister: in-memory-cache͔ΒσʔλΛऔಘ͢Δ WorkQueue: Controller͕ॲཧ͢ΔΞΠςϜΛొ͓ͯ͘͠Ωϡʔ runtime.Object: શͯͷAPI
Objectڞ௨ͷInterface Scheme: Kubernetes APIͱGo TypeΛඥ͚Δػߏ ࠓճઆ໌লུ -
Appendix) InformerΛ࣮ͬͨ func main() { ... clientset, err := kubernetes.NewForConfig(config)
// Create InformerFactory informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30) // Create pod informer by informerFactory podInformer := informerFactory.Core().V1().Pods() // Add EventHandler to informer podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(new interface{}) { log.Println("Added") }, UpdateFunc: func(old, new interface{}) { log.Println("Updated") }, DeleteFunc: func(old interface{}) { log.Println("Deleted") }, }) // Start Go routines informerFactory.Start(wait.NeverStop) // Wait until finish caching with List API informerFactory.WaitForCacheSync(wait.NeverStop) // Create Pod Lister podLister := podInformer.Lister() // Get List of pods _, err = podLister.List(labels.Nothing()) … } https://github.com/govargo/kubecontorller-book-sample-snippet/blob/master/02/podinformer/podinformer.go ᶃ kubeconfig͔ΒconfigΛੜ ᶄ config͔ΒclientsetΛੜ ᶅ informerFactoryੜ ᶆ ᶅΛͬͯinformerੜ ᶇ EventHandlerΛՃ Add, Update, Delete࣌ʹൃՐ ᶈ InformerΛىಈ ᶉ Lister(Getter)Λੜ ᶊ ListerΛͬͯGetList -
Appendix) Shared Informer InformerΛར༻͢Δͱ͖ɺInformerͦͷͷͰͳ͘ɺ Shared InformerͱݺΕΔͷΛར༻͢Δ Shared InformerɺͻͱͭͷόΠφϦͰಉ͡ResourceΛڞ༗ Ͱ͖Δ %FQMPZNFOU
3FQMJDB4FU %BFNPO4FU 4FSWJDF +PC ʜ 4IBSFE*OGPSNFSTIBSFEDBDIFGPSTBNFSFTPVSDF kube-controller-manager -
3FqFDUPS %FMUB'*'0 RVFVF -JTU8BUDI JONFNPSZDBDIF 4UPSF *OEFYFS -JTUFS (FUPS-JTU FH
DPSFW1PE (FUPS-JTU 3FRVFTU *OGPSNFS )BOEMF%FMUBT ৄࡉInformer -
3FqFDUPS %FMUB'*'0 RVFVF ᶄ1PQ JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI
ৄࡉInformer ~σʔλΛΩϟογϡ~ -
3FqFDUPS %FMUB'*'0 RVFVF ᶆJOEFYFS"EE JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI
ৄࡉInformer ~σʔλΛΩϟογϡ~ -
3FqFDUPS %FMUB'*'0 RVFVF JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI ᶅ)BOEMF%FMUB
ৄࡉInformer ~σʔλΛΩϟογϡ~ ᶄ1PQ -
3FqFDUPS %FMUB'*'0 RVFVF JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI ᶆJOEFYFS"EE
ৄࡉInformer ~σʔλΛΩϟογϡ~ -
3FqFDUPS %FMUB'*'0 RVFVF JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI ৄࡉInformer
~σʔλΛऔΓग़͢~ -JTUFS (FUPS-JTU 3FRVFTU -
3FqFDUPS %FMUB'*'0 RVFVF JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI JOEFYFS(FU#Z,FZ
PS JOEFYFS-JTU ৄࡉInformer ~σʔλΛऔΓग़͢~ -JTUFS (FUPS-JTU 3FRVFTU -
Appendix) Informer cache ιʔε ᶃ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/reflector.go#L188 ᶄ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L153 ᶅ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/shared_informer.go#L455
ᶆ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/shared_informer.go#L464 -
Informerͱίϯϙʔωϯτ Informer: ObjectͷEventΛࢹ͠ɺin-memory-cacheʹσʔλΛ֨ೲ͢Δ Reflector: api-serverͷEventΛࢹ͢Δ DeltaFIFO: ObjectͷEvent͕ൃੜ͢Δͨͼʹ͏FIFOΩϡʔ Indexer: in-memory-cacheͷॻ͖ࠐΈಡΈࠐΈΛߦ͏ Store:
in-memory-cache Lister: in-memory-cache͔ΒIndexerܦ༝ͰσʔλΛऔಘ͢Δ -
Appendix) WorkQueueΛ࣮ͬͨ func main() { ... clientset, err := kubernetes.NewForConfig(config)
// Create InformerFactory informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30) // Create pod informer by informerFactory podInformer := informerFactory.Core().V1().Pods() // Create RateLimitQueue queue := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter()) // shutdown when process ends defer queue.ShutDown() // Add EventHandler to informer podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(old interface{}) { var key string var err error if key, err = cache.MetaNamespaceKeyFunc(old); err != nil { runtime.HandleError(err) return } queue.Add(key) log.Println("Added: " + key) }, UpdateFunc: func(old, new interface{}) { … }, DeleteFunc: func(old interface{}) { … }, }) … } https://github.com/govargo/kubecontorller-book-sample-snippet/blob/master/02/workqueue/enqueuePod.go ᶃ kubeconfig͔ΒconfigΛੜ ᶄ config͔ΒclientsetΛੜ ᶅ informerFactoryੜ ᶆ ᶅΛͬͯinformerੜ ᶇ EventHandlerΛՃ Add, Update, Delete࣌ʹൃՐ ᶈ InformerΛىಈ ᶉ Lister(Getter)Λੜ ᶊ ListerΛͬͯGetList ᶇ WorkQueueੜ -
3FqFDUPS %FMUB'*'0 RVFVF -JTU8BUDI 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS
8PSL2VFVF 1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue -
3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF
1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ &WFOU -JTU8BUDI -
3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF
1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ᶃ8BUDI ৄࡉWorkQueue ~Enqueue~ -
3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF
1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶄ1PQ -
3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF
1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶅ "EE'VOD 6QEBUF'VOD %FMFUF'VOD -
3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF
1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI namesapce/name e.g. default/nginx ᶆXPSLRVFVF"EE -
3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF
1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶅ "EE'VOD 6QEBUF'VOD %FMFUF'VOD -
3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF
1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶆXPSLRVFVF"EE -
3FqFDUPS %FMUB'*'0 RVFVF 3FTPVSDF &WFOU )BOEMFS JONFNPSZDBDIF TUPSF *OEFYFS 8PSL2VFVF
1SPDFTT *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Dequeue~ 8BUDI ᶇXPSLRVFVF(FU -
Appendix) Informer enqueue ιʔε ᶃ https://github.com/kubernetes/client-go/blob/master/tools/cache/reflector.go#L267 ᶄ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L153 ᶅ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L198
https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L153 ※ ReplicaSet Controllerͷ߹ ᶆ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L417 ※ ReplicaSet Controllerͷ߹ ᶇ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L438 ※ ReplicaSet Controllerͷ߹ -
Appendix) InformerͱResync Period Informerىಈ࣌ʹɺResync Periodͱ͍͏ҾΛࢦఆ͢Δ Informerapi-server͔ΒObjectͷมߋΛࢹ͍ͯ͠Δ Resync PeriodΛա͗ΔͱɺԿͷEvent͕ൃੜ͍ͯ͠ͳ͍ͯ͘ UpdateFunc͕ݺͼग़͞Εͯɺ݁ՌReconcile͕࣮ߦ͞ΕΔ ※͜ͷ࣌ɺResyncin-memory-cacheΛࢀর͢Δɻ
Resync(cacheͷࢀর)ͱRelist(api-serverͷ͍߹Θͤ)ผ informer.Start SFTZODQFSJPE List Watch Event Reflector Added Updated Updated Added AddFunc UpdateFunc AddFunc UpdateFunc Handler -
*OGPSNFS 1SPDFTT *UFN 3FTPVSDF &WFOU )BOEMFS 8PSL2VFVF ControllerͷCycle Informer +
WorkQueue + Controller ͷશମͷCycleΛ֬ೝ $POUSPMMFS ϝΠϯϩδοΫ -
*OGPSNFS 1SPDFTT *UFN 3FTPVSDF &WFOU )BOEMFS ControllerͷCycle XBUDI 8PSL2VFVF ʲԾఆʳ
Pod͕͢Ͱʹ2ͭ͋Δ ͔͜͜ΒUpdate Event͕ൃੜ͢Δ -
*OGPSNFS 1SPDFTT *UFN 3FTPVSDF &WFOU )BOEMFS ControllerͷCycle XBUDI BEE'VOD VQEBUF'VOD
EFMFUF'VOD ᶄ 8PSL2VFVF -
*OGPSNFS 1SPDFTT *UFN 3FTPVSDF &WFOU )BOEMFS ControllerͷCycle XBUDI ᶈXPSL2VFVF'PSHFU ᶉXPSL2VFVF%POF
Reconcile͕ਖ਼ৗऴྃͨ͠Β ΞΠςϜΛ8PSL2VFVF͔Βআ֎ 8PSL2VFVF -
*OGPSNFS 1SPDFTT *UFN 3FTPVSDF &WFOU )BOEMFS ControllerͷCycle XBUDI ᶇ`XPSL2VFVF"EE3BUF-JNJUFE Reconcile͕Τϥʔ
8PSL2VFVFʹ3FRVFVFͯ͠ɺ3FDPODJMFΛ࠶ࢼߦ 8PSL2VFVF -
*OGPSNFS 1SPDFTT *UFN 3FTPVSDF &WFOU )BOEMFS ControllerͷCycle XBUDI ᶇ`XPSL2VFVF"EE3BUF-JNJUFE Reconcile͕Τϥʔ
8PSL2VFVFʹ3FRVFVFͯ͠ɺ3FDPODJMFΛ࠶ࢼߦ 8PSL2VFVF -
Controllerͷجຊઓུ ReadIn-memory-cache͔ΒɺWriteapi-server ※cacheΛ͍͡ΔͱҰ؏ੑΛอͭͷ͕͍͠ͷͰɺ Write͢Δͱ͖ɺҰճDeepCopy͔ͯ͠Βߋ৽͢Δ e.g. kubernetes/pkg/controller/replicaset/replica_set.go rs = rs.DeepCopy() newStatus
:= calculateStatus(rs, filteredPods, manageReplicasErr) // Always updates status as pods come up or die. updatedRS, err := updateReplicaSetStatus(rsc.kubeClient.AppsV1(). ReplicaSets(rs.Namespace), rs, newStatus) https://github.com/kubernetes/kubernetes/blob/release-1.15/pkg/controller/replicaset/replica_set.go#L611 -
XPSLFS XPSLFS QSPDFTT/FYU8PSL*UFN TZOD)BOEMFS ControllerͷMain Logic worker: processNextWorkItemΛແݶϧʔϓ processNextWorkItem: WorkQueueΛૢ࡞(Get,
Add)ͯ͠ ReconcileΛݺͼग़͢ syncHandler: Reconcileʹ͋ͨΔϝΠϯϩδοΫ Add TZOD)BOEMFS Update Delete Event Reconcile Eventͷछྨʹ͔͔ΘΒͣɺ Reconcile͢Δ -
Appendix) ReplicaSet Controller worker: processNextWorkItemΛແݶϧʔϓ processNextWorkItem: WorkQueueΛૢ࡞(Get, Add)ͯ͠ ReconcileΛݺͼग़͢ syncReplicaSet:
Reconcileʹ͋ͨΔϝΠϯϩδοΫ XPSLFS XPSLFS QSPDFTT/FYU8PSL*UFN TZOD3FQMJDB4FU https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L432 https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L437 https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L562 ReplicaSet Controllerͷιʔε v1.16 ※DeploymentྨࣅͷߏΛ͍ͯ͠Δ -
༻ޠཧ(ৼΓฦΓ) Informer: ObjectͷEventΛࢹ͠ɺin-memory-cacheʹσʔλΛ֨ೲ͢Δ EventHandlerΛ௨ͯ͡ɺWorkQueueʹΞΠςϜΛՃ͢Δ Lister: ObjectΛऔಘ͍ͨ࣌͠ in-memory-cache͔ΒσʔλΛऔಘ͢Δ WorkQueue: Controller͕ॲཧ͢ΔΞΠςϜΛొ͓ͯ͘͠Ωϡʔ ͜ͷΩϡʔʹஷ·͍ͬͯΔΞΠςϜ͕Reconcileͷର
ReconcileͰΤϥʔ͕ൃੜͨ͠߹ɺRequeue͞Εɺ ControllerଆͰReconcileΛ࠶ࢼߦ͢Δ -
Controllerͷ·ͱΊ ɾControllerɺControl Loop(Reconciliation Loop)ʹΑͬͯ એݴతͳAPIΛ࣮ݱ ɾࢄίϯϙʔωϯτͦΕͧΕಠཱ͓ͯ͠Γɺ Event͕ίϯϙʔωϯτؒΛ݁ͼ͚ͭΔ ɾclient-go, apimachinery, code-generator͕ControllerΛࢧ͑Δ
ɾInformerೋͭͷతΛ࣋ͭ ᶃ in-memory-cacheʹObjectͷσʔλΛ֨ೲ͢Δ ᶄ EventHandlerܦ༝ͰWorkQueueʹΞΠςϜΛ֨ೲ͢Δ ɾWorkQueueʹஷΊΒΕͨΞΠςϜReconcileͰॲཧ͞ΕΔ ※ ͜ΕΒΛΒͳͯ͘ɺControllerར༻Ͱ͖·͢ -
Deep Diveͷಓ ʙ ͬͱਂ͘Γ͍ͨਓ͚ʹ ʙ ɾSample ControllerΛಈ͔͢ ɾKubernetesຊମͷ࣮ΛݟͯΈΔ - Deployment
Controller - ReplicaSet Controller ɾCustom ControllerΛ࡞ͬͯΈΔ(+ CRD) ϦϯΫ: https://github.com/kubernetes/sample-controller mkdir -p $GOPATH/src/k8s.io && cd $GOPATH/src/k8s.io && git clone https://github.com/kubernetes/sample-controller.git export GO111MODULE=on go build -o sample-controller . ./sample-controller -kubeconfig $HOME/.kube/config https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/deployment/deployment_controller.go https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go -
Programming Kubernetes https://programming-kubernetes.info/ հϒϩάࢀߟ: https://go-vargo.hatenablog.com/entry/2019/08/05/201546 O’Reilly Media, Inc.ΑΓग़൛ ʮProgramming Kubernetesʯ
Stefan Schimanski, Michael Hausenblasஶ 1~2ষ: Kubernetesͷجຊࢥ, API Object 3ষ: client-go 4ষ: CRD 5ষ: code-generator 6~7ষ: Custom Controller 8~9ষ: Custom API Server, CRDԠ༻ػೳ -
Reference ɾWeb Article - https://kubernetes.io/docs/concepts/architecture/nodes/ - https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/ - https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md -
A deep dive into Kubernetes controllers (https://engineering.bitnami.com/articles/a-deep-dive-into-kubernetes-controllers.html) - Core Kubernetes: Jazz Improv over Orchestration (https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca) - Events, the DNA of Kubernetes(https://www.mgasch.com/post/k8sevents/) ɾPresentation - Kubernete Meetup Tokyo #18 - Kubebuilder/controller-runtime ೖ (https://www.slideshare.net/pfi/kubernete-meetup-tokyo-18-kubebuildercontrollerruntime) - KubernetesͷιʔείʔυϦʔσΟϯάೖ (https://speakerdeck.com/smatsuzaki/kubernetesfalsesosukodorideinguru-men) ɾBook - Programming Kubernetes (https://programming-kubernetes.info/) -
Reference ɾRepository - Kubernetes(https://github.com/kubernetes/kubernetes) - Sample Controller(https://github.com/kubernetes/sample-controller) - client-go(https://github.com/kubernetes/client-go) -
apimachinery(https://github.com/kubernetes/apimachinery) - codegenerator(https://github.com/kubernetes/code-generator) - what-happens-when-k8s(https://github.com/jamiehannaford/what-happens-when-k8s)