ゼロから始めるKubernetes Controller / Under the Kubernetes Controller

Transcript

1. θϩ͔Β࢝ΊΔ Kubernetes Controller 2019/09/27 Kubernetes Meetup #23 Operator Deep Dive
2. ࠓ೔ͷ໨ඪɾ࿩͢͜ͱ ɾKubernetesͷResourceΛ͋Δఔ౓஌͍ͬͯΔਓ͕ɺɹ ɹControllerͷ஌ࣝθϩͷঢ়ଶ͔ΒͰ΋Deep DiveʹΑͬͯɺ ɹControllerͷ࢓૊Έɾ಺෦࣮૷ʹ͍ͭͯཧղͰ͖Δ͜ͱ ɾKubernetes Controllerͷ֓ཁɾࢥ૝ ɾKubernetes Controllerͷ࢓૊Έ, ಺෦࣮૷ ɾControllerΛࢧ͑Δίϯϙʔωϯτ = ControllerͷϋΠϨϕϧʙϩʔϨϕϧ·Ͱ ࿩͢͜ͱ ໨ඪ
3. ࿩͞ͳ͍͜ͱ ɾKubernetes Custom Controller + CRDͷৄࡉ ※ ͨͩ͠ɺ७ਖ਼Controllerͷ࢓૊ΈΛ஌Δ͜ͱ͸ɺ Custom ControllerΛ࣮૷͢Δ্Ͱ΋໾ཱͪ·͢ ɾKubernetes Custom ControllerΛ࣮૷͢ΔͨΊͷϑϨʔϜϫʔΫ KubebuilderɺOperator SDKͷৄࡉʹ͍ͭͯ͸࿩͠·ͤΜ ಺෦SDKͰ͋Δcontroller-runtime΍controller-toolsʹ͍ͭͯ΋ ຊൃදͰ͸৮Ε·ͤΜ ࿩͞ͳ͍͜ͱ controller-runtimeࢀߟ: https://www.slideshare.net/pfi/kubernete-meetup-tokyo-18-kubebuildercontrollerruntime
4. ΞδΣϯμ ɾ Kubernetes Controllerͱ͸ʁ ɾ Control Loop(Reconciliation Loop) ɾControllerΛࢧ͑Δɺ஌͓ͬͯ͘΂͖ίϯϙʔωϯτ ɾClient-Go - Informer - WorkQueue ɾControllerͷCycle, Main Logic ɾControllerͷ·ͱΊ - Reference
5. Kubernetes Controllerͱ͸ʁ ʙ ϋΠϨϕϧΞʔΩςΫνϟ ʙ
6. .BTUFS 8PSLFS 8PSLFS 8PSLFS KubernetesΞʔΩςΫνϟ͓͞Β͍ Master Node Worker Node
7. KubernetesϋΠϨϕϧΞʔΩςΫνϟ .BTUFS 8PSLFS FUDE DMPVEDPOUSPMMFS NBOBHFS /PEF $POUBJOFS
   3VOUJNF LVCFMFU
   
   
   LVCF
   QSPYZ DPOUSPMMFS NBOBHFS BQJTFSWFS TDIFEVMFS /PEF $POUBJOFS
   3VOUJNF LVCFMFU /PEF $POUBJOFS
   3VOUJNF LVCFMFU DMPVE
   
   
   LVCF
   QSPYZ
   
   
   LVCF
   QSPYZ
8. KubernetesͷΞʔΩςΫνϟ ɾKubernetes͸෼ࢄΞʔΩςΫνϟɾ෼ࢄίϯϙʔωϯτ Master: ೝূɾೝՄ΍Resource ͷ؅ཧɺ ίϯςφͷεέδϡʔϦϯάͳͲͷશମ؅ཧ Worker: ίϯςφͷ࣮ߦ෦෼
9. KubernetesϋΠϨϕϧΞʔΩςΫνϟ .BTUFS 8PSLFS FUDE DMPVEDPOUSPMMFS NBOBHFS /PEF $POUBJOFS
   3VOUJNF LVCFMFU
   
   
   LVCF
   QSPYZ DPOUSPMMFS NBOBHFS BQJTFSWFS TDIFEVMFS /PEF $POUBJOFS
   3VOUJNF LVCFMFU /PEF $POUBJOFS
   3VOUJNF LVCFMFU DMPVE
   
   
   LVCF
   QSPYZ
   
   
   LVCF
   QSPYZ
10. api-server / controller-manager api-server: Deployment΍ServiceͳͲͷObjectͷ࡞੒ɾߋ৽ɾ࡟আ (CRUD)ͷϦΫΤετΛड͚औΓɺૢ࡞Λߦ͏ Objectͷ৘ใΛσʔλετΞͰ͋Δetcdʹ֨ೲ͢Δ ※etcdʹΞΫηε͢Δίϯϙʔωϯτ͸api-serverͷΈ controller-manager: Controller͸ɺDeployment΍ServiceͳͲͷObjectͷ ؅ཧΛߦ͏ɻ controller-manager͸ɺController܈ͷ͔ͨ·Γ
11. controller-manager controller-manager: Ұͭͷϓϩηεʹෳ਺ͷController %FQMPZNFOU 3FQMJDB4FU %BFNPO4FU 4FSWJDF +PC $SPO+PC &OEQPJOU 4UBUFGVM4FU ʜ
12. 3FQMJDB4FU
    $POUSPMMFS LJOE
    3FQMJDB4FU
    NFUBEBUB
    
    
    
    
    OBNF
    YYYYYY
    TQFD
    ʜ 3FTPVSDF
     3FTPVSDF
     3FTPVSDF
    ʜ 3FTPVSDF
    / ؅ཧ $POUSPM
    -PPQ .BOJGFTU ControllerͱResource Controller͕Resourceͷ؅ཧΛߦ͏ 3FQMJDB4FU
    $POUSPMMFSͷྫ
13. 3FQMJDB4FU
    $POUSPMMFS ControllerͱResource Controller͸ɺͻͱͭͷResourceΛ؅ཧ͢Δ 3FQMJDB4FU 1PE %FQMPZNFOU
    $POUSPMMFS %FQMPZNFOU NBOBHF DSFBUF NBOBHF DSFBUF ࢀߟ: OwnerReference https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/ ্ҐResource͕ԼҐResourceΛ؅ཧ͢Δ࢓૊Έͱͯ͠ʮOwnerReferenceʯ͕͋Δ ্ҐResourceΛ࡟আ͢ΔͱɺԼҐResource͸ΨϕʔδίϨΫγϣϯͰ࡟আ͞ΕΔ
14. Control Loop (Reconciliation Loop)
15. "DUVBM
    4UBUF $POUSPM
    -PPQ %FTJSFE
    4UBUF 3FBE
    SFBM
    SFTPVSDFT $IBOHF
    JOUFSOBMFYUFSOBM
    SFTPVSDFT PCTFSWF BOBMZ[F "DU Controllerͷجຊݪཧ جຊݪཧ: Control Loop(Reconciliation Loop)
16. Control Loop(Reconciliation Loop) Controller Loop͸ɺControllerͷجຊݪཧ ※Reconciliation Loopͱ΋ݺ͹Ε͍ͯΔ Controller LoopͷྲྀΕ: 1. Resourceͷݱࡏͷঢ়ଶΛಡΈࠐΉ 2. ResourceΛ๬Μͩঢ়ଶʹมߋ͢Δ 3. Resourceͷঢ়ଶ(Status)Λߋ৽͢Δ Controller LoopʹΑͬͯɺ๬Μͩঢ়ଶΛએݴ͢Δ͜ͱͰ ؀ڥ͕ߏங͞ΕΔɻ = Immutable Infrastructure͕࣮ݱͰ͖Δ ϧʔϓ
17. 3FQMJDBT
     ReplicaSetͷControl Loopྫ 3FQMJDBT 3FQMJDB4FU
    $POUSPMMFS 3FQMJDB4FU
    $POUSPMMFS 0CTFSWF ๬Μͩঢ়ଶ ݱࡏͷঢ়ଶ "DU
18. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ LJOE
    3FQMJDB4FU
    NFUBEBUB
    
    
    
    
    OBNF
    YYYYYY
    TQFD
    ʜ .BOJGFTU 6TFS .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS ,VCFDUM
    BQQMZ
    G
    NBOJGFTUZBNM †
    3FTPVSDFΛ"QQMZ͢Δ 3FQMJDB4FU
    $POUSPMMFS ᶃ
    3FQMJDB4FU͕ੜ੒͞ΕΔ
19. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF
    LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU
    $POUSPMMFS $POUSPM
    -PPQ ᶄ
    3FQMJDB4FU͕ੜ੒͞Εͨ͜ͱΛɺ3FQMJDB4FU
    $POUSPMMFS͕ݕ஌ ᶄ`
    ϝΠϯϩδοΫʹΑͬͯɺ
    
    
    
    
    
    TQFDOPEF/BNF͕ۭͷ1PEΛ࡞੒͢Δ DSFBUF ˞8PSLFSʹ͸·ͩσϦόϦ͞Εͳ͍
20. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF
    LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU
    $POUSPMMFS ᶅ
    4DIFEVMFS͕1PEͷ࡞੒Λݕ஌ queue ᶅ`
    1PEͷTQFDOPEF/BNF͕ۭͳͷͰɺ
    
    
    
    
    
    TDIEVMFS͕RVFVFʹ௥Ճ͢Δ
21. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF
    LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU
    $POUSPMMFS ᶆ
    LVCFMFU͕1PEͷ࡞੒Λݕ஌ queue ᶆ`
    1PEͷTQFDOPEF/BNF͕ۭͳͷͰɺ
    
    
    
    
    
    LVCFMFU͸ॲཧΛεΩοϓ skip
22. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF
    LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU
    $POUSPMMFS ᶇ
    4DIFEVMFS͕1PEΛRVFVF͔ΒऔΓग़͢ queue ᶇ`
    ΞαΠϯՄೳͳ/PEFʹ1PEΛ
    
    
    
    
    
    εέδϡʔϦϯά͢Δ schedule ᶇ`
    1PEͷTQFDOPEF/BNFΛߋ৽ 6QEBUF TQFDOPEF/BNF
23. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF
    LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU
    $POUSPMMFS queue ᶈ
    LVCFMFU͕1PEͷߋ৽Λݕ஌ ᶈ`
    LVCFMFU͕ίϯςφΛىಈ
24. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF
    LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU
    $POUSPMMFS queue ᶉ
    LVCFMFU͕BQJTFSWFSʹϦΫΤετ͠ɺ
    
    
    
    
    1PEͷ4UBUVTΛߋ৽ TUBUVTDPOEJUJPOT
25. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF
    LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU
    $POUSPMMFS queue ͜͜Ͱɺίϯςφ͕ͳΜΒ͔ͷཧ༝Ͱࢮ๢ͨ͠ͱ͠·͢
26. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF
    LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU
    $POUSPMMFS queue 1PEͷ5FSNJOBUJOH TUBUVTDPOEJUJPOT ᶊ
    LVCFMFU͕BQJTFSWFSʹϦΫΤετ͠ɺ
    
    
    
    
    1PEͷ4UBUVTΛߋ৽
27. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF
    LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU
    $POUSPMMFS queue 1PEͷ5FSNJOBUJOH ᶋ
    3FQMJDB4FU
    $POUSPMMFS͕1PEͷߋ৽Λ
    
    
    
    
    ݕ஌ EFMFUF ᶋ`
    3FQMJDB4FU
    $POUSPMMFS͕1PEΛ࡟আ
28. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ .BTUFS 8PSLFS FUDE /PEF $POUBJOFS
    3VOUJNF
    LVCFMFU
    
    
    LVCF
    QSPYZ BQJTFSWFS TDIFEVMFS 3FQMJDB4FU
    $POUSPMMFS queue 3FDPODJMF ᶌ
    3FQMJDB4FU
    $POUSPMMFS͕3FDPODJMF͠ɺ
    
    
    
    
    ๬Μͩঢ়ଶ 3FQMJDBT ʹ෮چ͠ɺ
    
    
    
    
    1PE͕৽͘͠࡞੒͞ΕΔ EFMFUF Ҏ߱ɺϧʔϓʜ
    ᶄʙᶌͷॲཧΛ܁Γฦ͢
29. Appendix) ιʔε ᶄ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L487 ᶅ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/eventhandlers.go#L436 ᶆ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/kubelet/config/apiserver.go#L33 ᶇ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/scheduler.go#L535 ᶈ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L803 ᶉ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kubelet.go#L1527 ᶊ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kubelet.go#L2006 ᶋ https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/controller/replicaset/replica_set.go#L535 ᶌ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L487
30. Controllerͱίϯϙʔωϯτͨͪ ֤ίϯϙʔωϯτͷಇ͖ ɾapi-server: ResourceͷCRUD ɾscheduler: ResourceͷεέδϡʔϦϯά ɾkubelet: ίϯςφͷىಈ ɾcontroller: ResourceͷReconcile(ௐ੔) ֤ίϯϙʔωϯτ͸ͦΕͧΕͷ੹຿ʹूத = શମΛ౷཰ɾ౷੍͢Δࢦشऀ͸͍ͳ͍
31. Controllerͱௐ࿨ Kubernetes͸ͦΕͧΕͷίϯϙʔωϯτ͕ڠௐͯ͠ಈ͍͍ͯΔ Θ͚Ͱ͸ͳ͍ ໌ࣔతʹ໋ྩΛ͍ͯ͠ͳ͍ͷʹɺશମͱͯ͠੔߹ੑ͕औ Ε͍ͯΔ ͜͜Ͱ… ίϯϙʔωϯτͨͪΛɺͦΕͧΕҰͭͷControllerͱߟ͑Δ ͦΕͧΕͷίϯτϩʔϥ͸ɺ ͦΕͧΕͷControl LoopΛ࣮ߦ͢Δ͜ͱ͚ͩʹूத͢Δ = ෆࢥٞͱɺશମͱͯ͠ͷ੔߹ੑ͕औΕ͍ͯΔ
32. Kubernetes is jazz improv OrchestrationͰ͸ͳ͘ɺ Player(Controller)͕֤ʑͷԋ૗ (Control Loop)ʹूத͢Δ͜ͱͰ ଈڵͷԻָͱͯ͠ɺશମ͕੒Γཱ͍ͬͯΔ Kubernetes is more jazz improv than orchestration. Joe Beda https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca Co-founder Core Kubernetes: Jazz Improv over Orchestration
33. ͳͥControl LoopΛߦ͏ͷ͔ʁ Control LoopΛߦ͏ཧ༝Λߟ͑Δ࣌ʹɺॏཁʹͳΔ΋ͷ͕ ʮEventʯ ※ e.g. Added, Modified, Deleted, Error… ෼ࢄίϯϙʔωϯτͰ੒ΓཱͭKubernetesʹͱͬͯɺ Event͸ɺͦΕͧΕͷίϯϙʔωϯτΛͭͳ͙ॏཁͳ΋ͷ ࢀߟ: Events, the DNA of Kubernetes EventΛͲͷΑ͏ʹѻ͏͔ɺ伴ͱͳΔߟ͑ํ͕ ɾEdge-driven Triggers ɾLevel-driven Triggers https://www.mgasch.com/post/k8sevents/
34. Appendix) Edge vs. Level https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d ࢀߟ: Level Triggering and Reconciliation in Kubernetes Edge-driven Triggers Level-driven Triggers EventͷൃੜʹԠͯ͡Trigger ಛఆͷঢ়ଶʹͳΔͱTrigger up down low high low
35. ͳͥControl LoopΛߦ͏ͷ͔ʁ ͳͥController͕Control LoopΛߦ͏ͷ͔Λߟ͑ΔͨΊʹɺ Kubernetes͕એݴతͳڍಈ(Reconcile)Ͱ͸ͳ͘ɺ खଓ͖తͳڍಈ(Procedure)Λ͢ΔͱԾఆ LJOE
    YYYYYYYYY
    NFUBEBUB
    
    
    
    
    OBNF
    YYYYYY
    TQFD
    ʜ ࣮ࡍͷKubernetes Ծఆ Reconcile Procedure
36. ΋͠΋Controller͕खଓ͖ܕͩͬͨΒ ݱࡏͷ3FQMJDBT਺   ˢ
    ࡞ ੒  ˢ
    ࡟ ݮ खଓ͖తʹಈͨ͘ΊɺEvent͕ൃੜ͢Δͨͼʹ Edge-driven Triggerͱͯ͠ಈ͘ Ұݟ໰୊ͳ͍Α͏ʹݟ͑Δ͕ɺऑ఺΋͋Δ &WFOU
37. ΋͠΋Reconcile͕ͳ͔ͬͨΒ ݱࡏͷ3FQMJDBT਺   ˢ
    ࡞ ੒ ˢ
    ࡟ ݮ Ұ࣌తͳωοτϫʔΫͷෆௐ΍όάͳͲͷো֐͕͋ͬͨ࣌ʹ Eventͷ৘ใ͕ࣦΘΕͯ͠·͏ ো ֐ &WFOU
38. Resync IntervalͱReconcile ݱࡏͷ3FQMJDBT਺   ˢ
    ࡞ ੒  ˢ
    ࡟ ݮ Resync Interval͝ͱʹReconcile(ௐ੔)Λ͢Δ͜ͱͰɺ Event͕ࣦΘΕͯ΋ཧ૝తͳঢ়ଶʹ͚ۙͮΔ ো ֐ SFTZOD
    JOUFSWBM &WFOU Kubernetes = Edge-driven Trigger + Level-driven Trigger
39. ControllerΛࢧ͑Δɺ ஌͓ͬͯ͘΂͖ίϯϙʔωϯτ ʙ ϛυϧϨϕϧΞʔΩςΫνϟ ʙ
40. ༻ޠͷ੔ཧ Kind: API Objectͷछྨ(Deployment, ServiceͳͲ) 
 Resource: Kindͱಉ͡ҙຯ߹͍Ͱ༻͍ΒΕΔɻ HTTP Endpointͱͯ͠΋ར༻͞ΕΔɻ খจࣈ͔ͭෳ਺ܗͰදݱ͞ΕΔ(pods, servicesͳͲ) Object: ࡞੒͞ΕͨAPI Object ͷ࣮ମ Metadata΍Spec, StatusͳͲͷσʔλΛอ࣋͢Δ
41. ControllerΛࢧ͑ΔϥΠϒϥϦ client-go Informer Lister WorkQueue api-machinery runtime.Object Scheme code-generator ϥΠϒϥϦ ίϯϙʔωϯτ লུ
42. Appendix) Custom Controller SDK Kubebuilder Informer ϑϨʔϜϫʔΫ ίϯϙʔωϯτ Lister Operator SDK ϥΠϒϥϦ (ϋΠϨϕϧ) controller-runtime controller-tools ϥΠϒϥϦ (ϩʔϨϕϧ) client-go api-machinery etc… Scheme runtime.Object WorkQueue etc…
43. ControllerΛࢧ͑ΔϥΠϒϥϦ client-go: KubernetesͷެࣜΫϥΠΞϯτϥΠϒϥϦ Kubernetesຊମͷ։ൃʹ΋࢖ΘΕ͍ͯΔ Controller࡞੒ʹ͸͔ܽͤͳ͍ϥΠϒϥϦ api-machinery: Kubernetes API Object & Kubernetes API like Object ʹඞཁͳػೳΛඋ͑ͨϥΠϒϥϦ e.g. conversion, decode, encode, etc… Controller͸API ObjectΛѻ͏ͷͰɺඞཁʹͳΔ code-generator: Informer, Lister, clientset, DeepCopyͳͲͷίʔυΛੜ੒
44. ControllerΛࢧ͑Δίϯϙʔωϯτ ͦΕͧΕͷৄࡉ͸ɺControllerͷ࢓૊ΈΛࢀরͭͭ͠ޙड़͠·͢ɻ Informer: ObjectͷEventΛ؂ࢹ͠ɺin-memory-cacheʹσʔλΛ֨ೲ͢Δ Lister: in-memory-cache͔ΒσʔλΛऔಘ͢Δ WorkQueue: Controller͕ॲཧ͢ΔΞΠςϜΛొ࿥͓ͯ͘͠Ωϡʔ runtime.Object: શͯͷAPI Objectڞ௨ͷInterface Scheme: Kubernetes APIͱGo TypeΛඥ෇͚Δػߏ ࠓճ͸આ໌লུ
45. client-go Informer ʙ ϩʔϨϕϧΞʔΩςΫνϟ ʙ
46. client-goͱInformer client-go Informer ϥΠϒϥϦ ίϯϙʔωϯτ Reflector DeltaFIFO Indexer Store Lister
47. Informer Informer͸Objectͷมߋ(Event)Λ؂ࢹ͢Δίϯϙʔωϯτ ObjectͷมߋΛ؂ࢹ͢ΔͨΊʹɺapi-serverʹঢ়ଶΛຖճ ໰͍߹ΘͤΔͱɺapi-serverʹෛՙ͕͔͔Δ ὎ in-memory-cacheʹObjectͷσʔλΛ֨ೲ͠ɺ ͦͷcacheΛࢀর͢Δ͜ͱͰapi-server΁ͷෛՙΛܰݮͤ͞Δ ղܾࡦΛ࠾༻ $POUSPMMFS $POUSPMMFS *OGPSNFS in-memory-cache watch watch
48. Appendix) InformerΛ࢖࣮ͬͨ૷ func main() { ... clientset, err := kubernetes.NewForConfig(config) // Create InformerFactory informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30) // Create pod informer by informerFactory podInformer := informerFactory.Core().V1().Pods() // Add EventHandler to informer podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(new interface{}) { log.Println("Added") }, UpdateFunc: func(old, new interface{}) { log.Println("Updated") }, DeleteFunc: func(old interface{}) { log.Println("Deleted") }, }) // Start Go routines informerFactory.Start(wait.NeverStop) // Wait until finish caching with List API informerFactory.WaitForCacheSync(wait.NeverStop) // Create Pod Lister podLister := podInformer.Lister() // Get List of pods _, err = podLister.List(labels.Nothing()) … } https://github.com/govargo/kubecontorller-book-sample-snippet/blob/master/02/podinformer/podinformer.go ᶃ kubeconfig͔ΒconfigΛੜ੒ ᶄ config͔ΒclientsetΛੜ੒ ᶅ informerFactoryੜ੒ ᶆ ᶅΛ࢖ͬͯinformerੜ੒ ᶇ EventHandlerΛ௥Ճ Add, Update, Delete࣌ʹൃՐ ᶈ InformerΛىಈ ᶉ Lister(Getter)Λੜ੒ ᶊ ListerΛ࢖ͬͯGet΍List
49. Appendix) Shared Informer InformerΛར༻͢Δͱ͖͸ɺInformerͦͷ΋ͷͰ͸ͳ͘ɺ Shared Informerͱݺ͹ΕΔ΋ͷΛར༻͢Δ Shared Informer͸ɺͻͱͭͷόΠφϦ಺Ͱಉ͡ResourceΛڞ༗ Ͱ͖Δ %FQMPZNFOU 3FQMJDB4FU %BFNPO4FU 4FSWJDF +PC ʜ 4IBSFE
    *OGPSNFS
    
    TIBSFE
    DBDIF
    GPS
    TBNF
    SFTPVSDF kube-controller-manager
50. https://github.com/kubernetes/sample-controller/blob/master/docs/controller-client-go.md Ҿ༻ݩ: InformerͱWorkQueue֓؍
51. 3FqFDUPS %FMUB'*'0
    RVFVF -JTU
    
    8BUDI JONFNPSZDBDIF 4UPSF *OEFYFS -JTUFS (FU
    PS
    -JTU FH
    DPSFW1PE (FU
    PS
    -JTU
    3FRVFTU *OGPSNFS )BOEMF%FMUBT ৄࡉInformer
52. 3FqFDUPS %FMUB'*'0
    RVFVF ᶃ-JTU"OE8BUDI JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT ৄࡉInformer ~σʔλΛΩϟογϡ~
53. 3FqFDUPS %FMUB'*'0
    RVFVF ᶄ1PQ JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI ৄࡉInformer ~σʔλΛΩϟογϡ~
54. 3FqFDUPS %FMUB'*'0
    RVFVF ᶅ)BOEMF%FMUB JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI ৄࡉInformer ~σʔλΛΩϟογϡ~
55. 3FqFDUPS %FMUB'*'0
    RVFVF ᶆJOEFYFS"EE JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI ৄࡉInformer ~σʔλΛΩϟογϡ~
56. 3FqFDUPS %FMUB'*'0
    RVFVF JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI ᶅ)BOEMF%FMUB ৄࡉInformer ~σʔλΛΩϟογϡ~ ᶄ1PQ
57. 3FqFDUPS %FMUB'*'0
    RVFVF JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI ᶆJOEFYFS"EE ৄࡉInformer ~σʔλΛΩϟογϡ~
58. 3FqFDUPS %FMUB'*'0
    RVFVF JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI ৄࡉInformer ~σʔλΛऔΓग़͢~ -JTUFS (FU
    PS
    -JTU
    3FRVFTU
59. 3FqFDUPS %FMUB'*'0
    RVFVF JONFNPSZDBDIF 4UPSF *OEFYFS *OGPSNFS )BOEMF%FMUBT 8BUDI JOEFYFS(FU#Z,FZ
    PS
    JOEFYFS-JTU ৄࡉInformer ~σʔλΛऔΓग़͢~ -JTUFS (FU
    PS
    -JTU
    3FRVFTU
60. Appendix) Informer cache ιʔε ᶃ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/reflector.go#L188 ᶄ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L153 ᶅ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/shared_informer.go#L455 ᶆ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/shared_informer.go#L464
61. Informerͱίϯϙʔωϯτ Informer: ObjectͷEventΛ؂ࢹ͠ɺin-memory-cacheʹσʔλΛ֨ೲ͢Δ Reflector: api-serverͷEventΛ؂ࢹ͢Δ DeltaFIFO: ObjectͷEvent͕ൃੜ͢Δͨͼʹ࢖͏FIFOΩϡʔ Indexer: in-memory-cache΁ͷॻ͖ࠐΈ΍ಡΈࠐΈΛߦ͏ Store: in-memory-cache Lister: in-memory-cache͔ΒIndexerܦ༝ͰσʔλΛऔಘ͢Δ
62. client-go WorkQueue ʙ ϩʔϨϕϧΞʔΩςΫνϟ ʙ
63. client-goͱWorkQueue client-go WorkQueue ϥΠϒϥϦ ίϯϙʔωϯτ RateLimitingQueue DelayedQueue
64. WorkQueue WorkQueue͸ɺDeltaFIFOͱ͸ҟͳΔ΋͏ͻͱͭͷΩϡʔ WorkQueue͸ɺControl LoopͷΞΠςϜΛཷΊ͓ͯͨ͘Ίʹ ࢖ΘΕΔ ඪ४ControllerͰ͸ɺInformerͷEventHandler͕ൃՐ͢Δͱɺ WorkQueueʹΞΠςϜΛenqueue͢Δ࣮૷ʹͳ͍ͬͯΔ Event $POUSPMMFS Added Updated Deleted WorkQueue
65. Appendix) WorkQueueΛ࢖࣮ͬͨ૷ func main() { ... clientset, err := kubernetes.NewForConfig(config) // Create InformerFactory informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30) // Create pod informer by informerFactory podInformer := informerFactory.Core().V1().Pods() // Create RateLimitQueue queue := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter()) // shutdown when process ends defer queue.ShutDown() // Add EventHandler to informer podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(old interface{}) { var key string var err error if key, err = cache.MetaNamespaceKeyFunc(old); err != nil { runtime.HandleError(err) return } queue.Add(key) log.Println("Added: " + key) }, UpdateFunc: func(old, new interface{}) { … }, DeleteFunc: func(old interface{}) { … }, }) … } https://github.com/govargo/kubecontorller-book-sample-snippet/blob/master/02/workqueue/enqueuePod.go ᶃ kubeconfig͔ΒconfigΛੜ੒ ᶄ config͔ΒclientsetΛੜ੒ ᶅ informerFactoryੜ੒ ᶆ ᶅΛ࢖ͬͯinformerੜ੒ ᶇ EventHandlerΛ௥Ճ Add, Update, Delete࣌ʹൃՐ ᶈ InformerΛىಈ ᶉ Lister(Getter)Λੜ੒ ᶊ ListerΛ࢖ͬͯGet΍List ᶇ WorkQueueੜ੒
66. 3FqFDUPS %FMUB'*'0
    RVFVF -JTU
    
    8BUDI 3FTPVSDF
    &WFOU
    )BOEMFS JONFNPSZDBDIF
    TUPSF *OEFYFS 8PSL2VFVF 1SPDFTT
    *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue
67. 3FqFDUPS %FMUB'*'0
    RVFVF 3FTPVSDF
    &WFOU
    )BOEMFS JONFNPSZDBDIF
    TUPSF *OEFYFS 8PSL2VFVF 1SPDFTT
    *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ &WFOU -JTU
    
    8BUDI
68. 3FqFDUPS %FMUB'*'0
    RVFVF 3FTPVSDF
    &WFOU
    )BOEMFS JONFNPSZDBDIF
    TUPSF *OEFYFS 8PSL2VFVF 1SPDFTT
    *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ᶃ8BUDI ৄࡉWorkQueue ~Enqueue~
69. 3FqFDUPS %FMUB'*'0
    RVFVF 3FTPVSDF
    &WFOU
    )BOEMFS JONFNPSZDBDIF
    TUPSF *OEFYFS 8PSL2VFVF 1SPDFTT
    *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶄ1PQ
70. 3FqFDUPS %FMUB'*'0
    RVFVF 3FTPVSDF
    &WFOU
    )BOEMFS JONFNPSZDBDIF
    TUPSF *OEFYFS 8PSL2VFVF 1SPDFTT
    *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶅ
    "EE'VOD
    6QEBUF'VOD
    %FMFUF'VOD
71. 3FqFDUPS %FMUB'*'0
    RVFVF 3FTPVSDF
    &WFOU
    )BOEMFS JONFNPSZDBDIF
    TUPSF *OEFYFS 8PSL2VFVF 1SPDFTT
    *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI namesapce/name e.g. default/nginx ᶆXPSLRVFVF"EE
72. 3FqFDUPS %FMUB'*'0
    RVFVF 3FTPVSDF
    &WFOU
    )BOEMFS JONFNPSZDBDIF
    TUPSF *OEFYFS 8PSL2VFVF 1SPDFTT
    *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶅ
    "EE'VOD
    6QEBUF'VOD
    %FMFUF'VOD
73. 3FqFDUPS %FMUB'*'0
    RVFVF 3FTPVSDF
    &WFOU
    )BOEMFS JONFNPSZDBDIF
    TUPSF *OEFYFS 8PSL2VFVF 1SPDFTT
    *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Enqueue~ 8BUDI ᶆXPSLRVFVF"EE
74. 3FqFDUPS %FMUB'*'0
    RVFVF 3FTPVSDF
    &WFOU
    )BOEMFS JONFNPSZDBDIF
    TUPSF *OEFYFS 8PSL2VFVF 1SPDFTT
    *UFN $POUSPMMFSϩδοΫ *OGPSNFS &WFOU)BOEMFS ৄࡉWorkQueue ~Dequeue~ 8BUDI ᶇXPSLRVFVF(FU
75. Appendix) Informer enqueue ιʔε ᶃ https://github.com/kubernetes/client-go/blob/master/tools/cache/reflector.go#L267 ᶄ https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L153 ᶅ
 https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L198 https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L153 ※ ReplicaSet Controllerͷ৔߹ ᶆ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L417 ※ ReplicaSet Controllerͷ৔߹ ᶇ https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L438 ※ ReplicaSet Controllerͷ৔߹
76. Appendix) InformerͱResync Period Informer͸ىಈ࣌ʹɺResync Periodͱ͍͏Ҿ਺Λࢦఆ͢Δ Informer͸api-server͔ΒObjectͷมߋΛ؂ࢹ͍ͯ͠Δ Resync PeriodΛա͗ΔͱɺԿͷEvent͕ൃੜ͍ͯ͠ͳ͍ͯ͘΋ UpdateFunc͕ݺͼग़͞Εͯɺ݁ՌReconcile͕࣮ߦ͞ΕΔ ※͜ͷ࣌ɺResync͸in-memory-cacheΛࢀর͢Δɻ Resync(cacheͷࢀর)ͱRelist(api-server΁ͷ໰͍߹Θͤ)͸ผ෺ informer.Start SFTZOD
    QFSJPE List Watch Event Reflector Added Updated Updated Added AddFunc UpdateFunc AddFunc UpdateFunc Handler
77. ControllerͷCycle Main Logic
78. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS 8PSL2VFVF ControllerͷCycle Informer + WorkQueue + Controller ͷશମͷCycleΛ֬ೝ $POUSPMMFS ϝΠϯϩδοΫ
79. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle XBUDI 8PSL2VFVF ʲԾఆʳ Pod͕͢Ͱʹ2ͭ͋Δ ͔͜͜ΒUpdate Event͕ൃੜ͢Δ
80. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle 6QEBUF XBUDI 8PSL2VFVF
81. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle XBUDI BEE'VOD
    VQEBUF'VOD
    EFMFUF'VOD ᶄ 8PSL2VFVF
82. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle XBUDI ᶅXPSL2VFVF"EE 8PSL2VFVF
83. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle XBUDI ᶅXPSL2VFVF"EE 8PSL2VFVF
84. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle XBUDI ᶆXPSL2VFVF(FU 8PSL2VFVF
85. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle XBUDI ᶆXPSL2VFVF(FU 8PSL2VFVF
86. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle XBUDI ᶇ4ZOD)BOEMFS
    
    3FDPODJMF 8PSL2VFVF
87. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle XBUDI ᶈXPSL2VFVF'PSHFU ᶉXPSL2VFVF%POF Reconcile͕ਖ਼ৗऴྃͨ͠Β
    
    ΞΠςϜΛ8PSL2VFVF͔Βআ֎ 8PSL2VFVF
88. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle XBUDI ᶇ`
    XPSL2VFVF"EE3BUF-JNJUFE Reconcile͕Τϥʔ 8PSL2VFVFʹ3FRVFVFͯ͠ɺ3FDPODJMFΛ࠶ࢼߦ 8PSL2VFVF
89. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle XBUDI ᶇ`
    XPSL2VFVF"EE3BUF-JNJUFE Reconcile͕Τϥʔ 8PSL2VFVFʹ3FRVFVFͯ͠ɺ3FDPODJMFΛ࠶ࢼߦ 8PSL2VFVF
90. *OGPSNFS 1SPDFTT
    *UFN 3FTPVSDF
    &WFOU
    )BOEMFS ControllerͷCycle Event͕ൃੜ͢ΔͨͼʹWorkQueueʹΞΠςϜ͕ೖΓଓ͚Δ WorkQueueʹஷ·ͬͨΞΠςϜΛReconcileͰॲཧ͠ଓ͚Δ ͜ͷϧʔϓ͕ɺController͕ࢭ·Δ·ͰԆʑͱଓ͘ 8PSL2VFVF
91. Controllerͷجຊઓུ Read͸In-memory-cache͔ΒɺWrite͸api-server΁ ※cacheΛ͍͡ΔͱҰ؏ੑΛอͭͷ͕೉͍͠ͷͰɺ Write͢Δͱ͖͸ɺҰճDeepCopy͔ͯ͠Βߋ৽͢Δ e.g. kubernetes/pkg/controller/replicaset/replica_set.go rs = rs.DeepCopy() newStatus := calculateStatus(rs, filteredPods, manageReplicasErr) // Always updates status as pods come up or die. updatedRS, err := updateReplicaSetStatus(rsc.kubeClient.AppsV1(). ὎ ReplicaSets(rs.Namespace), rs, newStatus) https://github.com/kubernetes/kubernetes/blob/release-1.15/pkg/controller/replicaset/replica_set.go#L611
92. XPSLFS XPSLFS QSPDFTT/FYU8PSL*UFN TZOD)BOEMFS ControllerͷMain Logic worker: processNextWorkItemΛແݶϧʔϓ processNextWorkItem: WorkQueueΛૢ࡞(Get, Add)ͯ͠ ReconcileΛݺͼग़͢ syncHandler: Reconcileʹ͋ͨΔϝΠϯϩδοΫ Add TZOD)BOEMFS Update Delete Event Reconcile Eventͷछྨʹ͔͔ΘΒͣɺ Reconcile͢Δ
93. Appendix) ReplicaSet Controller worker: processNextWorkItemΛແݶϧʔϓ processNextWorkItem: WorkQueueΛૢ࡞(Get, Add)ͯ͠ ReconcileΛݺͼग़͢ syncReplicaSet: Reconcileʹ͋ͨΔϝΠϯϩδοΫ XPSLFS XPSLFS QSPDFTT/FYU8PSL*UFN TZOD3FQMJDB4FU https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L432 https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L437 https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L562 ReplicaSet Controllerͷιʔε v1.16 ※Deployment΋ྨࣅͷߏ଄Λ͍ͯ͠Δ
94. Appendix) in-memory-cacheͱetcdͷಉظ in-memory-cache͸InformerΛ࢖ͬͯඇಉظʹetcdͷ৘ใΛ Ωϟογϡ͍ͯ͠Δ ʮΩϟογϡͱetcdͰ৘ใ͕ζϨͳ͍͔ʁʯͱࢥ͏͔΋͠Εͳ ͍͕ɺKubernetesͰ͸resourceVersionͱ͍͏࢓૊Έ͕͋Δɻ resourceVersion͕ҟͳΔͱΤϥʔʹͳͬͯɺRecocileΛ࠶ࢼߦ ͢ΔͷͰɺζϨ͍ͯͯ΋໰୊͸ͳ͍ Rv2 Rv1 Update Rv2 version too old Requeue Reconcile (retry)
95. ༻ޠ੔ཧ(ৼΓฦΓ) Informer: ObjectͷEventΛ؂ࢹ͠ɺin-memory-cacheʹσʔλΛ֨ೲ͢Δ EventHandlerΛ௨ͯ͡ɺWorkQueueʹΞΠςϜΛ௥Ճ͢Δ Lister: ObjectΛऔಘ͍ͨ࣌͠ in-memory-cache͔ΒσʔλΛऔಘ͢Δ WorkQueue: Controller͕ॲཧ͢ΔΞΠςϜΛొ࿥͓ͯ͘͠Ωϡʔ ͜ͷΩϡʔʹஷ·͍ͬͯΔΞΠςϜ͕Reconcileͷର৅ ReconcileͰΤϥʔ͕ൃੜͨ͠৔߹͸ɺRequeue͞Εɺ ControllerଆͰReconcileΛ࠶ࢼߦ͢Δ
96. Controllerͷ·ͱΊ
97. Controllerͷ·ͱΊ ɾController͸ɺControl Loop(Reconciliation Loop)ʹΑͬͯ એݴతͳAPIΛ࣮ݱ ɾ෼ࢄίϯϙʔωϯτ͸ͦΕͧΕಠཱ͓ͯ͠Γɺ Event͕ίϯϙʔωϯτؒΛ݁ͼ͚ͭΔ ɾclient-go, apimachinery, code-generator౳͕ControllerΛࢧ͑Δ ɾInformer͸ೋͭͷ໨తΛ࣋ͭ ᶃ in-memory-cacheʹObjectͷσʔλΛ֨ೲ͢Δ ᶄ EventHandlerܦ༝ͰWorkQueueʹΞΠςϜΛ֨ೲ͢Δ ɾWorkQueueʹஷΊΒΕͨΞΠςϜ͸ReconcileͰॲཧ͞ΕΔ ※ ͜ΕΒΛ஌Βͳͯ͘΋ɺController͸ར༻Ͱ͖·͢
98. Deep Dive΁ͷಓ ʙ ΋ͬͱਂ͘஌Γ͍ͨਓ޲͚ʹ ʙ ɾSample ControllerΛಈ͔͢ ɾKubernetesຊମͷ࣮૷ΛݟͯΈΔ - Deployment Controller - ReplicaSet Controller ɾCustom ControllerΛ࡞ͬͯΈΔ(+ CRD) ϦϯΫ: https://github.com/kubernetes/sample-controller mkdir -p $GOPATH/src/k8s.io && cd $GOPATH/src/k8s.io && git clone https://github.com/kubernetes/sample-controller.git export GO111MODULE=on go build -o sample-controller . ./sample-controller -kubeconfig $HOME/.kube/config https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/deployment/deployment_controller.go https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go
99. Programming Kubernetes https://programming-kubernetes.info/ ঺հϒϩάࢀߟ: https://go-vargo.hatenablog.com/entry/2019/08/05/201546 O’Reilly Media, Inc.ΑΓग़൛ ʮProgramming Kubernetesʯ Stefan Schimanski, Michael Hausenblasஶ 1~2ষ: Kubernetesͷجຊࢥ૝, API Object 3ষ: client-go 4ষ: CRD 5ষ: code-generator 6~7ষ: Custom Controller 8~9ষ: Custom API Server, CRDԠ༻ػೳ
100. Thank you
101. Reference
102. Reference ɾWeb Article - https://kubernetes.io/docs/concepts/architecture/nodes/ - https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/ - https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md - A deep dive into Kubernetes controllers (https://engineering.bitnami.com/articles/a-deep-dive-into-kubernetes-controllers.html) - Core Kubernetes: Jazz Improv over Orchestration (https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca) - Events, the DNA of Kubernetes(https://www.mgasch.com/post/k8sevents/) ɾPresentation - Kubernete Meetup Tokyo #18 - Kubebuilder/controller-runtime ೖ໳ (https://www.slideshare.net/pfi/kubernete-meetup-tokyo-18-kubebuildercontrollerruntime) - KubernetesͷιʔείʔυϦʔσΟϯάೖ໳ (https://speakerdeck.com/smatsuzaki/kubernetesfalsesosukodorideinguru-men) ɾBook - Programming Kubernetes (https://programming-kubernetes.info/)
103. Reference ɾRepository - Kubernetes(https://github.com/kubernetes/kubernetes) - Sample Controller(https://github.com/kubernetes/sample-controller) - client-go(https://github.com/kubernetes/client-go) - apimachinery(https://github.com/kubernetes/apimachinery) - codegenerator(https://github.com/kubernetes/code-generator) - what-happens-when-k8s(https://github.com/jamiehannaford/what-happens-when-k8s)