ゼロから始めるKubernetes Controller / Under the Kubernetes Controller

θϩ͔Β࢝ΊΔ
Kubernetes Controller
2019/09/27
Kubernetes Meetup #23 Operator Deep Dive

View Slide

ࠓ೔ͷ໨ඪɾ࿩͢͜ͱ
ɾKubernetesͷResourceΛ͋Δఔ౓஌͍ͬͯΔਓ͕ɺɹ
ɹControllerͷ஌ࣝθϩͷঢ়ଶ͔ΒͰ΋Deep DiveʹΑͬͯɺ
ɹControllerͷ࢓૊Έɾ಺෦࣮૷ʹ͍ͭͯཧղͰ͖Δ͜ͱ
ɾKubernetes Controllerͷ֓ཁɾࢥ૝
ɾKubernetes Controllerͷ࢓૊Έ, ಺෦࣮૷
ɾControllerΛࢧ͑Δίϯϙʔωϯτ
= ControllerͷϋΠϨϕϧʙϩʔϨϕϧ·Ͱ
࿩͢͜ͱ
໨ඪ

View Slide

࿩͞ͳ͍͜ͱ
ɾKubernetes Custom Controller + CRDͷৄࡉ
※ ͨͩ͠ɺ७ਖ਼Controllerͷ࢓૊ΈΛ஌Δ͜ͱ͸ɺ
Custom ControllerΛ࣮૷͢Δ্Ͱ΋໾ཱͪ·͢
ɾKubernetes Custom ControllerΛ࣮૷͢ΔͨΊͷϑϨʔϜϫʔΫ
KubebuilderɺOperator SDKͷৄࡉʹ͍ͭͯ͸࿩͠·ͤΜ
಺෦SDKͰ͋Δcontroller-runtime΍controller-toolsʹ͍ͭͯ΋
ຊൃදͰ͸৮Ε·ͤΜ
࿩͞ͳ͍͜ͱ
controller-runtimeࢀߟ:
https://www.slideshare.net/pﬁ/kubernete-meetup-tokyo-18-kubebuildercontrollerruntime

View Slide

ΞδΣϯμ
ɾ Kubernetes Controllerͱ͸ʁ
ɾ Control Loop(Reconciliation Loop)
ɾControllerΛࢧ͑Δɺ஌͓ͬͯ͘΂͖ίϯϙʔωϯτ
ɾClient-Go
- Informer
- WorkQueue
ɾControllerͷCycle, Main Logic
ɾControllerͷ·ͱΊ
- Reference

View Slide

Kubernetes Controllerͱ͸ʁ
ʙ ϋΠϨϕϧΞʔΩςΫνϟ ʙ

View Slide

.BTUFS
8PSLFS
8PSLFS
8PSLFS
KubernetesΞʔΩςΫνϟ͓͞Β͍
Master Node Worker Node

View Slide

KubernetesϋΠϨϕϧΞʔΩςΫνϟ
.BTUFS 8PSLFS
FUDE
DMPVEDPOUSPMMFS
NBOBHFS
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
DPOUSPMMFS
NBOBHFS
BQJTFSWFS
TDIFEVMFS
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
DMPVE
LVCF
QSPYZ
LVCF
QSPYZ

View Slide

KubernetesͷΞʔΩςΫνϟ
ɾKubernetes͸෼ࢄΞʔΩςΫνϟɾ෼ࢄίϯϙʔωϯτ
Master:
ೝূɾೝՄ΍Resource ͷ؅ཧɺ
ίϯςφͷεέδϡʔϦϯάͳͲͷશମ؅ཧ
Worker:
ίϯςφͷ࣮ߦ෦෼

View Slide

KubernetesϋΠϨϕϧΞʔΩςΫνϟ
.BTUFS 8PSLFS
FUDE
DMPVEDPOUSPMMFS
NBOBHFS
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
DPOUSPMMFS
NBOBHFS
BQJTFSWFS
TDIFEVMFS
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
DMPVE
LVCF
QSPYZ
LVCF
QSPYZ

View Slide

api-server / controller-manager
api-server:
Deployment΍ServiceͳͲͷObjectͷ࡞੒ɾߋ৽ɾ࡟আ
(CRUD)ͷϦΫΤετΛड͚औΓɺૢ࡞Λߦ͏
Objectͷ৘ใΛσʔλετΞͰ͋Δetcdʹ֨ೲ͢Δ
※etcdʹΞΫηε͢Δίϯϙʔωϯτ͸api-serverͷΈ
controller-manager:
Controller͸ɺDeployment΍ServiceͳͲͷObjectͷ
؅ཧΛߦ͏ɻ
controller-manager͸ɺController܈ͷ͔ͨ·Γ

View Slide

controller-manager
controller-manager: Ұͭͷϓϩηεʹෳ਺ͷController
%FQMPZNFOU 3FQMJDB4FU %BFNPO4FU
4FSWJDF +PC $SPO+PC
&OEQPJOU 4UBUFGVM4FU ʜ

View Slide

3FQMJDB4FU
$POUSPMMFS
LJOE3FQMJDB4FU
NFUBEBUB
OBNFYYYYYY
TQFD
ʜ
3FTPVSDF

3FTPVSDF

3FTPVSDF
ʜ
3FTPVSDF
/
؅ཧ
$POUSPM-PPQ
.BOJGFTU
ControllerͱResource
Controller͕Resourceͷ؅ཧΛߦ͏
3FQMJDB4FU$POUSPMMFSͷྫ

View Slide

3FQMJDB4FU
$POUSPMMFS
ControllerͱResource
Controller͸ɺͻͱͭͷResourceΛ؅ཧ͢Δ
3FQMJDB4FU
1PE
%FQMPZNFOU
$POUSPMMFS
%FQMPZNFOU
NBOBHF
DSFBUF
NBOBHF
DSFBUF
ࢀߟ: OwnerReference
https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/
্ҐResource͕ԼҐResourceΛ؅ཧ͢Δ࢓૊Έͱͯ͠ʮOwnerReferenceʯ͕͋Δ
্ҐResourceΛ࡟আ͢ΔͱɺԼҐResource͸ΨϕʔδίϨΫγϣϯͰ࡟আ͞ΕΔ

View Slide

Control Loop
(Reconciliation Loop)

View Slide

"DUVBM4UBUF
$POUSPM-PPQ
%FTJSFE4UBUF
3FBESFBMSFTPVSDFT
$IBOHFJOUFSOBMFYUFSOBMSFTPVSDFT
PCTFSWF
BOBMZ[F
"DU
Controllerͷجຊݪཧ
جຊݪཧ: Control Loop(Reconciliation Loop)

View Slide

Control Loop(Reconciliation Loop)
Controller Loop͸ɺControllerͷجຊݪཧ
※Reconciliation Loopͱ΋ݺ͹Ε͍ͯΔ
Controller LoopͷྲྀΕ:
1. Resourceͷݱࡏͷঢ়ଶΛಡΈࠐΉ
2. ResourceΛ๬Μͩঢ়ଶʹมߋ͢Δ
3. Resourceͷঢ়ଶ(Status)Λߋ৽͢Δ
Controller LoopʹΑͬͯɺ๬Μͩঢ়ଶΛએݴ͢Δ͜ͱͰ
؀ڥ͕ߏங͞ΕΔɻ
= Immutable Infrastructure͕࣮ݱͰ͖Δ
ϧʔϓ

View Slide

3FQMJDBT
ReplicaSetͷControl Loopྫ
3FQMJDBT
3FQMJDB4FU
$POUSPMMFS
3FQMJDB4FU
$POUSPMMFS
0CTFSWF
๬Μͩঢ়ଶ ݱࡏͷঢ়ଶ
"DU

View Slide

ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
LJOE3FQMJDB4FU
NFUBEBUB
OBNFYYYYYY
TQFD
ʜ
.BOJGFTU
6TFS
.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
,VCFDUMBQQMZGNBOJGFTUZBNM
†3FTPVSDFΛ"QQMZ͢Δ
3FQMJDB4FU
$POUSPMMFS
ᶃ3FQMJDB4FU͕ੜ੒͞ΕΔ

View Slide

.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
3FQMJDB4FU
$POUSPMMFS
$POUSPM-PPQ
ᶄ3FQMJDB4FU͕ੜ੒͞Εͨ͜ͱΛɺ3FQMJDB4FU$POUSPMMFS͕ݕ஌
ᶄ`ϝΠϯϩδοΫʹΑͬͯɺ
TQFDOPEF/BNF͕ۭͷ1PEΛ࡞੒͢Δ
DSFBUF ˞8PSLFSʹ͸·ͩσϦόϦ͞Εͳ͍

View Slide

.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
3FQMJDB4FU
$POUSPMMFS
ᶅ4DIFEVMFS͕1PEͷ࡞੒Λݕ஌
queue
ᶅ`1PEͷTQFDOPEF/BNF͕ۭͳͷͰɺ
TDIEVMFS͕RVFVFʹ௥Ճ͢Δ

View Slide

.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
3FQMJDB4FU
$POUSPMMFS
ᶆLVCFMFU͕1PEͷ࡞੒Λݕ஌
queue
ᶆ`1PEͷTQFDOPEF/BNF͕ۭͳͷͰɺ
LVCFMFU͸ॲཧΛεΩοϓ
skip

View Slide

.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
3FQMJDB4FU
$POUSPMMFS
ᶇ4DIFEVMFS͕1PEΛRVFVF͔ΒऔΓग़͢
queue
ᶇ`ΞαΠϯՄೳͳ/PEFʹ1PEΛ
εέδϡʔϦϯά͢Δ
schedule
ᶇ`1PEͷTQFDOPEF/BNFΛߋ৽
6QEBUF
TQFDOPEF/BNF

View Slide

.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
3FQMJDB4FU
$POUSPMMFS
queue
ᶈLVCFMFU͕1PEͷߋ৽Λݕ஌
ᶈ`LVCFMFU͕ίϯςφΛىಈ

View Slide

.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
3FQMJDB4FU
$POUSPMMFS
queue
ᶉLVCFMFU͕BQJTFSWFSʹϦΫΤετ͠ɺ
1PEͷ4UBUVTΛߋ৽
TUBUVTDPOEJUJPOT

View Slide

.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
3FQMJDB4FU
$POUSPMMFS
queue
͜͜Ͱɺίϯςφ͕ͳΜΒ͔ͷཧ༝Ͱࢮ๢ͨ͠ͱ͠·͢

View Slide

.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
3FQMJDB4FU
$POUSPMMFS
queue
1PEͷ5FSNJOBUJOH
TUBUVTDPOEJUJPOT
ᶊLVCFMFU͕BQJTFSWFSʹϦΫΤετ͠ɺ
1PEͷ4UBUVTΛߋ৽

View Slide

.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
3FQMJDB4FU
$POUSPMMFS
queue
1PEͷ5FSNJOBUJOH
ᶋ3FQMJDB4FU$POUSPMMFS͕1PEͷߋ৽Λ
ݕ஌
EFMFUF
ᶋ`3FQMJDB4FU$POUSPMMFS͕1PEΛ࡟আ

View Slide

.BTUFS
8PSLFS
FUDE
/PEF
$POUBJOFS
3VOUJNF
LVCFMFU
LVCF
QSPYZ
BQJTFSWFS
TDIFEVMFS
3FQMJDB4FU
$POUSPMMFS
queue
3FDPODJMF
ᶌ3FQMJDB4FU$POUSPMMFS͕3FDPODJMF͠ɺ
๬Μͩঢ়ଶ 3FQMJDBT
ʹ෮چ͠ɺ
1PE͕৽͘͠࡞੒͞ΕΔ
EFMFUF
Ҏ߱ɺϧʔϓʜ
ᶄʙᶌͷॲཧΛ܁Γฦ͢

View Slide

Appendix) ιʔε
ᶄ
https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L487
ᶅ
https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/eventhandlers.go#L436
ᶆ
https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/kubelet/conﬁg/apiserver.go#L33
ᶇ
https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/scheduler.go#L535
ᶈ
https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L803
ᶉ
https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kubelet.go#L1527
ᶊ
https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kubelet.go#L2006
ᶋ
https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/controller/replicaset/replica_set.go#L535
ᶌ

View Slide

Controllerͱίϯϙʔωϯτͨͪ
֤ίϯϙʔωϯτͷಇ͖
ɾapi-server: ResourceͷCRUD
ɾscheduler: ResourceͷεέδϡʔϦϯά
ɾkubelet: ίϯςφͷىಈ
ɾcontroller: ResourceͷReconcile(ௐ੔)
֤ίϯϙʔωϯτ͸ͦΕͧΕͷ੹຿ʹूத
= શମΛ౷཰ɾ౷੍͢Δࢦشऀ͸͍ͳ͍

View Slide

Controllerͱௐ࿨
Kubernetes͸ͦΕͧΕͷίϯϙʔωϯτ͕ڠௐͯ͠ಈ͍͍ͯΔ
Θ͚Ͱ͸ͳ͍
໌ࣔతʹ໋ྩΛ͍ͯ͠ͳ͍ͷʹɺશମͱͯ͠੔߹ੑ͕औ
Ε͍ͯΔ
͜͜Ͱ…
ίϯϙʔωϯτͨͪΛɺͦΕͧΕҰͭͷControllerͱߟ͑Δ
ͦΕͧΕͷίϯτϩʔϥ͸ɺ
ͦΕͧΕͷControl LoopΛ࣮ߦ͢Δ͜ͱ͚ͩʹूத͢Δ
= ෆࢥٞͱɺશମͱͯ͠ͷ੔߹ੑ͕औΕ͍ͯΔ

View Slide

Kubernetes is jazz improv
OrchestrationͰ͸ͳ͘ɺ
Player(Controller)͕֤ʑͷԋ૗
(Control Loop)ʹूத͢Δ͜ͱͰ
ଈڵͷԻָͱͯ͠ɺશମ͕੒Γཱ͍ͬͯΔ
Kubernetes is more jazz improv
than orchestration.
Joe Beda
https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca
Co-founder
Core Kubernetes: Jazz Improv over Orchestration

View Slide

ͳͥControl LoopΛߦ͏ͷ͔ʁ
Control LoopΛߦ͏ཧ༝Λߟ͑Δ࣌ʹɺॏཁʹͳΔ΋ͷ͕
ʮEventʯ ※ e.g. Added, Modiﬁed, Deleted, Error…
෼ࢄίϯϙʔωϯτͰ੒ΓཱͭKubernetesʹͱͬͯɺ
Event͸ɺͦΕͧΕͷίϯϙʔωϯτΛͭͳ͙ॏཁͳ΋ͷ
ࢀߟ: Events, the DNA of Kubernetes
EventΛͲͷΑ͏ʹѻ͏͔ɺ伴ͱͳΔߟ͑ํ͕
ɾEdge-driven Triggers
ɾLevel-driven Triggers
https://www.mgasch.com/post/k8sevents/

View Slide

Appendix) Edge vs. Level
https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d
ࢀߟ: Level Triggering and Reconciliation in Kubernetes
Edge-driven Triggers
Level-driven Triggers
EventͷൃੜʹԠͯ͡Trigger
ಛఆͷঢ়ଶʹͳΔͱTrigger
up
down
low
high
low

View Slide

ͳͥControl LoopΛߦ͏ͷ͔ʁ
ͳͥController͕Control LoopΛߦ͏ͷ͔Λߟ͑ΔͨΊʹɺ
Kubernetes͕એݴతͳڍಈ(Reconcile)Ͱ͸ͳ͘ɺ
खଓ͖తͳڍಈ(Procedure)Λ͢ΔͱԾఆ
LJOEYYYYYYYYY
NFUBEBUB
OBNFYYYYYY
TQFD
ʜ
࣮ࡍͷKubernetes Ծఆ
Reconcile Procedure

View Slide

΋͠΋Controller͕खଓ͖ܕͩͬͨΒ
ݱࡏͷ3FQMJDBT਺
ˢ

࡞
੒

ˢ

࡟
ݮ
खଓ͖తʹಈͨ͘ΊɺEvent͕ൃੜ͢Δͨͼʹ
Edge-driven Triggerͱͯ͠ಈ͘
Ұݟ໰୊ͳ͍Α͏ʹݟ͑Δ͕ɺऑ఺΋͋Δ
&WFOU

View Slide

΋͠΋Reconcile͕ͳ͔ͬͨΒ
ݱࡏͷ3FQMJDBT਺
ˢ

࡞
੒
ˢ

࡟
ݮ
Ұ࣌తͳωοτϫʔΫͷෆௐ΍όάͳͲͷো֐͕͋ͬͨ࣌ʹ
Eventͷ৘ใ͕ࣦΘΕͯ͠·͏
ো
֐
&WFOU

View Slide

Resync IntervalͱReconcile
ݱࡏͷ3FQMJDBT਺
ˢ

࡞
੒

ˢ

࡟
ݮ
Resync Interval͝ͱʹReconcile(ௐ੔)Λ͢Δ͜ͱͰɺ
Event͕ࣦΘΕͯ΋ཧ૝తͳঢ়ଶʹ͚ۙͮΔ
ো
֐
SFTZODJOUFSWBM
&WFOU
Kubernetes = Edge-driven Trigger + Level-driven Trigger

View Slide

ControllerΛࢧ͑Δɺ
஌͓ͬͯ͘΂͖ίϯϙʔωϯτ
ʙ ϛυϧϨϕϧΞʔΩςΫνϟ ʙ

View Slide

༻ޠͷ੔ཧ
Kind:
API Objectͷछྨ(Deployment, ServiceͳͲ)
 
Resource:
Kindͱಉ͡ҙຯ߹͍Ͱ༻͍ΒΕΔɻ
HTTP Endpointͱͯ͠΋ར༻͞ΕΔɻ
খจࣈ͔ͭෳ਺ܗͰදݱ͞ΕΔ(pods, servicesͳͲ)
Object:
࡞੒͞ΕͨAPI Object ͷ࣮ମ
Metadata΍Spec, StatusͳͲͷσʔλΛอ࣋͢Δ

View Slide

ControllerΛࢧ͑ΔϥΠϒϥϦ
client-go
Informer
Lister
WorkQueue
api-machinery
runtime.Object
Scheme
code-generator
ϥΠϒϥϦ ίϯϙʔωϯτ
লུ

View Slide

Appendix) Custom Controller SDK
Kubebuilder
Informer
ϑϨʔϜϫʔΫ
ίϯϙʔωϯτ
Lister
Operator SDK
ϥΠϒϥϦ
(ϋΠϨϕϧ)
controller-runtime controller-tools
ϥΠϒϥϦ
(ϩʔϨϕϧ)
client-go api-machinery etc…
Scheme
runtime.Object
WorkQueue
etc…

View Slide

ControllerΛࢧ͑ΔϥΠϒϥϦ
client-go:
KubernetesͷެࣜΫϥΠΞϯτϥΠϒϥϦ
Kubernetesຊମͷ։ൃʹ΋࢖ΘΕ͍ͯΔ
Controller࡞੒ʹ͸͔ܽͤͳ͍ϥΠϒϥϦ
api-machinery:
Kubernetes API Object & Kubernetes API like Object
ʹඞཁͳػೳΛඋ͑ͨϥΠϒϥϦ
e.g. conversion, decode, encode, etc…
Controller͸API ObjectΛѻ͏ͷͰɺඞཁʹͳΔ
code-generator:
Informer, Lister, clientset, DeepCopyͳͲͷίʔυΛੜ੒

View Slide

ControllerΛࢧ͑Δίϯϙʔωϯτ
ͦΕͧΕͷৄࡉ͸ɺControllerͷ࢓૊ΈΛࢀরͭͭ͠ޙड़͠·͢ɻ
Informer:
ObjectͷEventΛ؂ࢹ͠ɺin-memory-cacheʹσʔλΛ֨ೲ͢Δ
Lister:
in-memory-cache͔ΒσʔλΛऔಘ͢Δ
WorkQueue:
Controller͕ॲཧ͢ΔΞΠςϜΛొ࿥͓ͯ͘͠Ωϡʔ
runtime.Object:
શͯͷAPI Objectڞ௨ͷInterface
Scheme:
Kubernetes APIͱGo TypeΛඥ෇͚Δػߏ
ࠓճ͸આ໌লུ

View Slide

client-go
Informer
ʙ ϩʔϨϕϧΞʔΩςΫνϟ ʙ

View Slide

client-goͱInformer
client-go
Informer
ϥΠϒϥϦ
ίϯϙʔωϯτ
Reﬂector DeltaFIFO Indexer
Store Lister

View Slide

Informer
Informer͸Objectͷมߋ(Event)Λ؂ࢹ͢Δίϯϙʔωϯτ
ObjectͷมߋΛ؂ࢹ͢ΔͨΊʹɺapi-serverʹঢ়ଶΛຖճ
໰͍߹ΘͤΔͱɺapi-serverʹෛՙ͕͔͔Δ
὎ in-memory-cacheʹObjectͷσʔλΛ֨ೲ͠ɺ
ͦͷcacheΛࢀর͢Δ͜ͱͰapi-server΁ͷෛՙΛܰݮͤ͞Δ
ղܾࡦΛ࠾༻
$POUSPMMFS $POUSPMMFS
*OGPSNFS
in-memory-cache
watch
watch

View Slide

Appendix) InformerΛ࢖࣮ͬͨ૷
func main() {
...
clientset, err := kubernetes.NewForConfig(config)
// Create InformerFactory
informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30)
// Create pod informer by informerFactory
podInformer := informerFactory.Core().V1().Pods()
// Add EventHandler to informer
podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: func(new interface{}) { log.Println("Added") },
UpdateFunc: func(old, new interface{}) { log.Println("Updated") },
DeleteFunc: func(old interface{}) { log.Println("Deleted") },
})
// Start Go routines
informerFactory.Start(wait.NeverStop)
// Wait until finish caching with List API
informerFactory.WaitForCacheSync(wait.NeverStop)
// Create Pod Lister
podLister := podInformer.Lister()
// Get List of pods
_, err = podLister.List(labels.Nothing())
…
} https://github.com/govargo/kubecontorller-book-sample-snippet/blob/master/02/podinformer/podinformer.go
ᶃ kubeconfig͔ΒconfigΛੜ੒
ᶄ config͔ΒclientsetΛੜ੒
ᶅ informerFactoryੜ੒
ᶆ ᶅΛ࢖ͬͯinformerੜ੒
ᶇ EventHandlerΛ௥Ճ
Add, Update, Delete࣌ʹൃՐ
ᶈ InformerΛىಈ
ᶉ Lister(Getter)Λੜ੒
ᶊ ListerΛ࢖ͬͯGet΍List

View Slide

Appendix) Shared Informer
InformerΛར༻͢Δͱ͖͸ɺInformerͦͷ΋ͷͰ͸ͳ͘ɺ
Shared Informerͱݺ͹ΕΔ΋ͷΛར༻͢Δ
Shared Informer͸ɺͻͱͭͷόΠφϦ಺Ͱಉ͡ResourceΛڞ༗
Ͱ͖Δ
%FQMPZNFOU 3FQMJDB4FU %BFNPO4FU
4FSWJDF +PC ʜ
4IBSFE*OGPSNFSTIBSFEDBDIFGPSTBNFSFTPVSDF
kube-controller-manager

View Slide

https://github.com/kubernetes/sample-controller/blob/master/docs/controller-client-go.md
Ҿ༻ݩ:
InformerͱWorkQueue֓؍

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

-JTU8BUDI
JONFNPSZDBDIF 4UPSF

*OEFYFS
-JTUFS
(FUPS-JTU
FHDPSFW1PE
(FUPS-JTU
3FRVFTU
*OGPSNFS
)BOEMF%FMUBT
ৄࡉInformer

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

ᶃ-JTU"OE8BUDI
JONFNPSZDBDIF 4UPSF

*OEFYFS
*OGPSNFS
)BOEMF%FMUBT
ৄࡉInformer ~σʔλΛΩϟογϡ~

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

ᶄ1PQ
JONFNPSZDBDIF 4UPSF

*OEFYFS
*OGPSNFS
)BOEMF%FMUBT
8BUDI

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

ᶅ)BOEMF%FMUB
JONFNPSZDBDIF 4UPSF

*OEFYFS
*OGPSNFS
)BOEMF%FMUBT
8BUDI

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

ᶆJOEFYFS"EE
JONFNPSZDBDIF 4UPSF

*OEFYFS
*OGPSNFS
)BOEMF%FMUBT
8BUDI

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

JONFNPSZDBDIF 4UPSF

*OEFYFS
*OGPSNFS
)BOEMF%FMUBT
8BUDI
ᶅ)BOEMF%FMUB
ᶄ1PQ

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

JONFNPSZDBDIF 4UPSF

*OEFYFS
*OGPSNFS
)BOEMF%FMUBT
8BUDI
ᶆJOEFYFS"EE

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

JONFNPSZDBDIF 4UPSF

*OEFYFS
*OGPSNFS
)BOEMF%FMUBT
8BUDI
ৄࡉInformer ~σʔλΛऔΓग़͢~
-JTUFS
(FUPS-JTU
3FRVFTU

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

JONFNPSZDBDIF 4UPSF

*OEFYFS
*OGPSNFS
)BOEMF%FMUBT
8BUDI
JOEFYFS(FU#Z,FZ
PS
JOEFYFS-JTU
ৄࡉInformer ~σʔλΛऔΓग़͢~
-JTUFS
(FUPS-JTU
3FRVFTU

View Slide

Appendix) Informer cache ιʔε
ᶃ
https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/reﬂector.go#L188
ᶄ
https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L153
ᶅ
https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/shared_informer.go#L455
ᶆ
https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/shared_informer.go#L464

View Slide

Informerͱίϯϙʔωϯτ
Informer:
Reﬂector:
api-serverͷEventΛ؂ࢹ͢Δ
DeltaFIFO:
ObjectͷEvent͕ൃੜ͢Δͨͼʹ࢖͏FIFOΩϡʔ
Indexer:
in-memory-cache΁ͷॻ͖ࠐΈ΍ಡΈࠐΈΛߦ͏
Store:
in-memory-cache
Lister:
in-memory-cache͔ΒIndexerܦ༝ͰσʔλΛऔಘ͢Δ

View Slide

client-go
WorkQueue
ʙ ϩʔϨϕϧΞʔΩςΫνϟ ʙ

View Slide

client-goͱWorkQueue
client-go
WorkQueue
ϥΠϒϥϦ
ίϯϙʔωϯτ
RateLimitingQueue
DelayedQueue

View Slide

WorkQueue
WorkQueue͸ɺDeltaFIFOͱ͸ҟͳΔ΋͏ͻͱͭͷΩϡʔ
WorkQueue͸ɺControl LoopͷΞΠςϜΛཷΊ͓ͯͨ͘Ίʹ
࢖ΘΕΔ
ඪ४ControllerͰ͸ɺInformerͷEventHandler͕ൃՐ͢Δͱɺ
WorkQueueʹΞΠςϜΛenqueue͢Δ࣮૷ʹͳ͍ͬͯΔ
Event
$POUSPMMFS
Added
Updated
Deleted WorkQueue

View Slide

Appendix) WorkQueueΛ࢖࣮ͬͨ૷
func main() {
...
clientset, err := kubernetes.NewForConfig(config)
// Create InformerFactory
informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30)
// Create pod informer by informerFactory
podInformer := informerFactory.Core().V1().Pods()
// Create RateLimitQueue
queue := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter())
// shutdown when process ends
defer queue.ShutDown()
// Add EventHandler to informer
podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: func(old interface{}) {
var key string
var err error
if key, err = cache.MetaNamespaceKeyFunc(old); err != nil {
runtime.HandleError(err)
return
}
queue.Add(key)
log.Println("Added: " + key)
},
UpdateFunc: func(old, new interface{}) { … },
DeleteFunc: func(old interface{}) { … },
})
…
}
https://github.com/govargo/kubecontorller-book-sample-snippet/blob/master/02/workqueue/enqueuePod.go
ᶃ kubeconfig͔ΒconfigΛੜ੒
ᶄ config͔ΒclientsetΛੜ੒
ᶅ informerFactoryੜ੒
ᶆ ᶅΛ࢖ͬͯinformerੜ੒
ᶇ EventHandlerΛ௥Ճ
Add, Update, Delete࣌ʹൃՐ
ᶈ InformerΛىಈ
ᶉ Lister(Getter)Λੜ੒
ᶊ ListerΛ࢖ͬͯGet΍List
ᶇ WorkQueueੜ੒

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

-JTU8BUDI
3FTPVSDF
&WFOU
)BOEMFS
JONFNPSZDBDIF
TUPSF

*OEFYFS
8PSL2VFVF
1SPDFTT
*UFN
$POUSPMMFSϩδοΫ
*OGPSNFS
&WFOU)BOEMFS
ৄࡉWorkQueue

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

3FTPVSDF
&WFOU
)BOEMFS
JONFNPSZDBDIF
TUPSF

*OEFYFS
8PSL2VFVF
1SPDFTT
*UFN
$POUSPMMFSϩδοΫ
*OGPSNFS
&WFOU)BOEMFS
ৄࡉWorkQueue ~Enqueue~
&WFOU
-JTU8BUDI

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

3FTPVSDF
&WFOU
)BOEMFS
JONFNPSZDBDIF
TUPSF

*OEFYFS
8PSL2VFVF
1SPDFTT
*UFN
$POUSPMMFSϩδοΫ
*OGPSNFS
&WFOU)BOEMFS
ᶃ8BUDI

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

3FTPVSDF
&WFOU
)BOEMFS
JONFNPSZDBDIF
TUPSF

*OEFYFS
8PSL2VFVF
1SPDFTT
*UFN
$POUSPMMFSϩδοΫ
*OGPSNFS
&WFOU)BOEMFS
8BUDI
ᶄ1PQ

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

3FTPVSDF
&WFOU
)BOEMFS
JONFNPSZDBDIF
TUPSF

*OEFYFS
8PSL2VFVF
1SPDFTT
*UFN
$POUSPMMFSϩδοΫ
*OGPSNFS
&WFOU)BOEMFS
8BUDI
ᶅ
"EE'VOD
6QEBUF'VOD
%FMFUF'VOD

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

3FTPVSDF
&WFOU
)BOEMFS
JONFNPSZDBDIF
TUPSF

*OEFYFS
8PSL2VFVF
1SPDFTT
*UFN
$POUSPMMFSϩδοΫ
*OGPSNFS
&WFOU)BOEMFS
8BUDI
namesapce/name
e.g. default/nginx
ᶆXPSLRVFVF"EE

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

3FTPVSDF
&WFOU
)BOEMFS
JONFNPSZDBDIF
TUPSF

*OEFYFS
8PSL2VFVF
1SPDFTT
*UFN
$POUSPMMFSϩδοΫ
*OGPSNFS
&WFOU)BOEMFS
8BUDI
ᶅ
"EE'VOD
6QEBUF'VOD
%FMFUF'VOD

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

3FTPVSDF
&WFOU
)BOEMFS
JONFNPSZDBDIF
TUPSF

*OEFYFS
8PSL2VFVF
1SPDFTT
*UFN
$POUSPMMFSϩδοΫ
*OGPSNFS
&WFOU)BOEMFS
8BUDI
ᶆXPSLRVFVF"EE

View Slide

3FqFDUPS
%FMUB'*'0
RVFVF

3FTPVSDF
&WFOU
)BOEMFS
JONFNPSZDBDIF
TUPSF

*OEFYFS
8PSL2VFVF
1SPDFTT
*UFN
$POUSPMMFSϩδοΫ
*OGPSNFS
&WFOU)BOEMFS
ৄࡉWorkQueue ~Dequeue~
8BUDI
ᶇXPSLRVFVF(FU

View Slide

Appendix) Informer enqueue ιʔε
ᶃ
https://github.com/kubernetes/client-go/blob/master/tools/cache/reﬂector.go#L267
ᶄ
ᶅ 
※ ReplicaSet Controllerͷ৔߹
ᶆ
ᶇ

View Slide

Appendix) InformerͱResync Period
Informer͸ىಈ࣌ʹɺResync Periodͱ͍͏Ҿ਺Λࢦఆ͢Δ
Informer͸api-server͔ΒObjectͷมߋΛ؂ࢹ͍ͯ͠Δ
Resync PeriodΛա͗ΔͱɺԿͷEvent͕ൃੜ͍ͯ͠ͳ͍ͯ͘΋
UpdateFunc͕ݺͼग़͞Εͯɺ݁ՌReconcile͕࣮ߦ͞ΕΔ
※͜ͷ࣌ɺResync͸in-memory-cacheΛࢀর͢Δɻ
Resync(cacheͷࢀর)ͱRelist(api-server΁ͷ໰͍߹Θͤ)͸ผ෺
informer.Start
SFTZODQFSJPE
List
Watch
Event
Reﬂector
Added Updated Updated
Added
AddFunc UpdateFunc AddFunc UpdateFunc
Handler

View Slide

ControllerͷCycle
Main Logic

View Slide

*OGPSNFS
1SPDFTT
*UFN
3FTPVSDF
&WFOU
)BOEMFS
8PSL2VFVF
ControllerͷCycle
Informer + WorkQueue + Controller
ͷશମͷCycleΛ֬ೝ
$POUSPMMFS
ϝΠϯϩδοΫ

View Slide

*OGPSNFS
1SPDFTT
*UFN
3FTPVSDF
&WFOU
)BOEMFS
ControllerͷCycle
XBUDI
8PSL2VFVF
ʲԾఆʳ
Pod͕͢Ͱʹ2ͭ͋Δ
͔͜͜ΒUpdate Event͕ൃੜ͢Δ

View Slide

*OGPSNFS
1SPDFTT
*UFN
3FTPVSDF
&WFOU
)BOEMFS
ControllerͷCycle
6QEBUF
XBUDI
8PSL2VFVF

View Slide

*OGPSNFS
1SPDFTT
*UFN
3FTPVSDF
&WFOU
)BOEMFS
ControllerͷCycle
XBUDI
BEE'VOD
VQEBUF'VOD
EFMFUF'VOD
ᶄ
8PSL2VFVF

View Slide

*OGPSNFS
1SPDFTT
*UFN
3FTPVSDF
&WFOU
)BOEMFS
ControllerͷCycle
XBUDI
ᶅXPSL2VFVF"EE
8PSL2VFVF

View Slide

*OGPSNFS
1SPDFTT
*UFN
3FTPVSDF
&WFOU
)BOEMFS
ControllerͷCycle
XBUDI
ᶆXPSL2VFVF(FU
8PSL2VFVF

View Slide

*OGPSNFS
1SPDFTT
*UFN
3FTPVSDF
&WFOU
)BOEMFS
ControllerͷCycle
XBUDI
ᶇ4ZOD)BOEMFS
3FDPODJMF
8PSL2VFVF

View Slide

*OGPSNFS
1SPDFTT
*UFN
3FTPVSDF
&WFOU
)BOEMFS
ControllerͷCycle
XBUDI
ᶈXPSL2VFVF'PSHFU
ᶉXPSL2VFVF%POF
Reconcile͕ਖ਼ৗऴྃͨ͠Β
ΞΠςϜΛ8PSL2VFVF͔Βআ֎
8PSL2VFVF

View Slide

*OGPSNFS
1SPDFTT
*UFN
3FTPVSDF
&WFOU
)BOEMFS
ControllerͷCycle
XBUDI
ᶇ`XPSL2VFVF"EE3BUF-JNJUFE
Reconcile͕Τϥʔ
8PSL2VFVFʹ3FRVFVFͯ͠ɺ3FDPODJMFΛ࠶ࢼߦ
8PSL2VFVF

View Slide

*OGPSNFS
1SPDFTT
*UFN
3FTPVSDF
&WFOU
)BOEMFS
ControllerͷCycle
Event͕ൃੜ͢ΔͨͼʹWorkQueueʹΞΠςϜ͕ೖΓଓ͚Δ
WorkQueueʹஷ·ͬͨΞΠςϜΛReconcileͰॲཧ͠ଓ͚Δ
͜ͷϧʔϓ͕ɺController͕ࢭ·Δ·ͰԆʑͱଓ͘
8PSL2VFVF

View Slide

Controllerͷجຊઓུ
Read͸In-memory-cache͔ΒɺWrite͸api-server΁
※cacheΛ͍͡ΔͱҰ؏ੑΛอͭͷ͕೉͍͠ͷͰɺ
Write͢Δͱ͖͸ɺҰճDeepCopy͔ͯ͠Βߋ৽͢Δ
e.g. kubernetes/pkg/controller/replicaset/replica_set.go
rs = rs.DeepCopy()
newStatus := calculateStatus(rs, ﬁlteredPods, manageReplicasErr)
// Always updates status as pods come up or die.
updatedRS, err := updateReplicaSetStatus(rsc.kubeClient.AppsV1(). ὎
ReplicaSets(rs.Namespace), rs, newStatus)

View Slide

XPSLFS
XPSLFS
QSPDFTT/FYU8PSL*UFN
TZOD)BOEMFS
ControllerͷMain Logic
worker:
processNextWorkItemΛແݶϧʔϓ
processNextWorkItem:
WorkQueueΛૢ࡞(Get, Add)ͯ͠
ReconcileΛݺͼग़͢
syncHandler:
Reconcileʹ͋ͨΔϝΠϯϩδοΫ
Add
TZOD)BOEMFS
Update Delete
Event
Reconcile
Eventͷछྨʹ͔͔ΘΒͣɺ
Reconcile͢Δ

View Slide

Appendix) ReplicaSet Controller
worker:
processNextWorkItemΛແݶϧʔϓ
processNextWorkItem:
WorkQueueΛૢ࡞(Get, Add)ͯ͠
ReconcileΛݺͼग़͢
syncReplicaSet:
Reconcileʹ͋ͨΔϝΠϯϩδοΫ
XPSLFS
XPSLFS
QSPDFTT/FYU8PSL*UFN
TZOD3FQMJDB4FU
ReplicaSet Controllerͷιʔε v1.16
※Deployment΋ྨࣅͷߏ଄Λ͍ͯ͠Δ

View Slide

Appendix) in-memory-cacheͱetcdͷಉظ
in-memory-cache͸InformerΛ࢖ͬͯඇಉظʹetcdͷ৘ใΛ
Ωϟογϡ͍ͯ͠Δ
ʮΩϟογϡͱetcdͰ৘ใ͕ζϨͳ͍͔ʁʯͱࢥ͏͔΋͠Εͳ
͍͕ɺKubernetesͰ͸resourceVersionͱ͍͏࢓૊Έ͕͋Δɻ
resourceVersion͕ҟͳΔͱΤϥʔʹͳͬͯɺRecocileΛ࠶ࢼߦ
͢ΔͷͰɺζϨ͍ͯͯ΋໰୊͸ͳ͍
Rv2
Rv1
Update
Rv2
version too old
Requeue
Reconcile
(retry)

View Slide

༻ޠ੔ཧ(ৼΓฦΓ)
Informer:
EventHandlerΛ௨ͯ͡ɺWorkQueueʹΞΠςϜΛ௥Ճ͢Δ
Lister:
ObjectΛऔಘ͍ͨ࣌͠ in-memory-cache͔ΒσʔλΛऔಘ͢Δ
WorkQueue:
Controller͕ॲཧ͢ΔΞΠςϜΛొ࿥͓ͯ͘͠Ωϡʔ
͜ͷΩϡʔʹஷ·͍ͬͯΔΞΠςϜ͕Reconcileͷର৅
ReconcileͰΤϥʔ͕ൃੜͨ͠৔߹͸ɺRequeue͞Εɺ
ControllerଆͰReconcileΛ࠶ࢼߦ͢Δ

View Slide

Controllerͷ·ͱΊ

View Slide

Controllerͷ·ͱΊ
ɾController͸ɺControl Loop(Reconciliation Loop)ʹΑͬͯ
એݴతͳAPIΛ࣮ݱ
ɾ෼ࢄίϯϙʔωϯτ͸ͦΕͧΕಠཱ͓ͯ͠Γɺ
Event͕ίϯϙʔωϯτؒΛ݁ͼ͚ͭΔ
ɾclient-go, apimachinery, code-generator౳͕ControllerΛࢧ͑Δ
ɾInformer͸ೋͭͷ໨తΛ࣋ͭ
ᶃ in-memory-cacheʹObjectͷσʔλΛ֨ೲ͢Δ
ᶄ EventHandlerܦ༝ͰWorkQueueʹΞΠςϜΛ֨ೲ͢Δ
ɾWorkQueueʹஷΊΒΕͨΞΠςϜ͸ReconcileͰॲཧ͞ΕΔ
※ ͜ΕΒΛ஌Βͳͯ͘΋ɺController͸ར༻Ͱ͖·͢

View Slide

Deep Dive΁ͷಓ
ʙ ΋ͬͱਂ͘஌Γ͍ͨਓ޲͚ʹ ʙ
ɾSample ControllerΛಈ͔͢
ɾKubernetesຊମͷ࣮૷ΛݟͯΈΔ
- Deployment Controller
- ReplicaSet Controller
ɾCustom ControllerΛ࡞ͬͯΈΔ(+ CRD)
ϦϯΫ: https://github.com/kubernetes/sample-controller
mkdir -p $GOPATH/src/k8s.io && cd $GOPATH/src/k8s.io && git clone https://github.com/kubernetes/sample-controller.git
export GO111MODULE=on
go build -o sample-controller .
./sample-controller -kubeconﬁg $HOME/.kube/conﬁg
https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/deployment/deployment_controller.go
https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go

View Slide

Programming Kubernetes
https://programming-kubernetes.info/
঺հϒϩάࢀߟ: https://go-vargo.hatenablog.com/entry/2019/08/05/201546
O’Reilly Media, Inc.ΑΓग़൛
ʮProgramming Kubernetesʯ
Stefan Schimanski, Michael Hausenblasஶ
1~2ষ: Kubernetesͷجຊࢥ૝, API Object
3ষ: client-go
4ষ: CRD
5ষ: code-generator
6~7ষ: Custom Controller
8~9ষ: Custom API Server, CRDԠ༻ػೳ

View Slide

Thank you

View Slide

Reference

View Slide

Reference
ɾWeb Article
- https://kubernetes.io/docs/concepts/architecture/nodes/
- https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/
- https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md
- A deep dive into Kubernetes controllers
(https://engineering.bitnami.com/articles/a-deep-dive-into-kubernetes-controllers.html)
- Core Kubernetes: Jazz Improv over Orchestration
(https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca)
- Events, the DNA of Kubernetes(https://www.mgasch.com/post/k8sevents/)
ɾPresentation
- Kubernete Meetup Tokyo #18 - Kubebuilder/controller-runtime ೖ໳
(https://www.slideshare.net/pﬁ/kubernete-meetup-tokyo-18-kubebuildercontrollerruntime)
- KubernetesͷιʔείʔυϦʔσΟϯάೖ໳
(https://speakerdeck.com/smatsuzaki/kubernetesfalsesosukodorideinguru-men)
ɾBook
- Programming Kubernetes (https://programming-kubernetes.info/)

View Slide

Reference
ɾRepository
- Kubernetes(https://github.com/kubernetes/kubernetes)
- Sample Controller(https://github.com/kubernetes/sample-controller)
- client-go(https://github.com/kubernetes/client-go)
- apimachinery(https://github.com/kubernetes/apimachinery)
- codegenerator(https://github.com/kubernetes/code-generator)
- what-happens-when-k8s(https://github.com/jamiehannaford/what-happens-when-k8s)

View Slide

ゼロから始めるKubernetes Controller / Under the Kubernetes Controller

ゼロから始めるKubernetes Controller / Under the Kubernetes Controller

go_vargo

More Decks by go_vargo

Other Decks in Technology

Featured

Transcript