$30 off During Our Annual Pro Sale. View Details »

ゼロから始めるKubernetes Controller / Under the Kubernetes Controller

go_vargo
September 27, 2019

ゼロから始めるKubernetes Controller / Under the Kubernetes Controller

go_vargo

September 27, 2019
Tweet

More Decks by go_vargo

Other Decks in Technology

Transcript

  1. θϩ͔Β࢝ΊΔ
    Kubernetes Controller
    2019/09/27
    Kubernetes Meetup #23 Operator Deep Dive

    View Slide

  2. ࠓ೔ͷ໨ඪɾ࿩͢͜ͱ
    ɾKubernetesͷResourceΛ͋Δఔ౓஌͍ͬͯΔਓ͕ɺɹ
    ɹControllerͷ஌ࣝθϩͷঢ়ଶ͔ΒͰ΋Deep DiveʹΑͬͯɺ
    ɹControllerͷ࢓૊Έɾ಺෦࣮૷ʹ͍ͭͯཧղͰ͖Δ͜ͱ
    ɾKubernetes Controllerͷ֓ཁɾࢥ૝
    ɾKubernetes Controllerͷ࢓૊Έ, ಺෦࣮૷
    ɾControllerΛࢧ͑Δίϯϙʔωϯτ
    = ControllerͷϋΠϨϕϧʙϩʔϨϕϧ·Ͱ
    ࿩͢͜ͱ
    ໨ඪ

    View Slide

  3. ࿩͞ͳ͍͜ͱ
    ɾKubernetes Custom Controller + CRDͷৄࡉ
    ※ ͨͩ͠ɺ७ਖ਼Controllerͷ࢓૊ΈΛ஌Δ͜ͱ͸ɺ
    Custom ControllerΛ࣮૷͢Δ্Ͱ΋໾ཱͪ·͢
    ɾKubernetes Custom ControllerΛ࣮૷͢ΔͨΊͷϑϨʔϜϫʔΫ
    KubebuilderɺOperator SDKͷৄࡉʹ͍ͭͯ͸࿩͠·ͤΜ
    ಺෦SDKͰ͋Δcontroller-runtime΍controller-toolsʹ͍ͭͯ΋
    ຊൃදͰ͸৮Ε·ͤΜ
    ࿩͞ͳ͍͜ͱ
    controller-runtimeࢀߟ:
    https://www.slideshare.net/pfi/kubernete-meetup-tokyo-18-kubebuildercontrollerruntime

    View Slide

  4. ΞδΣϯμ
    ɾ Kubernetes Controllerͱ͸ʁ
    ɾ Control Loop(Reconciliation Loop)
    ɾControllerΛࢧ͑Δɺ஌͓ͬͯ͘΂͖ίϯϙʔωϯτ
    ɾClient-Go
    - Informer
    - WorkQueue
    ɾControllerͷCycle, Main Logic
    ɾControllerͷ·ͱΊ
    - Reference

    View Slide

  5. Kubernetes Controllerͱ͸ʁ
    ʙ ϋΠϨϕϧΞʔΩςΫνϟ ʙ

    View Slide

  6. .BTUFS
    8PSLFS
    8PSLFS
    8PSLFS
    KubernetesΞʔΩςΫνϟ͓͞Β͍
    Master Node Worker Node

    View Slide

  7. KubernetesϋΠϨϕϧΞʔΩςΫνϟ
    .BTUFS 8PSLFS
    FUDE
    DMPVEDPOUSPMMFS
    NBOBHFS
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    DPOUSPMMFS
    NBOBHFS
    BQJTFSWFS
    TDIFEVMFS
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    DMPVE
    LVCF
    QSPYZ
    LVCF
    QSPYZ

    View Slide

  8. KubernetesͷΞʔΩςΫνϟ
    ɾKubernetes͸෼ࢄΞʔΩςΫνϟɾ෼ࢄίϯϙʔωϯτ
    Master:
    ೝূɾೝՄ΍Resource ͷ؅ཧɺ
    ίϯςφͷεέδϡʔϦϯάͳͲͷશମ؅ཧ
    Worker:
    ίϯςφͷ࣮ߦ෦෼

    View Slide

  9. KubernetesϋΠϨϕϧΞʔΩςΫνϟ
    .BTUFS 8PSLFS
    FUDE
    DMPVEDPOUSPMMFS
    NBOBHFS
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    DPOUSPMMFS
    NBOBHFS
    BQJTFSWFS
    TDIFEVMFS
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    DMPVE
    LVCF
    QSPYZ
    LVCF
    QSPYZ

    View Slide

  10. api-server / controller-manager
    api-server:
    Deployment΍ServiceͳͲͷObjectͷ࡞੒ɾߋ৽ɾ࡟আ
    (CRUD)ͷϦΫΤετΛड͚औΓɺૢ࡞Λߦ͏
    Objectͷ৘ใΛσʔλετΞͰ͋Δetcdʹ֨ೲ͢Δ
    ※etcdʹΞΫηε͢Δίϯϙʔωϯτ͸api-serverͷΈ
    controller-manager:
    Controller͸ɺDeployment΍ServiceͳͲͷObjectͷ
    ؅ཧΛߦ͏ɻ
    controller-manager͸ɺController܈ͷ͔ͨ·Γ

    View Slide

  11. controller-manager
    controller-manager: Ұͭͷϓϩηεʹෳ਺ͷController
    %FQMPZNFOU 3FQMJDB4FU %BFNPO4FU
    4FSWJDF +PC $SPO+PC
    &OEQPJOU 4UBUFGVM4FU ʜ

    View Slide

  12. 3FQMJDB4FU
    $POUSPMMFS
    LJOE3FQMJDB4FU
    NFUBEBUB
    OBNFYYYYYY
    TQFD
    ʜ
    3FTPVSDF

    3FTPVSDF

    3FTPVSDF
    ʜ
    3FTPVSDF
    /
    ؅ཧ
    $POUSPM-PPQ
    .BOJGFTU
    ControllerͱResource
    Controller͕Resourceͷ؅ཧΛߦ͏
    3FQMJDB4FU$POUSPMMFSͷྫ

    View Slide

  13. 3FQMJDB4FU
    $POUSPMMFS
    ControllerͱResource
    Controller͸ɺͻͱͭͷResourceΛ؅ཧ͢Δ
    3FQMJDB4FU
    1PE
    %FQMPZNFOU
    $POUSPMMFS
    %FQMPZNFOU
    NBOBHF
    DSFBUF
    NBOBHF
    DSFBUF
    ࢀߟ: OwnerReference
    https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/
    ্ҐResource͕ԼҐResourceΛ؅ཧ͢Δ࢓૊Έͱͯ͠ʮOwnerReferenceʯ͕͋Δ
    ্ҐResourceΛ࡟আ͢ΔͱɺԼҐResource͸ΨϕʔδίϨΫγϣϯͰ࡟আ͞ΕΔ

    View Slide

  14. Control Loop
    (Reconciliation Loop)

    View Slide

  15. "DUVBM4UBUF
    $POUSPM-PPQ
    %FTJSFE4UBUF
    3FBESFBMSFTPVSDFT
    $IBOHFJOUFSOBMFYUFSOBMSFTPVSDFT
    PCTFSWF
    BOBMZ[F
    "DU
    Controllerͷجຊݪཧ
    جຊݪཧ: Control Loop(Reconciliation Loop)

    View Slide

  16. Control Loop(Reconciliation Loop)
    Controller Loop͸ɺControllerͷجຊݪཧ
    ※Reconciliation Loopͱ΋ݺ͹Ε͍ͯΔ
    Controller LoopͷྲྀΕ:
    1. Resourceͷݱࡏͷঢ়ଶΛಡΈࠐΉ
    2. ResourceΛ๬Μͩঢ়ଶʹมߋ͢Δ
    3. Resourceͷঢ়ଶ(Status)Λߋ৽͢Δ
    Controller LoopʹΑͬͯɺ๬Μͩঢ়ଶΛએݴ͢Δ͜ͱͰ
    ؀ڥ͕ߏங͞ΕΔɻ
    = Immutable Infrastructure͕࣮ݱͰ͖Δ
    ϧʔϓ

    View Slide

  17. 3FQMJDBT
    ReplicaSetͷControl Loopྫ
    3FQMJDBT
    3FQMJDB4FU
    $POUSPMMFS
    3FQMJDB4FU
    $POUSPMMFS
    0CTFSWF
    ๬Μͩঢ়ଶ ݱࡏͷঢ়ଶ
    "DU

    View Slide

  18. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    LJOE3FQMJDB4FU
    NFUBEBUB
    OBNFYYYYYY
    TQFD
    ʜ
    .BOJGFTU
    6TFS
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    ,VCFDUMBQQMZGNBOJGFTUZBNM
    †3FTPVSDFΛ"QQMZ͢Δ
    3FQMJDB4FU
    $POUSPMMFS
    ᶃ3FQMJDB4FU͕ੜ੒͞ΕΔ

    View Slide

  19. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    3FQMJDB4FU
    $POUSPMMFS
    $POUSPM-PPQ
    ᶄ3FQMJDB4FU͕ੜ੒͞Εͨ͜ͱΛɺ3FQMJDB4FU$POUSPMMFS͕ݕ஌
    ᶄ`ϝΠϯϩδοΫʹΑͬͯɺ
    TQFDOPEF/BNF͕ۭͷ1PEΛ࡞੒͢Δ
    DSFBUF ˞8PSLFSʹ͸·ͩσϦόϦ͞Εͳ͍

    View Slide

  20. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    3FQMJDB4FU
    $POUSPMMFS
    ᶅ4DIFEVMFS͕1PEͷ࡞੒Λݕ஌
    queue
    ᶅ`1PEͷTQFDOPEF/BNF͕ۭͳͷͰɺ
    TDIEVMFS͕RVFVFʹ௥Ճ͢Δ

    View Slide

  21. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    3FQMJDB4FU
    $POUSPMMFS
    ᶆLVCFMFU͕1PEͷ࡞੒Λݕ஌
    queue
    ᶆ`1PEͷTQFDOPEF/BNF͕ۭͳͷͰɺ
    LVCFMFU͸ॲཧΛεΩοϓ
    skip

    View Slide

  22. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    3FQMJDB4FU
    $POUSPMMFS
    ᶇ4DIFEVMFS͕1PEΛRVFVF͔ΒऔΓग़͢
    queue
    ᶇ`ΞαΠϯՄೳͳ/PEFʹ1PEΛ
    εέδϡʔϦϯά͢Δ
    schedule
    ᶇ`1PEͷTQFDOPEF/BNFΛߋ৽
    6QEBUF
    TQFDOPEF/BNF

    View Slide

  23. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    3FQMJDB4FU
    $POUSPMMFS
    queue
    ᶈLVCFMFU͕1PEͷߋ৽Λݕ஌
    ᶈ`LVCFMFU͕ίϯςφΛىಈ

    View Slide

  24. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    3FQMJDB4FU
    $POUSPMMFS
    queue
    ᶉLVCFMFU͕BQJTFSWFSʹϦΫΤετ͠ɺ
    1PEͷ4UBUVTΛߋ৽
    TUBUVTDPOEJUJPOT

    View Slide

  25. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    3FQMJDB4FU
    $POUSPMMFS
    queue
    ͜͜Ͱɺίϯςφ͕ͳΜΒ͔ͷཧ༝Ͱࢮ๢ͨ͠ͱ͠·͢

    View Slide

  26. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    3FQMJDB4FU
    $POUSPMMFS
    queue
    1PEͷ5FSNJOBUJOH
    TUBUVTDPOEJUJPOT
    ᶊLVCFMFU͕BQJTFSWFSʹϦΫΤετ͠ɺ
    1PEͷ4UBUVTΛߋ৽

    View Slide

  27. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    3FQMJDB4FU
    $POUSPMMFS
    queue
    1PEͷ5FSNJOBUJOH
    ᶋ3FQMJDB4FU$POUSPMMFS͕1PEͷߋ৽Λ
    ݕ஌
    EFMFUF
    ᶋ`3FQMJDB4FU$POUSPMMFS͕1PEΛ࡟আ

    View Slide

  28. ReplicaSetΛApply ʙ σϦόϦ͞ΕΔ·Ͱ
    .BTUFS
    8PSLFS
    FUDE
    /PEF
    $POUBJOFS
    3VOUJNF
    LVCFMFU
    LVCF
    QSPYZ
    BQJTFSWFS
    TDIFEVMFS
    3FQMJDB4FU
    $POUSPMMFS
    queue
    3FDPODJMF
    ᶌ3FQMJDB4FU$POUSPMMFS͕3FDPODJMF͠ɺ
    ๬Μͩঢ়ଶ 3FQMJDBT
    ʹ෮چ͠ɺ
    1PE͕৽͘͠࡞੒͞ΕΔ
    EFMFUF
    Ҏ߱ɺϧʔϓʜ
    ᶄʙᶌͷॲཧΛ܁Γฦ͢

    View Slide

  29. Appendix) ιʔε

    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L487

    https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/eventhandlers.go#L436

    https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/kubelet/config/apiserver.go#L33

    https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/scheduler.go#L535

    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L803

    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kubelet.go#L1527

    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/kubelet/kubelet.go#L2006

    https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/controller/replicaset/replica_set.go#L535

    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L487

    View Slide

  30. Controllerͱίϯϙʔωϯτͨͪ
    ֤ίϯϙʔωϯτͷಇ͖
    ɾapi-server: ResourceͷCRUD
    ɾscheduler: ResourceͷεέδϡʔϦϯά
    ɾkubelet: ίϯςφͷىಈ
    ɾcontroller: ResourceͷReconcile(ௐ੔)
    ֤ίϯϙʔωϯτ͸ͦΕͧΕͷ੹຿ʹूத
    = શମΛ౷཰ɾ౷੍͢Δࢦشऀ͸͍ͳ͍

    View Slide

  31. Controllerͱௐ࿨
    Kubernetes͸ͦΕͧΕͷίϯϙʔωϯτ͕ڠௐͯ͠ಈ͍͍ͯΔ
    Θ͚Ͱ͸ͳ͍
    ໌ࣔతʹ໋ྩΛ͍ͯ͠ͳ͍ͷʹɺશମͱͯ͠੔߹ੑ͕औ
    Ε͍ͯΔ
    ͜͜Ͱ…
    ίϯϙʔωϯτͨͪΛɺͦΕͧΕҰͭͷControllerͱߟ͑Δ
    ͦΕͧΕͷίϯτϩʔϥ͸ɺ
    ͦΕͧΕͷControl LoopΛ࣮ߦ͢Δ͜ͱ͚ͩʹूத͢Δ
    = ෆࢥٞͱɺશମͱͯ͠ͷ੔߹ੑ͕औΕ͍ͯΔ

    View Slide

  32. Kubernetes is jazz improv
    OrchestrationͰ͸ͳ͘ɺ
    Player(Controller)͕֤ʑͷԋ૗
    (Control Loop)ʹूத͢Δ͜ͱͰ
    ଈڵͷԻָͱͯ͠ɺશମ͕੒Γཱ͍ͬͯΔ
    Kubernetes is more jazz improv
    than orchestration.
    Joe Beda
    https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca
    Co-founder
    Core Kubernetes: Jazz Improv over Orchestration

    View Slide

  33. ͳͥControl LoopΛߦ͏ͷ͔ʁ
    Control LoopΛߦ͏ཧ༝Λߟ͑Δ࣌ʹɺॏཁʹͳΔ΋ͷ͕
    ʮEventʯ ※ e.g. Added, Modified, Deleted, Error…
    ෼ࢄίϯϙʔωϯτͰ੒ΓཱͭKubernetesʹͱͬͯɺ
    Event͸ɺͦΕͧΕͷίϯϙʔωϯτΛͭͳ͙ॏཁͳ΋ͷ
    ࢀߟ: Events, the DNA of Kubernetes
    EventΛͲͷΑ͏ʹѻ͏͔ɺ伴ͱͳΔߟ͑ํ͕
    ɾEdge-driven Triggers
    ɾLevel-driven Triggers
    https://www.mgasch.com/post/k8sevents/

    View Slide

  34. Appendix) Edge vs. Level
    https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d
    ࢀߟ: Level Triggering and Reconciliation in Kubernetes
    Edge-driven Triggers
    Level-driven Triggers
    EventͷൃੜʹԠͯ͡Trigger
    ಛఆͷঢ়ଶʹͳΔͱTrigger
    up
    down
    low
    high
    low

    View Slide

  35. ͳͥControl LoopΛߦ͏ͷ͔ʁ
    ͳͥController͕Control LoopΛߦ͏ͷ͔Λߟ͑ΔͨΊʹɺ
    Kubernetes͕એݴతͳڍಈ(Reconcile)Ͱ͸ͳ͘ɺ
    खଓ͖తͳڍಈ(Procedure)Λ͢ΔͱԾఆ
    LJOEYYYYYYYYY
    NFUBEBUB
    OBNFYYYYYY
    TQFD
    ʜ
    ࣮ࡍͷKubernetes Ծఆ
    Reconcile Procedure

    View Slide

  36. ΋͠΋Controller͕खଓ͖ܕͩͬͨΒ
    ݱࡏͷ3FQMJDBT਺
    ˢ




    ˢ


    ݮ
    खଓ͖తʹಈͨ͘ΊɺEvent͕ൃੜ͢Δͨͼʹ
    Edge-driven Triggerͱͯ͠ಈ͘
    Ұݟ໰୊ͳ͍Α͏ʹݟ͑Δ͕ɺऑ఺΋͋Δ
    &WFOU

    View Slide

  37. ΋͠΋Reconcile͕ͳ͔ͬͨΒ
    ݱࡏͷ3FQMJDBT਺
    ˢ



    ˢ


    ݮ
    Ұ࣌తͳωοτϫʔΫͷෆௐ΍όάͳͲͷো֐͕͋ͬͨ࣌ʹ
    Eventͷ৘ใ͕ࣦΘΕͯ͠·͏

    ֐
    &WFOU

    View Slide

  38. Resync IntervalͱReconcile
    ݱࡏͷ3FQMJDBT਺
    ˢ




    ˢ


    ݮ
    Resync Interval͝ͱʹReconcile(ௐ੔)Λ͢Δ͜ͱͰɺ
    Event͕ࣦΘΕͯ΋ཧ૝తͳঢ়ଶʹ͚ۙͮΔ

    ֐
    SFTZODJOUFSWBM
    &WFOU
    Kubernetes = Edge-driven Trigger + Level-driven Trigger

    View Slide

  39. ControllerΛࢧ͑Δɺ
    ஌͓ͬͯ͘΂͖ίϯϙʔωϯτ
    ʙ ϛυϧϨϕϧΞʔΩςΫνϟ ʙ

    View Slide

  40. ༻ޠͷ੔ཧ
    Kind:
    API Objectͷछྨ(Deployment, ServiceͳͲ)

    Resource:
    Kindͱಉ͡ҙຯ߹͍Ͱ༻͍ΒΕΔɻ
    HTTP Endpointͱͯ͠΋ར༻͞ΕΔɻ
    খจࣈ͔ͭෳ਺ܗͰදݱ͞ΕΔ(pods, servicesͳͲ)
    Object:
    ࡞੒͞ΕͨAPI Object ͷ࣮ମ
    Metadata΍Spec, StatusͳͲͷσʔλΛอ࣋͢Δ

    View Slide

  41. ControllerΛࢧ͑ΔϥΠϒϥϦ
    client-go
    Informer
    Lister
    WorkQueue
    api-machinery
    runtime.Object
    Scheme
    code-generator
    ϥΠϒϥϦ ίϯϙʔωϯτ
    লུ

    View Slide

  42. Appendix) Custom Controller SDK
    Kubebuilder
    Informer
    ϑϨʔϜϫʔΫ
    ίϯϙʔωϯτ
    Lister
    Operator SDK
    ϥΠϒϥϦ
    (ϋΠϨϕϧ)
    controller-runtime controller-tools
    ϥΠϒϥϦ
    (ϩʔϨϕϧ)
    client-go api-machinery etc…
    Scheme
    runtime.Object
    WorkQueue
    etc…

    View Slide

  43. ControllerΛࢧ͑ΔϥΠϒϥϦ
    client-go:
    KubernetesͷެࣜΫϥΠΞϯτϥΠϒϥϦ
    Kubernetesຊମͷ։ൃʹ΋࢖ΘΕ͍ͯΔ
    Controller࡞੒ʹ͸͔ܽͤͳ͍ϥΠϒϥϦ
    api-machinery:
    Kubernetes API Object & Kubernetes API like Object
    ʹඞཁͳػೳΛඋ͑ͨϥΠϒϥϦ
    e.g. conversion, decode, encode, etc…
    Controller͸API ObjectΛѻ͏ͷͰɺඞཁʹͳΔ
    code-generator:
    Informer, Lister, clientset, DeepCopyͳͲͷίʔυΛੜ੒

    View Slide

  44. ControllerΛࢧ͑Δίϯϙʔωϯτ
    ͦΕͧΕͷৄࡉ͸ɺControllerͷ࢓૊ΈΛࢀরͭͭ͠ޙड़͠·͢ɻ
    Informer:
    ObjectͷEventΛ؂ࢹ͠ɺin-memory-cacheʹσʔλΛ֨ೲ͢Δ
    Lister:
    in-memory-cache͔ΒσʔλΛऔಘ͢Δ
    WorkQueue:
    Controller͕ॲཧ͢ΔΞΠςϜΛొ࿥͓ͯ͘͠Ωϡʔ
    runtime.Object:
    શͯͷAPI Objectڞ௨ͷInterface
    Scheme:
    Kubernetes APIͱGo TypeΛඥ෇͚Δػߏ
    ࠓճ͸આ໌লུ

    View Slide

  45. client-go
    Informer
    ʙ ϩʔϨϕϧΞʔΩςΫνϟ ʙ

    View Slide

  46. client-goͱInformer
    client-go
    Informer
    ϥΠϒϥϦ
    ίϯϙʔωϯτ
    Reflector DeltaFIFO Indexer
    Store Lister

    View Slide

  47. Informer
    Informer͸Objectͷมߋ(Event)Λ؂ࢹ͢Δίϯϙʔωϯτ
    ObjectͷมߋΛ؂ࢹ͢ΔͨΊʹɺapi-serverʹঢ়ଶΛຖճ
    ໰͍߹ΘͤΔͱɺapi-serverʹෛՙ͕͔͔Δ
    ὎ in-memory-cacheʹObjectͷσʔλΛ֨ೲ͠ɺ
    ͦͷcacheΛࢀর͢Δ͜ͱͰapi-server΁ͷෛՙΛܰݮͤ͞Δ
    ղܾࡦΛ࠾༻
    $POUSPMMFS $POUSPMMFS
    *OGPSNFS
    in-memory-cache
    watch
    watch

    View Slide

  48. Appendix) InformerΛ࢖࣮ͬͨ૷
    func main() {
    ...
    clientset, err := kubernetes.NewForConfig(config)
    // Create InformerFactory
    informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30)
    // Create pod informer by informerFactory
    podInformer := informerFactory.Core().V1().Pods()
    // Add EventHandler to informer
    podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: func(new interface{}) { log.Println("Added") },
    UpdateFunc: func(old, new interface{}) { log.Println("Updated") },
    DeleteFunc: func(old interface{}) { log.Println("Deleted") },
    })
    // Start Go routines
    informerFactory.Start(wait.NeverStop)
    // Wait until finish caching with List API
    informerFactory.WaitForCacheSync(wait.NeverStop)
    // Create Pod Lister
    podLister := podInformer.Lister()
    // Get List of pods
    _, err = podLister.List(labels.Nothing())

    } https://github.com/govargo/kubecontorller-book-sample-snippet/blob/master/02/podinformer/podinformer.go
    ᶃ kubeconfig͔ΒconfigΛੜ੒
    ᶄ config͔ΒclientsetΛੜ੒
    ᶅ informerFactoryੜ੒
    ᶆ ᶅΛ࢖ͬͯinformerੜ੒
    ᶇ EventHandlerΛ௥Ճ
    Add, Update, Delete࣌ʹൃՐ
    ᶈ InformerΛىಈ
    ᶉ Lister(Getter)Λੜ੒
    ᶊ ListerΛ࢖ͬͯGet΍List

    View Slide

  49. Appendix) Shared Informer
    InformerΛར༻͢Δͱ͖͸ɺInformerͦͷ΋ͷͰ͸ͳ͘ɺ
    Shared Informerͱݺ͹ΕΔ΋ͷΛར༻͢Δ
    Shared Informer͸ɺͻͱͭͷόΠφϦ಺Ͱಉ͡ResourceΛڞ༗
    Ͱ͖Δ
    %FQMPZNFOU 3FQMJDB4FU %BFNPO4FU
    4FSWJDF +PC ʜ
    4IBSFE*OGPSNFSTIBSFEDBDIFGPSTBNFSFTPVSDF
    kube-controller-manager

    View Slide

  50. https://github.com/kubernetes/sample-controller/blob/master/docs/controller-client-go.md
    Ҿ༻ݩ:
    InformerͱWorkQueue֓؍

    View Slide

  51. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    -JTU8BUDI
    JONFNPSZDBDIF 4UPSF

    *OEFYFS
    -JTUFS
    (FUPS-JTU
    FHDPSFW1PE
    (FUPS-JTU
    3FRVFTU
    *OGPSNFS
    )BOEMF%FMUBT
    ৄࡉInformer

    View Slide

  52. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    ᶃ-JTU"OE8BUDI
    JONFNPSZDBDIF 4UPSF

    *OEFYFS
    *OGPSNFS
    )BOEMF%FMUBT
    ৄࡉInformer ~σʔλΛΩϟογϡ~

    View Slide

  53. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    ᶄ1PQ
    JONFNPSZDBDIF 4UPSF

    *OEFYFS
    *OGPSNFS
    )BOEMF%FMUBT
    8BUDI
    ৄࡉInformer ~σʔλΛΩϟογϡ~

    View Slide

  54. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    ᶅ)BOEMF%FMUB
    JONFNPSZDBDIF 4UPSF

    *OEFYFS
    *OGPSNFS
    )BOEMF%FMUBT
    8BUDI
    ৄࡉInformer ~σʔλΛΩϟογϡ~

    View Slide

  55. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    ᶆJOEFYFS"EE
    JONFNPSZDBDIF 4UPSF

    *OEFYFS
    *OGPSNFS
    )BOEMF%FMUBT
    8BUDI
    ৄࡉInformer ~σʔλΛΩϟογϡ~

    View Slide

  56. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    JONFNPSZDBDIF 4UPSF

    *OEFYFS
    *OGPSNFS
    )BOEMF%FMUBT
    8BUDI
    ᶅ)BOEMF%FMUB
    ৄࡉInformer ~σʔλΛΩϟογϡ~
    ᶄ1PQ

    View Slide

  57. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    JONFNPSZDBDIF 4UPSF

    *OEFYFS
    *OGPSNFS
    )BOEMF%FMUBT
    8BUDI
    ᶆJOEFYFS"EE
    ৄࡉInformer ~σʔλΛΩϟογϡ~

    View Slide

  58. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    JONFNPSZDBDIF 4UPSF

    *OEFYFS
    *OGPSNFS
    )BOEMF%FMUBT
    8BUDI
    ৄࡉInformer ~σʔλΛऔΓग़͢~
    -JTUFS
    (FUPS-JTU
    3FRVFTU

    View Slide

  59. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    JONFNPSZDBDIF 4UPSF

    *OEFYFS
    *OGPSNFS
    )BOEMF%FMUBT
    8BUDI
    JOEFYFS(FU#Z,FZ
    PS
    JOEFYFS-JTU
    ৄࡉInformer ~σʔλΛऔΓग़͢~
    -JTUFS
    (FUPS-JTU
    3FRVFTU

    View Slide

  60. Appendix) Informer cache ιʔε

    https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/reflector.go#L188

    https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L153

    https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/shared_informer.go#L455

    https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/shared_informer.go#L464

    View Slide

  61. Informerͱίϯϙʔωϯτ
    Informer:
    ObjectͷEventΛ؂ࢹ͠ɺin-memory-cacheʹσʔλΛ֨ೲ͢Δ
    Reflector:
    api-serverͷEventΛ؂ࢹ͢Δ
    DeltaFIFO:
    ObjectͷEvent͕ൃੜ͢Δͨͼʹ࢖͏FIFOΩϡʔ
    Indexer:
    in-memory-cache΁ͷॻ͖ࠐΈ΍ಡΈࠐΈΛߦ͏
    Store:
    in-memory-cache
    Lister:
    in-memory-cache͔ΒIndexerܦ༝ͰσʔλΛऔಘ͢Δ

    View Slide

  62. client-go
    WorkQueue
    ʙ ϩʔϨϕϧΞʔΩςΫνϟ ʙ

    View Slide

  63. client-goͱWorkQueue
    client-go
    WorkQueue
    ϥΠϒϥϦ
    ίϯϙʔωϯτ
    RateLimitingQueue
    DelayedQueue

    View Slide

  64. WorkQueue
    WorkQueue͸ɺDeltaFIFOͱ͸ҟͳΔ΋͏ͻͱͭͷΩϡʔ
    WorkQueue͸ɺControl LoopͷΞΠςϜΛཷΊ͓ͯͨ͘Ίʹ
    ࢖ΘΕΔ
    ඪ४ControllerͰ͸ɺInformerͷEventHandler͕ൃՐ͢Δͱɺ
    WorkQueueʹΞΠςϜΛenqueue͢Δ࣮૷ʹͳ͍ͬͯΔ
    Event
    $POUSPMMFS
    Added
    Updated
    Deleted WorkQueue

    View Slide

  65. Appendix) WorkQueueΛ࢖࣮ͬͨ૷
    func main() {
    ...
    clientset, err := kubernetes.NewForConfig(config)
    // Create InformerFactory
    informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30)
    // Create pod informer by informerFactory
    podInformer := informerFactory.Core().V1().Pods()
    // Create RateLimitQueue
    queue := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter())
    // shutdown when process ends
    defer queue.ShutDown()
    // Add EventHandler to informer
    podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: func(old interface{}) {
    var key string
    var err error
    if key, err = cache.MetaNamespaceKeyFunc(old); err != nil {
    runtime.HandleError(err)
    return
    }
    queue.Add(key)
    log.Println("Added: " + key)
    },
    UpdateFunc: func(old, new interface{}) { … },
    DeleteFunc: func(old interface{}) { … },
    })

    }
    https://github.com/govargo/kubecontorller-book-sample-snippet/blob/master/02/workqueue/enqueuePod.go
    ᶃ kubeconfig͔ΒconfigΛੜ੒
    ᶄ config͔ΒclientsetΛੜ੒
    ᶅ informerFactoryੜ੒
    ᶆ ᶅΛ࢖ͬͯinformerੜ੒
    ᶇ EventHandlerΛ௥Ճ
    Add, Update, Delete࣌ʹൃՐ
    ᶈ InformerΛىಈ
    ᶉ Lister(Getter)Λੜ੒
    ᶊ ListerΛ࢖ͬͯGet΍List
    ᶇ WorkQueueੜ੒

    View Slide

  66. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    -JTU8BUDI
    3FTPVSDF
    &WFOU
    )BOEMFS
    JONFNPSZDBDIF
    TUPSF

    *OEFYFS
    8PSL2VFVF
    1SPDFTT
    *UFN
    $POUSPMMFSϩδοΫ
    *OGPSNFS
    &WFOU)BOEMFS
    ৄࡉWorkQueue

    View Slide

  67. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    3FTPVSDF
    &WFOU
    )BOEMFS
    JONFNPSZDBDIF
    TUPSF

    *OEFYFS
    8PSL2VFVF
    1SPDFTT
    *UFN
    $POUSPMMFSϩδοΫ
    *OGPSNFS
    &WFOU)BOEMFS
    ৄࡉWorkQueue ~Enqueue~
    &WFOU
    -JTU8BUDI

    View Slide

  68. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    3FTPVSDF
    &WFOU
    )BOEMFS
    JONFNPSZDBDIF
    TUPSF

    *OEFYFS
    8PSL2VFVF
    1SPDFTT
    *UFN
    $POUSPMMFSϩδοΫ
    *OGPSNFS
    &WFOU)BOEMFS
    ᶃ8BUDI
    ৄࡉWorkQueue ~Enqueue~

    View Slide

  69. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    3FTPVSDF
    &WFOU
    )BOEMFS
    JONFNPSZDBDIF
    TUPSF

    *OEFYFS
    8PSL2VFVF
    1SPDFTT
    *UFN
    $POUSPMMFSϩδοΫ
    *OGPSNFS
    &WFOU)BOEMFS
    ৄࡉWorkQueue ~Enqueue~
    8BUDI
    ᶄ1PQ

    View Slide

  70. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    3FTPVSDF
    &WFOU
    )BOEMFS
    JONFNPSZDBDIF
    TUPSF

    *OEFYFS
    8PSL2VFVF
    1SPDFTT
    *UFN
    $POUSPMMFSϩδοΫ
    *OGPSNFS
    &WFOU)BOEMFS
    ৄࡉWorkQueue ~Enqueue~
    8BUDI

    "EE'VOD
    6QEBUF'VOD
    %FMFUF'VOD

    View Slide

  71. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    3FTPVSDF
    &WFOU
    )BOEMFS
    JONFNPSZDBDIF
    TUPSF

    *OEFYFS
    8PSL2VFVF
    1SPDFTT
    *UFN
    $POUSPMMFSϩδοΫ
    *OGPSNFS
    &WFOU)BOEMFS
    ৄࡉWorkQueue ~Enqueue~
    8BUDI
    namesapce/name
    e.g. default/nginx
    ᶆXPSLRVFVF"EE

    View Slide

  72. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    3FTPVSDF
    &WFOU
    )BOEMFS
    JONFNPSZDBDIF
    TUPSF

    *OEFYFS
    8PSL2VFVF
    1SPDFTT
    *UFN
    $POUSPMMFSϩδοΫ
    *OGPSNFS
    &WFOU)BOEMFS
    ৄࡉWorkQueue ~Enqueue~
    8BUDI

    "EE'VOD
    6QEBUF'VOD
    %FMFUF'VOD

    View Slide

  73. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    3FTPVSDF
    &WFOU
    )BOEMFS
    JONFNPSZDBDIF
    TUPSF

    *OEFYFS
    8PSL2VFVF
    1SPDFTT
    *UFN
    $POUSPMMFSϩδοΫ
    *OGPSNFS
    &WFOU)BOEMFS
    ৄࡉWorkQueue ~Enqueue~
    8BUDI
    ᶆXPSLRVFVF"EE

    View Slide

  74. 3FqFDUPS
    %FMUB'*'0
    RVFVF

    3FTPVSDF
    &WFOU
    )BOEMFS
    JONFNPSZDBDIF
    TUPSF

    *OEFYFS
    8PSL2VFVF
    1SPDFTT
    *UFN
    $POUSPMMFSϩδοΫ
    *OGPSNFS
    &WFOU)BOEMFS
    ৄࡉWorkQueue ~Dequeue~
    8BUDI
    ᶇXPSLRVFVF(FU

    View Slide

  75. Appendix) Informer enqueue ιʔε

    https://github.com/kubernetes/client-go/blob/master/tools/cache/reflector.go#L267

    https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L153
    ᶅ

    https://github.com/kubernetes/client-go/blob/release-13.0/tools/cache/controller.go#L198
    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L153
    ※ ReplicaSet Controllerͷ৔߹

    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L417
    ※ ReplicaSet Controllerͷ৔߹

    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L438
    ※ ReplicaSet Controllerͷ৔߹

    View Slide

  76. Appendix) InformerͱResync Period
    Informer͸ىಈ࣌ʹɺResync Periodͱ͍͏Ҿ਺Λࢦఆ͢Δ
    Informer͸api-server͔ΒObjectͷมߋΛ؂ࢹ͍ͯ͠Δ
    Resync PeriodΛա͗ΔͱɺԿͷEvent͕ൃੜ͍ͯ͠ͳ͍ͯ͘΋
    UpdateFunc͕ݺͼग़͞Εͯɺ݁ՌReconcile͕࣮ߦ͞ΕΔ
    ※͜ͷ࣌ɺResync͸in-memory-cacheΛࢀর͢Δɻ
    Resync(cacheͷࢀর)ͱRelist(api-server΁ͷ໰͍߹Θͤ)͸ผ෺
    informer.Start
    SFTZODQFSJPE
    List
    Watch
    Event
    Reflector
    Added Updated Updated
    Added
    AddFunc UpdateFunc AddFunc UpdateFunc
    Handler

    View Slide

  77. ControllerͷCycle
    Main Logic

    View Slide

  78. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    8PSL2VFVF
    ControllerͷCycle
    Informer + WorkQueue + Controller
    ͷશମͷCycleΛ֬ೝ
    $POUSPMMFS
    ϝΠϯϩδοΫ

    View Slide

  79. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    XBUDI
    8PSL2VFVF
    ʲԾఆʳ
    Pod͕͢Ͱʹ2ͭ͋Δ
    ͔͜͜ΒUpdate Event͕ൃੜ͢Δ

    View Slide

  80. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    6QEBUF
    XBUDI
    8PSL2VFVF

    View Slide

  81. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    XBUDI
    BEE'VOD
    VQEBUF'VOD
    EFMFUF'VOD

    8PSL2VFVF

    View Slide

  82. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    XBUDI
    ᶅXPSL2VFVF"EE
    8PSL2VFVF

    View Slide

  83. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    XBUDI
    ᶅXPSL2VFVF"EE
    8PSL2VFVF

    View Slide

  84. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    XBUDI
    ᶆXPSL2VFVF(FU
    8PSL2VFVF

    View Slide

  85. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    XBUDI
    ᶆXPSL2VFVF(FU
    8PSL2VFVF

    View Slide

  86. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    XBUDI
    ᶇ4ZOD)BOEMFS
    3FDPODJMF
    8PSL2VFVF

    View Slide

  87. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    XBUDI
    ᶈXPSL2VFVF'PSHFU
    ᶉXPSL2VFVF%POF
    Reconcile͕ਖ਼ৗऴྃͨ͠Β
    ΞΠςϜΛ8PSL2VFVF͔Βআ֎
    8PSL2VFVF

    View Slide

  88. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    XBUDI
    ᶇ`XPSL2VFVF"EE3BUF-JNJUFE
    Reconcile͕Τϥʔ
    8PSL2VFVFʹ3FRVFVFͯ͠ɺ3FDPODJMFΛ࠶ࢼߦ
    8PSL2VFVF

    View Slide

  89. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    XBUDI
    ᶇ`XPSL2VFVF"EE3BUF-JNJUFE
    Reconcile͕Τϥʔ
    8PSL2VFVFʹ3FRVFVFͯ͠ɺ3FDPODJMFΛ࠶ࢼߦ
    8PSL2VFVF

    View Slide

  90. *OGPSNFS
    1SPDFTT
    *UFN
    3FTPVSDF
    &WFOU
    )BOEMFS
    ControllerͷCycle
    Event͕ൃੜ͢ΔͨͼʹWorkQueueʹΞΠςϜ͕ೖΓଓ͚Δ
    WorkQueueʹஷ·ͬͨΞΠςϜΛReconcileͰॲཧ͠ଓ͚Δ
    ͜ͷϧʔϓ͕ɺController͕ࢭ·Δ·ͰԆʑͱଓ͘
    8PSL2VFVF

    View Slide

  91. Controllerͷجຊઓུ
    Read͸In-memory-cache͔ΒɺWrite͸api-server΁
    ※cacheΛ͍͡ΔͱҰ؏ੑΛอͭͷ͕೉͍͠ͷͰɺ
    Write͢Δͱ͖͸ɺҰճDeepCopy͔ͯ͠Βߋ৽͢Δ
    e.g. kubernetes/pkg/controller/replicaset/replica_set.go
    rs = rs.DeepCopy()
    newStatus := calculateStatus(rs, filteredPods, manageReplicasErr)
    // Always updates status as pods come up or die.
    updatedRS, err := updateReplicaSetStatus(rsc.kubeClient.AppsV1(). ὎
    ReplicaSets(rs.Namespace), rs, newStatus)
    https://github.com/kubernetes/kubernetes/blob/release-1.15/pkg/controller/replicaset/replica_set.go#L611

    View Slide

  92. XPSLFS
    XPSLFS
    QSPDFTT/FYU8PSL*UFN
    TZOD)BOEMFS
    ControllerͷMain Logic
    worker:
    processNextWorkItemΛແݶϧʔϓ
    processNextWorkItem:
    WorkQueueΛૢ࡞(Get, Add)ͯ͠
    ReconcileΛݺͼग़͢
    syncHandler:
    Reconcileʹ͋ͨΔϝΠϯϩδοΫ
    Add
    TZOD)BOEMFS
    Update Delete
    Event
    Reconcile
    Eventͷछྨʹ͔͔ΘΒͣɺ
    Reconcile͢Δ

    View Slide

  93. Appendix) ReplicaSet Controller
    worker:
    processNextWorkItemΛແݶϧʔϓ
    processNextWorkItem:
    WorkQueueΛૢ࡞(Get, Add)ͯ͠
    ReconcileΛݺͼग़͢
    syncReplicaSet:
    Reconcileʹ͋ͨΔϝΠϯϩδοΫ
    XPSLFS
    XPSLFS
    QSPDFTT/FYU8PSL*UFN
    TZOD3FQMJDB4FU
    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L432
    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L437
    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go#L562
    ReplicaSet Controllerͷιʔε v1.16
    ※Deployment΋ྨࣅͷߏ଄Λ͍ͯ͠Δ

    View Slide

  94. Appendix) in-memory-cacheͱetcdͷಉظ
    in-memory-cache͸InformerΛ࢖ͬͯඇಉظʹetcdͷ৘ใΛ
    Ωϟογϡ͍ͯ͠Δ
    ʮΩϟογϡͱetcdͰ৘ใ͕ζϨͳ͍͔ʁʯͱࢥ͏͔΋͠Εͳ
    ͍͕ɺKubernetesͰ͸resourceVersionͱ͍͏࢓૊Έ͕͋Δɻ
    resourceVersion͕ҟͳΔͱΤϥʔʹͳͬͯɺRecocileΛ࠶ࢼߦ
    ͢ΔͷͰɺζϨ͍ͯͯ΋໰୊͸ͳ͍
    Rv2
    Rv1
    Update
    Rv2
    version too old
    Requeue
    Reconcile
    (retry)

    View Slide

  95. ༻ޠ੔ཧ(ৼΓฦΓ)
    Informer:
    ObjectͷEventΛ؂ࢹ͠ɺin-memory-cacheʹσʔλΛ֨ೲ͢Δ
    EventHandlerΛ௨ͯ͡ɺWorkQueueʹΞΠςϜΛ௥Ճ͢Δ
    Lister:
    ObjectΛऔಘ͍ͨ࣌͠ in-memory-cache͔ΒσʔλΛऔಘ͢Δ
    WorkQueue:
    Controller͕ॲཧ͢ΔΞΠςϜΛొ࿥͓ͯ͘͠Ωϡʔ
    ͜ͷΩϡʔʹஷ·͍ͬͯΔΞΠςϜ͕Reconcileͷର৅
    ReconcileͰΤϥʔ͕ൃੜͨ͠৔߹͸ɺRequeue͞Εɺ
    ControllerଆͰReconcileΛ࠶ࢼߦ͢Δ

    View Slide

  96. Controllerͷ·ͱΊ

    View Slide

  97. Controllerͷ·ͱΊ
    ɾController͸ɺControl Loop(Reconciliation Loop)ʹΑͬͯ
    એݴతͳAPIΛ࣮ݱ
    ɾ෼ࢄίϯϙʔωϯτ͸ͦΕͧΕಠཱ͓ͯ͠Γɺ
    Event͕ίϯϙʔωϯτؒΛ݁ͼ͚ͭΔ
    ɾclient-go, apimachinery, code-generator౳͕ControllerΛࢧ͑Δ
    ɾInformer͸ೋͭͷ໨తΛ࣋ͭ
    ᶃ in-memory-cacheʹObjectͷσʔλΛ֨ೲ͢Δ
    ᶄ EventHandlerܦ༝ͰWorkQueueʹΞΠςϜΛ֨ೲ͢Δ
    ɾWorkQueueʹஷΊΒΕͨΞΠςϜ͸ReconcileͰॲཧ͞ΕΔ
    ※ ͜ΕΒΛ஌Βͳͯ͘΋ɺController͸ར༻Ͱ͖·͢

    View Slide

  98. Deep Dive΁ͷಓ
    ʙ ΋ͬͱਂ͘஌Γ͍ͨਓ޲͚ʹ ʙ
    ɾSample ControllerΛಈ͔͢
    ɾKubernetesຊମͷ࣮૷ΛݟͯΈΔ
    - Deployment Controller
    - ReplicaSet Controller
    ɾCustom ControllerΛ࡞ͬͯΈΔ(+ CRD)
    ϦϯΫ: https://github.com/kubernetes/sample-controller
    mkdir -p $GOPATH/src/k8s.io && cd $GOPATH/src/k8s.io && git clone https://github.com/kubernetes/sample-controller.git
    export GO111MODULE=on
    go build -o sample-controller .
    ./sample-controller -kubeconfig $HOME/.kube/config
    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/deployment/deployment_controller.go
    https://github.com/kubernetes/kubernetes/blob/release-1.16/pkg/controller/replicaset/replica_set.go

    View Slide

  99. Programming Kubernetes
    https://programming-kubernetes.info/
    ঺հϒϩάࢀߟ: https://go-vargo.hatenablog.com/entry/2019/08/05/201546
    O’Reilly Media, Inc.ΑΓग़൛
    ʮProgramming Kubernetesʯ
    Stefan Schimanski, Michael Hausenblasஶ
    1~2ষ: Kubernetesͷجຊࢥ૝, API Object
    3ষ: client-go
    4ষ: CRD
    5ষ: code-generator
    6~7ষ: Custom Controller
    8~9ষ: Custom API Server, CRDԠ༻ػೳ

    View Slide

  100. Thank you

    View Slide

  101. Reference

    View Slide

  102. Reference
    ɾWeb Article
    - https://kubernetes.io/docs/concepts/architecture/nodes/
    - https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/
    - https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md
    - A deep dive into Kubernetes controllers
    (https://engineering.bitnami.com/articles/a-deep-dive-into-kubernetes-controllers.html)
    - Core Kubernetes: Jazz Improv over Orchestration
    (https://blog.heptio.com/core-kubernetes-jazz-improv-over-orchestration-a7903ea92ca)
    - Events, the DNA of Kubernetes(https://www.mgasch.com/post/k8sevents/)
    ɾPresentation
    - Kubernete Meetup Tokyo #18 - Kubebuilder/controller-runtime ೖ໳
    (https://www.slideshare.net/pfi/kubernete-meetup-tokyo-18-kubebuildercontrollerruntime)
    - KubernetesͷιʔείʔυϦʔσΟϯάೖ໳
    (https://speakerdeck.com/smatsuzaki/kubernetesfalsesosukodorideinguru-men)
    ɾBook
    - Programming Kubernetes (https://programming-kubernetes.info/)

    View Slide

  103. Reference
    ɾRepository
    - Kubernetes(https://github.com/kubernetes/kubernetes)
    - Sample Controller(https://github.com/kubernetes/sample-controller)
    - client-go(https://github.com/kubernetes/client-go)
    - apimachinery(https://github.com/kubernetes/apimachinery)
    - codegenerator(https://github.com/kubernetes/code-generator)
    - what-happens-when-k8s(https://github.com/jamiehannaford/what-happens-when-k8s)

    View Slide