$30 off During Our Annual Pro Sale. View Details »

Leader Election in Kubernetes #k8sjp

d-kuro
January 29, 2020

Leader Election in Kubernetes #k8sjp

d-kuro

January 29, 2020
Tweet

More Decks by d-kuro

Other Decks in Technology

Transcript

  1. 20/01/29 Kubernetes Meetup Tokyo #27
    @ponde_m
    Leader Election
    in
    Kubernetes

    View Slide

  2. I work at
    @ponde_m @d-kuro

    View Slide

  3. Leader Election ͱ͸
    • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε
    τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩
    ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ
    ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ
    ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕
    ·Ε·͢ɻ”
    https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/

    View Slide

  4. • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε
    τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩
    ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ
    ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ
    ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕
    ·Ε·͢ɻ”
    https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/

    Leader Election ͱ͸

    View Slide

  5. Kubernetes Architecture
    API Server
    etcd
    Create
    Deployment
    Controller Manager
    Deployment
    Controller
    Watch
    Store
    kubectl

    View Slide

  6. Kubernetes Architecture
    API Server
    etcd
    Create
    Deployment
    Controller Manager
    Deployment
    Controller
    Watch
    Store
    kubectl
    API Server
    Create
    Deployment
    kubectl replica: 3

    View Slide

  7. Kubernetes Architecture
    API Server
    etcd
    Create
    Deployment
    Controller Manager
    Deployment
    Controller
    Watch
    Store
    kubectl
    API Server
    Create
    Deployment
    kubectl replica: 3
    API Server Controller Manager
    Deployment
    Controller
    Watch
    Create ReplicaSet
    replica: 3

    View Slide

  8. Kubernetes Architecture
    API Server
    etcd
    Create
    Deployment
    Controller Manager
    Deployment
    Controller
    Watch
    Store
    kubectl
    API Server
    Create
    Deployment
    kubectl replica: 3 -> 2
    Update

    View Slide

  9. Kubernetes Architecture
    API Server
    etcd
    Create
    Deployment
    Controller Manager
    Deployment
    Controller
    Watch
    Store
    kubectl
    API Server
    Create
    Deployment
    kubectl replica: 3 -> 2
    Update
    API Server Controller Manager
    Deployment
    Controller
    Watch
    Update ReplicaSet
    replica: 3 -> 2

    View Slide

  10. Reconciliation Loop
    Provided in Managing Kubernetes: Operating Kubernetes Clusters in the Real World, 23.
    https://www.oreilly.com/library/view/managing-kubernetes/9781492033905/

    View Slide

  11. API Server
    Create
    Deployment Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    kubectl
    2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

    View Slide

  12. API Server
    Create
    Deployment Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    kubectl
    API Server
    Create
    Deployment
    kubectl
    replica: 3
    2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

    View Slide

  13. API Server
    Create
    Deployment Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    kubectl
    API Server
    Create
    Deployment
    kubectl
    replica: 3
    Controller Manager
    Deployment
    Controller
    Watch
    API Server
    Create ReplicaSet
    replica: 3
    2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

    View Slide

  14. API Server
    Create
    Deployment Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    kubectl
    API Server
    Create
    Deployment
    kubectl
    replica: 3
    Controller Manager
    Deployment
    Controller
    Watch
    API Server
    Create ReplicaSet
    replica: 3
    API Server
    Controller Manager
    Deployment
    Controller Create ReplicaSet
    replica: 3
    2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

    View Slide

  15. API Server
    Create
    Deployment Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    kubectl
    API Server
    Create
    Deployment
    kubectl
    replica: 3
    Controller Manager
    Deployment
    Controller
    Watch
    API Server
    Create ReplicaSet
    replica: 3
    API Server
    Controller Manager
    Deployment
    Controller Create ReplicaSet
    replica: 3
    Error from server (AlreadyExists):
    deployments.apps
    “foo" already exists
    2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

    View Slide

  16. API Server
    Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    Leader Election in Kubernetes
    -FBEFS

    View Slide

  17. API Server
    Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    Leader Election in Kubernetes
    -FBEFS

    View Slide

  18. API Server
    Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    Leader Election in Kubernetes
    -FBEFS

    View Slide

  19. Leader Election in Kubernetes
    • Kubernetes ͷ Leader Election ͸
    ڝ߹Λආ͚ͭͭߴՄ༻ੑΛอͭͨΊͷ
    ΢ΥʔϜελϯόΠͷ࢓૊Έͱͯ͠࢖ΘΕ͍ͯΔ
    • ϥΠϒϥϦͱͯ͠ఏڙ͞Ε͍ͯΔͷͰ
    ࠓճઆ໌ͨ͠ Controller ͷྫҎ֎Ͱ΋
    Kubernetes ্Ͱಈ͘ΞϓϦέʔγϣϯͰ
    Leader Election ͍ͨ͠ͱ͍͏৔߹ʹ΋࢖༻Ͱ͖Δ

    View Slide

  20. Leader Election ͷ࣮૷
    • Leader-for-life
    • Operator SDK ͕ఏڙ͍ͯ͠Δ࣮૷
    • github.com/operator-framework/operator-sdk/pkg/leader
    • Leader-with-lease
    • controller-runtime(client-go) ͕ఏڙ͍ͯ͠Δ࣮૷
    • github.com/kubernetes-sigs/controller-runtime/pkg/leaderelection

    View Slide

  21. Leader Election ͷ࣮૷
    • Leader-for-life
    • NotReady Node ʹ Pod ͕͋ΔͱϦʔμʔͷબग़͕஗͘ͳΔ
    (pod-eviction-timeout, default: 5m ʹґଘ͢Δ)
    • Split Brain ͷϦεΫͳ͠
    • Leader-with-lease
    • Split Brain ͷϦεΫ͕͋Δ
    • NotReady Node ʹ Pod ͕͋ͬͯ΋Ϧʔμʔͷ੾Γସ͕͑ૣ͍

    View Slide

  22. Leader-for-life
    • OwnerReference ͕Ϧʔμʔͷ Pod Ͱ͋Δ
    ConfigMap Λ࡞੒͠, ϩοΫ͢Δ
    • Pod ͕࡟আ͞ΕΔͱ Kubernetes ͷΨϕʔδίϨΫγϣϯͷ
    ࢓૊ΈʹΑΓ ConfigMap ΋ࣗಈతʹ࡟আ͞ΕΔͨΊ,
    ଞͷ Pod ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ

    View Slide

  23. Leader-for-life

    View Slide

  24. Leader-for-life

    View Slide

  25. API Server
    Controller
    Pod A
    Watch
    Controller
    Pod B
    Leader-for-life
    ConfigMap
    Lock
    Create ConfigMap
    PXOFS3FGFSFODFT
    BQJ7FSTJPOW
    LJOE1PE
    OBNF$POUSPMMFS1PE"

    View Slide

  26. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-for-life
    ConfigMap
    Lock
    Create ConfigMap
    PXOFS3FGFSFODFT
    BQJ7FSTJPOW
    LJOE1PE
    OBNF$POUSPMMFS1PE"

    View Slide

  27. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-for-life
    ConfigMap
    Lock
    Create ConfigMap
    PXOFS3FGFSFODFT
    BQJ7FSTJPOW
    LJOE1PE
    OBNF$POUSPMMFS1PE"

    View Slide

  28. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-for-life

    View Slide

  29. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-for-life
    ConfigMap
    Lock
    PXOFS3FGFSFODFT
    BQJ7FSTJPOW
    LJOE1PE
    OBNF$POUSPMMFS1PE#
    Create ConfigMap
    Watch

    View Slide

  30. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-for-life
    ConfigMap
    Lock
    PXOFS3FGFSFODFT
    BQJ7FSTJPOW
    LJOE1PE
    OBNF$POUSPMMFS1PE#
    Create ConfigMap
    Watch

    View Slide

  31. Leader-with-lease
    • ConfigMap ·ͨ͸ Endpoints Λ༻͍ͯϩοΫ͢Δ
    • Ϧʔεظ͕ؒઃఆ͞Ε͓ͯΓ,
    Ϧʔμʔ͸ఆظతʹϦʔεͷߋ৽Λߦ͏
    • ԿΒ͔ͷཧ༝ͰϦʔε͕ظݶ੾Εʹͳͬͨࡍʹ͸
    ଴ػ͍ͯͨ͠ Controller ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ

    View Slide

  32. Leader-with-lease

    View Slide

  33. Leader-with-lease

    View Slide

  34. Leader-with-lease

    View Slide

  35. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch Update lease record
    Check lease record

    View Slide

  36. Check lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch Update lease record
    observedTime: 2020-01-28T05:33:18Z
    holderIdentity: Pod A
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80

    View Slide

  37. observedTime: 2020-01-28T05:33:18Z
    Check lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch Update lease record
    Controller
    Pod A
    ConfigMap
    Lock
    Update lease record
    Ϧʔμʔ͸ఆظతʹϦʔεΛߋ৽͢Δ
    (ߋ৽͢Δͱ renewTime ͕ॻ͖׵ΘΔ)
    holderIdentity: Pod A
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80

    View Slide

  38. Check lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch Update lease record
    PCTFSWFE5JNF5;
    Controller
    Pod A
    ConfigMap
    Lock
    Update lease record
    observedTime: 2020-01-28T05:33:18Z
    Controller ͸಺෦Ͱ
    observedTime ͱ͍͏஋Λ͓࣋ͬͯΓ
    ൑ఆʹ࢖༻͢Δ
    holderIdentity: Pod A
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80

    View Slide

  39. Check lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch
    holderIdentity: Pod A
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80

    View Slide

  40. Check lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch
    IPMEFS*EFOUJUZ1PE"
    MFBTF%VSBUJPO4FDPOET
    BDRVJSF5JNF5;
    SFOFX5JNF5;
    MFBEFS5SBOTJUJPOT
    Check lease record
    Controller
    Pod B
    ConfigMap
    Lock
    holderIdentity: Pod A
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80
    PCTFSWFE5JNF5;
    observedTime + leaseDurationSeconds < now
    ͳΒ
    ԿΒ͔ͷཧ༝ͰϦʔε͕ߋ৽͞Ε͍ͯͳ͍ͨΊ
    ࣗ෼͕ϦʔμʔʹͳΖ͏ͱߋ৽ΛࢼΈΔ

    View Slide

  41. Update lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch
    holderIdentity: Pod B
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80

    View Slide

  42. Update lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch
    Check lease record
    holderIdentity: Pod B
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80

    View Slide

  43. ·ͱΊ
    • Kubernetes Ͱ͸ Object Λ෼ࢄϩοΫʹ࢖༻͢Δ͜ͱͰ
    Leader Election Λ࣮ݱ͍ͯ͠Δ
    • ࡉ͔͍ίʔυͷ࿩ͱ͔͸͜͜ʹॻ͖·ͨ͠
    • https://d-kuro.github.io/posts/kubernetes-leader-election/

    View Slide

  44. Reference
    • Configuring leader election - Operator SDK
    • https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-leader-election.html
    • େମ஻ͬͨ͜ͱ͕ॻ͍ͯ͋Δ
    • Gopherize.me - A Gopher pic that's as unique as you
    • https://gopherize.me/
    • Gopher ܅͕࡞ΕΔαΠτ
    • community/icons at master · kubernetes/community
    • https://github.com/kubernetes/community/tree/master/icons
    • Kubernetes ͷΞΠίϯηοτ

    View Slide

  45. Thank You!

    View Slide