Leader Election in Kubernetes #k8sjp

https://k8sjp.connpass.com/event/162343/

Transcript

1. 20/01/29 Kubernetes Meetup Tokyo #27
   @ponde_m
   Leader Election
   in
   Kubernetes
   

   View Slide

2. I work at
   @ponde_m @d-kuro
   

   View Slide

3. Leader Election ͱ͸
   • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε
   τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩
   ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ
   ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ
   ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕
   ·Ε·͢ɻ”
   https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/
   

   View Slide

4. • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε
   τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩
   ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ
   ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ
   ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕
   ·Ε·͢ɻ”
   https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/
   
   Leader Election ͱ͸
   

   View Slide

5. Kubernetes Architecture
   API Server
   etcd
   Create
   Deployment
   Controller Manager
   Deployment
   Controller
   Watch
   Store
   kubectl
   

   View Slide

6. Kubernetes Architecture
   API Server
   etcd
   Create
   Deployment
   Controller Manager
   Deployment
   Controller
   Watch
   Store
   kubectl
   API Server
   Create
   Deployment
   kubectl replica: 3
   

   View Slide

7. Kubernetes Architecture
   API Server
   etcd
   Create
   Deployment
   Controller Manager
   Deployment
   Controller
   Watch
   Store
   kubectl
   API Server
   Create
   Deployment
   kubectl replica: 3
   API Server Controller Manager
   Deployment
   Controller
   Watch
   Create ReplicaSet
   replica: 3
   

   View Slide

8. Kubernetes Architecture
   API Server
   etcd
   Create
   Deployment
   Controller Manager
   Deployment
   Controller
   Watch
   Store
   kubectl
   API Server
   Create
   Deployment
   kubectl replica: 3 -> 2
   Update
   

   View Slide

9. Kubernetes Architecture
   API Server
   etcd
   Create
   Deployment
   Controller Manager
   Deployment
   Controller
   Watch
   Store
   kubectl
   API Server
   Create
   Deployment
   kubectl replica: 3 -> 2
   Update
   API Server Controller Manager
   Deployment
   Controller
   Watch
   Update ReplicaSet
   replica: 3 -> 2
   

   View Slide

10. Reconciliation Loop
    Provided in Managing Kubernetes: Operating Kubernetes Clusters in the Real World, 23.
    https://www.oreilly.com/library/view/managing-kubernetes/9781492033905/
    

    View Slide

11. API Server
    Create
    Deployment Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    kubectl
    2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
    

    View Slide

12. API Server
    Create
    Deployment Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    kubectl
    API Server
    Create
    Deployment
    kubectl
    replica: 3
    2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
    

    View Slide

13. API Server
    Create
    Deployment Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    kubectl
    API Server
    Create
    Deployment
    kubectl
    replica: 3
    Controller Manager
    Deployment
    Controller
    Watch
    API Server
    Create ReplicaSet
    replica: 3
    2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
    

    View Slide

14. API Server
    Create
    Deployment Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    kubectl
    API Server
    Create
    Deployment
    kubectl
    replica: 3
    Controller Manager
    Deployment
    Controller
    Watch
    API Server
    Create ReplicaSet
    replica: 3
    API Server
    Controller Manager
    Deployment
    Controller Create ReplicaSet
    replica: 3
    2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
    

    View Slide

15. API Server
    Create
    Deployment Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    kubectl
    API Server
    Create
    Deployment
    kubectl
    replica: 3
    Controller Manager
    Deployment
    Controller
    Watch
    API Server
    Create ReplicaSet
    replica: 3
    API Server
    Controller Manager
    Deployment
    Controller Create ReplicaSet
    replica: 3
    Error from server (AlreadyExists):
    deployments.apps
    “foo" already exists
    2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
    

    View Slide

16. API Server
    Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    Leader Election in Kubernetes
    -FBEFS
    

    View Slide

17. API Server
    Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    Leader Election in Kubernetes
    -FBEFS
    

    View Slide

18. API Server
    Controller Manager
    Deployment
    Controller
    Watch
    Controller Manager
    Deployment
    Controller
    Leader Election in Kubernetes
    -FBEFS
    

    View Slide

19. Leader Election in Kubernetes
    • Kubernetes ͷ Leader Election ͸
    ڝ߹Λආ͚ͭͭߴՄ༻ੑΛอͭͨΊͷ
    ΢ΥʔϜελϯόΠͷ࢓૊Έͱͯ͠࢖ΘΕ͍ͯΔ
    • ϥΠϒϥϦͱͯ͠ఏڙ͞Ε͍ͯΔͷͰ
    ࠓճઆ໌ͨ͠ Controller ͷྫҎ֎Ͱ΋
    Kubernetes ্Ͱಈ͘ΞϓϦέʔγϣϯͰ
    Leader Election ͍ͨ͠ͱ͍͏৔߹ʹ΋࢖༻Ͱ͖Δ
    

    View Slide

20. Leader Election ͷ࣮૷
    • Leader-for-life
    • Operator SDK ͕ఏڙ͍ͯ͠Δ࣮૷
    • github.com/operator-framework/operator-sdk/pkg/leader
    • Leader-with-lease
    • controller-runtime(client-go) ͕ఏڙ͍ͯ͠Δ࣮૷
    • github.com/kubernetes-sigs/controller-runtime/pkg/leaderelection
    

    View Slide

21. Leader Election ͷ࣮૷
    • Leader-for-life
    • NotReady Node ʹ Pod ͕͋ΔͱϦʔμʔͷબग़͕஗͘ͳΔ
    (pod-eviction-timeout, default: 5m ʹґଘ͢Δ)
    • Split Brain ͷϦεΫͳ͠
    • Leader-with-lease
    • Split Brain ͷϦεΫ͕͋Δ
    • NotReady Node ʹ Pod ͕͋ͬͯ΋Ϧʔμʔͷ੾Γସ͕͑ૣ͍
    

    View Slide

22. Leader-for-life
    • OwnerReference ͕Ϧʔμʔͷ Pod Ͱ͋Δ
    ConfigMap Λ࡞੒͠, ϩοΫ͢Δ
    • Pod ͕࡟আ͞ΕΔͱ Kubernetes ͷΨϕʔδίϨΫγϣϯͷ
    ࢓૊ΈʹΑΓ ConfigMap ΋ࣗಈతʹ࡟আ͞ΕΔͨΊ,
    ଞͷ Pod ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ
    

    View Slide

23. Leader-for-life
    

    View Slide

24. Leader-for-life
    

    View Slide

25. API Server
    Controller
    Pod A
    Watch
    Controller
    Pod B
    Leader-for-life
    ConfigMap
    Lock
    Create ConfigMap
    PXOFS3FGFSFODFT
    BQJ7FSTJPOW
    LJOE1PE
    OBNF$POUSPMMFS1PE"
    

    View Slide

26. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-for-life
    ConfigMap
    Lock
    Create ConfigMap
    PXOFS3FGFSFODFT
    BQJ7FSTJPOW
    LJOE1PE
    OBNF$POUSPMMFS1PE"
    

    View Slide

27. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-for-life
    ConfigMap
    Lock
    Create ConfigMap
    PXOFS3FGFSFODFT
    BQJ7FSTJPOW
    LJOE1PE
    OBNF$POUSPMMFS1PE"
    

    View Slide

28. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-for-life
    

    View Slide

29. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-for-life
    ConfigMap
    Lock
    PXOFS3FGFSFODFT
    BQJ7FSTJPOW
    LJOE1PE
    OBNF$POUSPMMFS1PE#
    Create ConfigMap
    Watch
    

    View Slide

30. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-for-life
    ConfigMap
    Lock
    PXOFS3FGFSFODFT
    BQJ7FSTJPOW
    LJOE1PE
    OBNF$POUSPMMFS1PE#
    Create ConfigMap
    Watch
    

    View Slide

31. Leader-with-lease
    • ConfigMap ·ͨ͸ Endpoints Λ༻͍ͯϩοΫ͢Δ
    • Ϧʔεظ͕ؒઃఆ͞Ε͓ͯΓ,
    Ϧʔμʔ͸ఆظతʹϦʔεͷߋ৽Λߦ͏
    • ԿΒ͔ͷཧ༝ͰϦʔε͕ظݶ੾Εʹͳͬͨࡍʹ͸
    ଴ػ͍ͯͨ͠ Controller ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ
    

    View Slide

32. Leader-with-lease
    

    View Slide

33. Leader-with-lease
    

    View Slide

34. Leader-with-lease
    

    View Slide

35. API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch Update lease record
    Check lease record
    

    View Slide

36. Check lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch Update lease record
    observedTime: 2020-01-28T05:33:18Z
    holderIdentity: Pod A
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80
    

    View Slide

37. observedTime: 2020-01-28T05:33:18Z
    Check lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch Update lease record
    Controller
    Pod A
    ConfigMap
    Lock
    Update lease record
    Ϧʔμʔ͸ఆظతʹϦʔεΛߋ৽͢Δ
    (ߋ৽͢Δͱ renewTime ͕ॻ͖׵ΘΔ)
    holderIdentity: Pod A
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80
    

    View Slide

38. Check lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch Update lease record
    PCTFSWFE5JNF5;
    Controller
    Pod A
    ConfigMap
    Lock
    Update lease record
    observedTime: 2020-01-28T05:33:18Z
    Controller ͸಺෦Ͱ
    observedTime ͱ͍͏஋Λ͓࣋ͬͯΓ
    ൑ఆʹ࢖༻͢Δ
    holderIdentity: Pod A
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80
    

    View Slide

39. Check lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch
    holderIdentity: Pod A
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80
    

    View Slide

40. Check lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch
    IPMEFS*EFOUJUZ1PE"
    MFBTF%VSBUJPO4FDPOET
    BDRVJSF5JNF5;
    SFOFX5JNF5;
    MFBEFS5SBOTJUJPOT
    Check lease record
    Controller
    Pod B
    ConfigMap
    Lock
    holderIdentity: Pod A
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80
    PCTFSWFE5JNF5;
    observedTime + leaseDurationSeconds < now
    ͳΒ
    ԿΒ͔ͷཧ༝ͰϦʔε͕ߋ৽͞Ε͍ͯͳ͍ͨΊ
    ࣗ෼͕ϦʔμʔʹͳΖ͏ͱߋ৽ΛࢼΈΔ
    

    View Slide

41. Update lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch
    holderIdentity: Pod B
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80
    

    View Slide

42. Update lease record
    API Server
    Controller
    Pod A
    Controller
    Pod B
    Leader-with-lease
    ConfigMap
    Lock
    Watch
    Check lease record
    holderIdentity: Pod B
    leaseDurationSeconds: 60
    acquireTime: 2020-01-27T03:44:58Z
    renewTime: 2020-01-28T05:33:18Z
    leaderTransitions: 80
    

    View Slide

43. ·ͱΊ
    • Kubernetes Ͱ͸ Object Λ෼ࢄϩοΫʹ࢖༻͢Δ͜ͱͰ
    Leader Election Λ࣮ݱ͍ͯ͠Δ
    • ࡉ͔͍ίʔυͷ࿩ͱ͔͸͜͜ʹॻ͖·ͨ͠
    • https://d-kuro.github.io/posts/kubernetes-leader-election/
    

    View Slide

44. Reference
    • Configuring leader election - Operator SDK
    • https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-leader-election.html
    • େମ஻ͬͨ͜ͱ͕ॻ͍ͯ͋Δ
    • Gopherize.me - A Gopher pic that's as unique as you
    • https://gopherize.me/
    • Gopher ܅͕࡞ΕΔαΠτ
    • community/icons at master · kubernetes/community
    • https://github.com/kubernetes/community/tree/master/icons
    • Kubernetes ͷΞΠίϯηοτ
    

    View Slide

45. Thank You!
    

    View Slide