Leader Election in Kubernetes #k8sjp

Transcript

1. 20/01/29 Kubernetes Meetup Tokyo #27 @ponde_m Leader Election in Kubernetes

2. I work at @ponde_m @d-kuro

3. Leader Election ͱ͸ • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩

   ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕ ·Ε·͢ɻ” https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/

4. • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩ ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕

   ·Ε·͢ɻ” https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/ Leader Election ͱ͸

5. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

   Controller Watch Store kubectl

6. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

   Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3

7. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

   Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 API Server Controller Manager Deployment Controller Watch Create ReplicaSet replica: 3

8. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

   Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 -> 2 Update

9. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

   Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 -> 2 Update API Server Controller Manager Deployment Controller Watch Update ReplicaSet replica: 3 -> 2

10. Reconciliation Loop Provided in Managing Kubernetes: Operating Kubernetes Clusters in

    the Real World, 23. https://www.oreilly.com/library/view/managing-kubernetes/9781492033905/

11. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

12. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

13. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

14. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 API Server Controller Manager Deployment Controller Create ReplicaSet replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

15. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 API Server Controller Manager Deployment Controller Create ReplicaSet replica: 3 Error from server (AlreadyExists): deployments.apps “foo" already exists 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

16. API Server Controller Manager Deployment Controller Watch Controller Manager Deployment

    Controller Leader Election in Kubernetes -FBEFS

17. API Server Controller Manager Deployment Controller Watch Controller Manager Deployment

    Controller Leader Election in Kubernetes -FBEFS

18. API Server Controller Manager Deployment Controller Watch Controller Manager Deployment

    Controller Leader Election in Kubernetes -FBEFS

19. Leader Election in Kubernetes • Kubernetes ͷ Leader Election ͸

    ڝ߹Λආ͚ͭͭߴՄ༻ੑΛอͭͨΊͷ ΢ΥʔϜελϯόΠͷ࢓૊Έͱͯ͠࢖ΘΕ͍ͯΔ • ϥΠϒϥϦͱͯ͠ఏڙ͞Ε͍ͯΔͷͰ ࠓճઆ໌ͨ͠ Controller ͷྫҎ֎Ͱ΋ Kubernetes ্Ͱಈ͘ΞϓϦέʔγϣϯͰ Leader Election ͍ͨ͠ͱ͍͏৔߹ʹ΋࢖༻Ͱ͖Δ

20. Leader Election ͷ࣮૷ • Leader-for-life • Operator SDK ͕ఏڙ͍ͯ͠Δ࣮૷ •

    github.com/operator-framework/operator-sdk/pkg/leader • Leader-with-lease • controller-runtime(client-go) ͕ఏڙ͍ͯ͠Δ࣮૷ • github.com/kubernetes-sigs/controller-runtime/pkg/leaderelection

21. Leader Election ͷ࣮૷ • Leader-for-life • NotReady Node ʹ Pod

    ͕͋ΔͱϦʔμʔͷબग़͕஗͘ͳΔ (pod-eviction-timeout, default: 5m ʹґଘ͢Δ) • Split Brain ͷϦεΫͳ͠ • Leader-with-lease • Split Brain ͷϦεΫ͕͋Δ • NotReady Node ʹ Pod ͕͋ͬͯ΋Ϧʔμʔͷ੾Γସ͕͑ૣ͍

22. Leader-for-life • OwnerReference ͕Ϧʔμʔͷ Pod Ͱ͋Δ ConfigMap Λ࡞੒͠, ϩοΫ͢Δ •

    Pod ͕࡟আ͞ΕΔͱ Kubernetes ͷΨϕʔδίϨΫγϣϯͷ ࢓૊ΈʹΑΓ ConfigMap ΋ࣗಈతʹ࡟আ͞ΕΔͨΊ, ଞͷ Pod ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ

23. Leader-for-life

24. Leader-for-life

25. API Server Controller Pod A Watch Controller Pod B Leader-for-life

    ConfigMap Lock Create ConfigMap
    
    PXOFS3FGFSFODFT
    
    
    
    BQJ7FSTJPO
    W
    
    
    
    
    LJOE
    1PE
    
    
    
    
    OBNF
    $POUSPMMFS
    1PE
    "

26. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock Create ConfigMap
    
    PXOFS3FGFSFODFT
    
    
    
    BQJ7FSTJPO
    W
    
    
    
    
    LJOE
    1PE
    
    
    
    
    OBNF
    $POUSPMMFS
    1PE
    "

27. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock Create ConfigMap
    
    PXOFS3FGFSFODFT
    
    
    
    BQJ7FSTJPO
    W
    
    
    
    
    LJOE
    1PE
    
    
    
    
    OBNF
    $POUSPMMFS
    1PE
    "

28. API Server Controller Pod A Controller Pod B Leader-for-life

29. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock
    
    PXOFS3FGFSFODFT
    
    
    
    BQJ7FSTJPO
    W
    
    
    
    
    LJOE
    1PE
    
    
    
    
    OBNF
    $POUSPMMFS
    1PE
    # Create ConfigMap Watch

30. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock
    
    PXOFS3FGFSFODFT
    
    
    
    BQJ7FSTJPO
    W
    
    
    
    
    LJOE
    1PE
    
    
    
    
    OBNF
    $POUSPMMFS
    1PE
    # Create ConfigMap Watch

31. Leader-with-lease • ConfigMap ·ͨ͸ Endpoints Λ༻͍ͯϩοΫ͢Δ • Ϧʔεظ͕ؒઃఆ͞Ε͓ͯΓ, Ϧʔμʔ͸ఆظతʹϦʔεͷߋ৽Λߦ͏ •

    ԿΒ͔ͷཧ༝ͰϦʔε͕ظݶ੾Εʹͳͬͨࡍʹ͸ ଴ػ͍ͯͨ͠ Controller ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ

32. Leader-with-lease

33. Leader-with-lease

34. Leader-with-lease

35. API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap

    Lock Watch Update lease record Check lease record

36. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch Update lease record observedTime: 2020-01-28T05:33:18Z holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

37. observedTime: 2020-01-28T05:33:18Z Check lease record API Server Controller Pod A

    Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record Controller Pod A ConfigMap Lock Update lease record Ϧʔμʔ͸ఆظతʹϦʔεΛߋ৽͢Δ (ߋ৽͢Δͱ renewTime ͕ॻ͖׵ΘΔ) holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

38. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch Update lease record PCTFSWFE5JNF
    5;
    Controller Pod A ConfigMap Lock Update lease record observedTime: 2020-01-28T05:33:18Z Controller ͸಺෦Ͱ observedTime ͱ͍͏஋Λ͓࣋ͬͯΓ ൑ఆʹ࢖༻͢Δ holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

39. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

40. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch IPMEFS*EFOUJUZ
    1PE
    "
    MFBTF%VSBUJPO4FDPOET
    
    BDRVJSF5JNF
    5;
    SFOFX5JNF
    5;
    MFBEFS5SBOTJUJPOT
     Check lease record Controller Pod B ConfigMap Lock holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80 PCTFSWFE5JNF
    5;
    observedTime + leaseDurationSeconds < now ͳΒ ԿΒ͔ͷཧ༝ͰϦʔε͕ߋ৽͞Ε͍ͯͳ͍ͨΊ ࣗ෼͕ϦʔμʔʹͳΖ͏ͱߋ৽ΛࢼΈΔ

41. Update lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch holderIdentity: Pod B leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

42. Update lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch Check lease record holderIdentity: Pod B leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

43. ·ͱΊ • Kubernetes Ͱ͸ Object Λ෼ࢄϩοΫʹ࢖༻͢Δ͜ͱͰ Leader Election Λ࣮ݱ͍ͯ͠Δ •

    ࡉ͔͍ίʔυͷ࿩ͱ͔͸͜͜ʹॻ͖·ͨ͠ • https://d-kuro.github.io/posts/kubernetes-leader-election/

44. Reference • Configuring leader election - Operator SDK • https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-leader-election.html

    • େମ஻ͬͨ͜ͱ͕ॻ͍ͯ͋Δ • Gopherize.me - A Gopher pic that's as unique as you • https://gopherize.me/ • Gopher ܅͕࡞ΕΔαΠτ • community/icons at master · kubernetes/community • https://github.com/kubernetes/community/tree/master/icons • Kubernetes ͷΞΠίϯηοτ

45. Thank You!