Leader Election in Kubernetes #k8sjp

https://k8sjp.connpass.com/event/162343/

1. 20/01/29 Kubernetes Meetup Tokyo #27 @ponde_m Leader Election in Kubernetes
2. I work at @ponde_m @d-kuro
3. Leader Election ͱ͸ • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩ ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕ ·Ε·͢ɻ” https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/
4. • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩ ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕ ·Ε·͢ɻ” https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/ Leader Election ͱ͸
5. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment Controller Watch Store kubectl
6. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3
7. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 API Server Controller Manager Deployment Controller Watch Create ReplicaSet replica: 3
8. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 -> 2 Update
9. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 -> 2 Update API Server Controller Manager Deployment Controller Watch Update ReplicaSet replica: 3 -> 2
10. Reconciliation Loop Provided in Managing Kubernetes: Operating Kubernetes Clusters in the Real World, 23. https://www.oreilly.com/library/view/managing-kubernetes/9781492033905/
11. API Server Create Deployment Controller Manager Deployment Controller Watch Controller Manager Deployment Controller kubectl 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
12. API Server Create Deployment Controller Manager Deployment Controller Watch Controller Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
13. API Server Create Deployment Controller Manager Deployment Controller Watch Controller Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
14. API Server Create Deployment Controller Manager Deployment Controller Watch Controller Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 API Server Controller Manager Deployment Controller Create ReplicaSet replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
15. API Server Create Deployment Controller Manager Deployment Controller Watch Controller Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 API Server Controller Manager Deployment Controller Create ReplicaSet replica: 3 Error from server (AlreadyExists): deployments.apps “foo" already exists 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
16. API Server Controller Manager Deployment Controller Watch Controller Manager Deployment Controller Leader Election in Kubernetes -FBEFS
17. API Server Controller Manager Deployment Controller Watch Controller Manager Deployment Controller Leader Election in Kubernetes -FBEFS
18. API Server Controller Manager Deployment Controller Watch Controller Manager Deployment Controller Leader Election in Kubernetes -FBEFS
19. Leader Election in Kubernetes • Kubernetes ͷ Leader Election ͸ ڝ߹Λආ͚ͭͭߴՄ༻ੑΛอͭͨΊͷ ΢ΥʔϜελϯόΠͷ࢓૊Έͱͯ͠࢖ΘΕ͍ͯΔ • ϥΠϒϥϦͱͯ͠ఏڙ͞Ε͍ͯΔͷͰ ࠓճઆ໌ͨ͠ Controller ͷྫҎ֎Ͱ΋ Kubernetes ্Ͱಈ͘ΞϓϦέʔγϣϯͰ Leader Election ͍ͨ͠ͱ͍͏৔߹ʹ΋࢖༻Ͱ͖Δ
20. Leader Election ͷ࣮૷ • Leader-for-life • Operator SDK ͕ఏڙ͍ͯ͠Δ࣮૷ • github.com/operator-framework/operator-sdk/pkg/leader • Leader-with-lease • controller-runtime(client-go) ͕ఏڙ͍ͯ͠Δ࣮૷ • github.com/kubernetes-sigs/controller-runtime/pkg/leaderelection
21. Leader Election ͷ࣮૷ • Leader-for-life • NotReady Node ʹ Pod ͕͋ΔͱϦʔμʔͷબग़͕஗͘ͳΔ (pod-eviction-timeout, default: 5m ʹґଘ͢Δ) • Split Brain ͷϦεΫͳ͠ • Leader-with-lease • Split Brain ͷϦεΫ͕͋Δ • NotReady Node ʹ Pod ͕͋ͬͯ΋Ϧʔμʔͷ੾Γସ͕͑ૣ͍
22. Leader-for-life • OwnerReference ͕Ϧʔμʔͷ Pod Ͱ͋Δ ConfigMap Λ࡞੒͠, ϩοΫ͢Δ • Pod ͕࡟আ͞ΕΔͱ Kubernetes ͷΨϕʔδίϨΫγϣϯͷ ࢓૊ΈʹΑΓ ConfigMap ΋ࣗಈతʹ࡟আ͞ΕΔͨΊ, ଞͷ Pod ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ
23. Leader-for-life
24. Leader-for-life
25. API Server Controller Pod A Watch Controller Pod B Leader-for-life ConfigMap Lock Create ConfigMap
    
    PXOFS3FGFSFODFT
    
    
    
    BQJ7FSTJPO
    W
    
    
    
    
    LJOE
    1PE
    
    
    
    
    OBNF
    $POUSPMMFS
    1PE
    "
26. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap Lock Create ConfigMap
    
    PXOFS3FGFSFODFT
    
    
    
    BQJ7FSTJPO
    W
    
    
    
    
    LJOE
    1PE
    
    
    
    
    OBNF
    $POUSPMMFS
    1PE
    "
27. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap Lock Create ConfigMap
    
    PXOFS3FGFSFODFT
    
    
    
    BQJ7FSTJPO
    W
    
    
    
    
    LJOE
    1PE
    
    
    
    
    OBNF
    $POUSPMMFS
    1PE
    "
28. API Server Controller Pod A Controller Pod B Leader-for-life
29. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap Lock
    
    PXOFS3FGFSFODFT
    
    
    
    BQJ7FSTJPO
    W
    
    
    
    
    LJOE
    1PE
    
    
    
    
    OBNF
    $POUSPMMFS
    1PE
    # Create ConfigMap Watch
30. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap Lock
    
    PXOFS3FGFSFODFT
    
    
    
    BQJ7FSTJPO
    W
    
    
    
    
    LJOE
    1PE
    
    
    
    
    OBNF
    $POUSPMMFS
    1PE
    # Create ConfigMap Watch
31. Leader-with-lease • ConfigMap ·ͨ͸ Endpoints Λ༻͍ͯϩοΫ͢Δ • Ϧʔεظ͕ؒઃఆ͞Ε͓ͯΓ, Ϧʔμʔ͸ఆظతʹϦʔεͷߋ৽Λߦ͏ • ԿΒ͔ͷཧ༝ͰϦʔε͕ظݶ੾Εʹͳͬͨࡍʹ͸ ଴ػ͍ͯͨ͠ Controller ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ
32. Leader-with-lease
33. Leader-with-lease
34. Leader-with-lease
35. API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record Check lease record
36. Check lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record observedTime: 2020-01-28T05:33:18Z holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
37. observedTime: 2020-01-28T05:33:18Z Check lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record Controller Pod A ConfigMap Lock Update lease record Ϧʔμʔ͸ఆظతʹϦʔεΛߋ৽͢Δ (ߋ৽͢Δͱ renewTime ͕ॻ͖׵ΘΔ) holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
38. Check lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record PCTFSWFE5JNF
    5;
    Controller Pod A ConfigMap Lock Update lease record observedTime: 2020-01-28T05:33:18Z Controller ͸಺෦Ͱ observedTime ͱ͍͏஋Λ͓࣋ͬͯΓ ൑ఆʹ࢖༻͢Δ holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
39. Check lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
40. Check lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch IPMEFS*EFOUJUZ
    1PE
    "
    MFBTF%VSBUJPO4FDPOET
    
    BDRVJSF5JNF
    5;
    SFOFX5JNF
    5;
    MFBEFS5SBOTJUJPOT
     Check lease record Controller Pod B ConfigMap Lock holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80 PCTFSWFE5JNF
    5;
    observedTime + leaseDurationSeconds < now ͳΒ ԿΒ͔ͷཧ༝ͰϦʔε͕ߋ৽͞Ε͍ͯͳ͍ͨΊ ࣗ෼͕ϦʔμʔʹͳΖ͏ͱߋ৽ΛࢼΈΔ
41. Update lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch holderIdentity: Pod B leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
42. Update lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch Check lease record holderIdentity: Pod B leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
43. ·ͱΊ • Kubernetes Ͱ͸ Object Λ෼ࢄϩοΫʹ࢖༻͢Δ͜ͱͰ Leader Election Λ࣮ݱ͍ͯ͠Δ • ࡉ͔͍ίʔυͷ࿩ͱ͔͸͜͜ʹॻ͖·ͨ͠ • https://d-kuro.github.io/posts/kubernetes-leader-election/
44. Reference • Configuring leader election - Operator SDK • https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-leader-election.html • େମ஻ͬͨ͜ͱ͕ॻ͍ͯ͋Δ • Gopherize.me - A Gopher pic that's as unique as you • https://gopherize.me/ • Gopher ܅͕࡞ΕΔαΠτ • community/icons at master · kubernetes/community • https://github.com/kubernetes/community/tree/master/icons • Kubernetes ͷΞΠίϯηοτ
45. Thank You!