Leader Election in Kubernetes #k8sjp

Cad656ed619672b702191833dc819943?s=47 d-kuro
January 29, 2020

Leader Election in Kubernetes #k8sjp

Cad656ed619672b702191833dc819943?s=128

d-kuro

January 29, 2020
Tweet

Transcript

  1. 20/01/29 Kubernetes Meetup Tokyo #27 @ponde_m Leader Election in Kubernetes

  2. I work at @ponde_m @d-kuro

  3. Leader Election ͱ͸ • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩

    ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕ ·Ε·͢ɻ” https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/
  4. • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩ ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕

    ·Ε·͢ɻ” https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/ Leader Election ͱ͸
  5. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

    Controller Watch Store kubectl
  6. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

    Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3
  7. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

    Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 API Server Controller Manager Deployment Controller Watch Create ReplicaSet replica: 3
  8. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

    Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 -> 2 Update
  9. Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment

    Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 -> 2 Update API Server Controller Manager Deployment Controller Watch Update ReplicaSet replica: 3 -> 2
  10. Reconciliation Loop Provided in Managing Kubernetes: Operating Kubernetes Clusters in

    the Real World, 23. https://www.oreilly.com/library/view/managing-kubernetes/9781492033905/
  11. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
  12. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
  13. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
  14. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 API Server Controller Manager Deployment Controller Create ReplicaSet replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
  15. API Server Create Deployment Controller Manager Deployment Controller Watch Controller

    Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 API Server Controller Manager Deployment Controller Create ReplicaSet replica: 3 Error from server (AlreadyExists): deployments.apps “foo" already exists 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹
  16. API Server Controller Manager Deployment Controller Watch Controller Manager Deployment

    Controller Leader Election in Kubernetes -FBEFS
  17. API Server Controller Manager Deployment Controller Watch Controller Manager Deployment

    Controller Leader Election in Kubernetes -FBEFS
  18. API Server Controller Manager Deployment Controller Watch Controller Manager Deployment

    Controller Leader Election in Kubernetes -FBEFS
  19. Leader Election in Kubernetes • Kubernetes ͷ Leader Election ͸

    ڝ߹Λආ͚ͭͭߴՄ༻ੑΛอͭͨΊͷ ΢ΥʔϜελϯόΠͷ࢓૊Έͱͯ͠࢖ΘΕ͍ͯΔ • ϥΠϒϥϦͱͯ͠ఏڙ͞Ε͍ͯΔͷͰ ࠓճઆ໌ͨ͠ Controller ͷྫҎ֎Ͱ΋ Kubernetes ্Ͱಈ͘ΞϓϦέʔγϣϯͰ Leader Election ͍ͨ͠ͱ͍͏৔߹ʹ΋࢖༻Ͱ͖Δ
  20. Leader Election ͷ࣮૷ • Leader-for-life • Operator SDK ͕ఏڙ͍ͯ͠Δ࣮૷ •

    github.com/operator-framework/operator-sdk/pkg/leader • Leader-with-lease • controller-runtime(client-go) ͕ఏڙ͍ͯ͠Δ࣮૷ • github.com/kubernetes-sigs/controller-runtime/pkg/leaderelection
  21. Leader Election ͷ࣮૷ • Leader-for-life • NotReady Node ʹ Pod

    ͕͋ΔͱϦʔμʔͷબग़͕஗͘ͳΔ (pod-eviction-timeout, default: 5m ʹґଘ͢Δ) • Split Brain ͷϦεΫͳ͠ • Leader-with-lease • Split Brain ͷϦεΫ͕͋Δ • NotReady Node ʹ Pod ͕͋ͬͯ΋Ϧʔμʔͷ੾Γସ͕͑ૣ͍
  22. Leader-for-life • OwnerReference ͕Ϧʔμʔͷ Pod Ͱ͋Δ ConfigMap Λ࡞੒͠, ϩοΫ͢Δ •

    Pod ͕࡟আ͞ΕΔͱ Kubernetes ͷΨϕʔδίϨΫγϣϯͷ ࢓૊ΈʹΑΓ ConfigMap ΋ࣗಈతʹ࡟আ͞ΕΔͨΊ, ଞͷ Pod ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ
  23. Leader-for-life

  24. Leader-for-life

  25. API Server Controller Pod A Watch Controller Pod B Leader-for-life

    ConfigMap Lock Create ConfigMap PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE"
  26. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock Create ConfigMap PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE"
  27. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock Create ConfigMap PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE"
  28. API Server Controller Pod A Controller Pod B Leader-for-life

  29. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE# Create ConfigMap Watch
  30. API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap

    Lock PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE# Create ConfigMap Watch
  31. Leader-with-lease • ConfigMap ·ͨ͸ Endpoints Λ༻͍ͯϩοΫ͢Δ • Ϧʔεظ͕ؒઃఆ͞Ε͓ͯΓ, Ϧʔμʔ͸ఆظతʹϦʔεͷߋ৽Λߦ͏ •

    ԿΒ͔ͷཧ༝ͰϦʔε͕ظݶ੾Εʹͳͬͨࡍʹ͸ ଴ػ͍ͯͨ͠ Controller ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ
  32. Leader-with-lease

  33. Leader-with-lease

  34. Leader-with-lease

  35. API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap

    Lock Watch Update lease record Check lease record
  36. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch Update lease record observedTime: 2020-01-28T05:33:18Z holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  37. observedTime: 2020-01-28T05:33:18Z Check lease record API Server Controller Pod A

    Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record Controller Pod A ConfigMap Lock Update lease record Ϧʔμʔ͸ఆظతʹϦʔεΛߋ৽͢Δ (ߋ৽͢Δͱ renewTime ͕ॻ͖׵ΘΔ) holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  38. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch Update lease record PCTFSWFE5JNF5; Controller Pod A ConfigMap Lock Update lease record observedTime: 2020-01-28T05:33:18Z Controller ͸಺෦Ͱ observedTime ͱ͍͏஋Λ͓࣋ͬͯΓ ൑ఆʹ࢖༻͢Δ holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  39. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  40. Check lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch IPMEFS*EFOUJUZ1PE" MFBTF%VSBUJPO4FDPOET BDRVJSF5JNF5; SFOFX5JNF5; MFBEFS5SBOTJUJPOT Check lease record Controller Pod B ConfigMap Lock holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80 PCTFSWFE5JNF5; observedTime + leaseDurationSeconds < now ͳΒ ԿΒ͔ͷཧ༝ͰϦʔε͕ߋ৽͞Ε͍ͯͳ͍ͨΊ ࣗ෼͕ϦʔμʔʹͳΖ͏ͱߋ৽ΛࢼΈΔ
  41. Update lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch holderIdentity: Pod B leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  42. Update lease record API Server Controller Pod A Controller Pod

    B Leader-with-lease ConfigMap Lock Watch Check lease record holderIdentity: Pod B leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80
  43. ·ͱΊ • Kubernetes Ͱ͸ Object Λ෼ࢄϩοΫʹ࢖༻͢Δ͜ͱͰ Leader Election Λ࣮ݱ͍ͯ͠Δ •

    ࡉ͔͍ίʔυͷ࿩ͱ͔͸͜͜ʹॻ͖·ͨ͠ • https://d-kuro.github.io/posts/kubernetes-leader-election/
  44. Reference • Configuring leader election - Operator SDK • https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-leader-election.html

    • େମ஻ͬͨ͜ͱ͕ॻ͍ͯ͋Δ • Gopherize.me - A Gopher pic that's as unique as you • https://gopherize.me/ • Gopher ܅͕࡞ΕΔαΠτ • community/icons at master · kubernetes/community • https://github.com/kubernetes/community/tree/master/icons • Kubernetes ͷΞΠίϯηοτ
  45. Thank You!