Slide 1

Slide 1 text

20/01/29 Kubernetes Meetup Tokyo #27 @ponde_m Leader Election in Kubernetes

Slide 2

Slide 2 text

I work at @ponde_m @d-kuro

Slide 3

Slide 3 text

Leader Election ͱ͸ • “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩ ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕ ·Ε·͢ɻ” https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/

Slide 4

Slide 4 text

• “Ϧʔμʔબڍ͸ɺ෼ࢄγεςϜ಺ͷ 1 ͭ (ϓϩηεɺϗε τɺεϨουɺΦϒδΣΫτɺ·ͨ͸ਓؒ) ʹಛผͳݖݶΛ༩ ͑Δͱ͍͏γϯϓϧͳΞΠσΞͰ͢ɻ͜ΕΒͷಛผͳݖݶʹ ͸ɺ࡞ۀΛׂΓ౰ͯΔػೳɺσʔλͷҰ෦Λมߋ͢Δػೳɺ ·ͨ͸γεςϜ಺ͷ͢΂ͯͷϦΫΤετΛॲཧ͢Δ੹೚ؚ͕ ·Ε·͢ɻ” https://aws.amazon.com/jp/builders-library/leader-election-in-distributed-systems/ Leader Election ͱ͸

Slide 5

Slide 5 text

Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment Controller Watch Store kubectl

Slide 6

Slide 6 text

Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3

Slide 7

Slide 7 text

Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 API Server Controller Manager Deployment Controller Watch Create ReplicaSet replica: 3

Slide 8

Slide 8 text

Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 -> 2 Update

Slide 9

Slide 9 text

Kubernetes Architecture API Server etcd Create Deployment Controller Manager Deployment Controller Watch Store kubectl API Server Create Deployment kubectl replica: 3 -> 2 Update API Server Controller Manager Deployment Controller Watch Update ReplicaSet replica: 3 -> 2

Slide 10

Slide 10 text

Reconciliation Loop Provided in Managing Kubernetes: Operating Kubernetes Clusters in the Real World, 23. https://www.oreilly.com/library/view/managing-kubernetes/9781492033905/

Slide 11

Slide 11 text

API Server Create Deployment Controller Manager Deployment Controller Watch Controller Manager Deployment Controller kubectl 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

Slide 12

Slide 12 text

API Server Create Deployment Controller Manager Deployment Controller Watch Controller Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

Slide 13

Slide 13 text

API Server Create Deployment Controller Manager Deployment Controller Watch Controller Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

Slide 14

Slide 14 text

API Server Create Deployment Controller Manager Deployment Controller Watch Controller Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 API Server Controller Manager Deployment Controller Create ReplicaSet replica: 3 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

Slide 15

Slide 15 text

API Server Create Deployment Controller Manager Deployment Controller Watch Controller Manager Deployment Controller kubectl API Server Create Deployment kubectl replica: 3 Controller Manager Deployment Controller Watch API Server Create ReplicaSet replica: 3 API Server Controller Manager Deployment Controller Create ReplicaSet replica: 3 Error from server (AlreadyExists): deployments.apps “foo" already exists 2 ͭͷ Controller ͕ಈ͍͍ͯΔ৔߹

Slide 16

Slide 16 text

API Server Controller Manager Deployment Controller Watch Controller Manager Deployment Controller Leader Election in Kubernetes -FBEFS

Slide 17

Slide 17 text

API Server Controller Manager Deployment Controller Watch Controller Manager Deployment Controller Leader Election in Kubernetes -FBEFS

Slide 18

Slide 18 text

API Server Controller Manager Deployment Controller Watch Controller Manager Deployment Controller Leader Election in Kubernetes -FBEFS

Slide 19

Slide 19 text

Leader Election in Kubernetes • Kubernetes ͷ Leader Election ͸ ڝ߹Λආ͚ͭͭߴՄ༻ੑΛอͭͨΊͷ ΢ΥʔϜελϯόΠͷ࢓૊Έͱͯ͠࢖ΘΕ͍ͯΔ • ϥΠϒϥϦͱͯ͠ఏڙ͞Ε͍ͯΔͷͰ ࠓճઆ໌ͨ͠ Controller ͷྫҎ֎Ͱ΋ Kubernetes ্Ͱಈ͘ΞϓϦέʔγϣϯͰ Leader Election ͍ͨ͠ͱ͍͏৔߹ʹ΋࢖༻Ͱ͖Δ

Slide 20

Slide 20 text

Leader Election ͷ࣮૷ • Leader-for-life • Operator SDK ͕ఏڙ͍ͯ͠Δ࣮૷ • github.com/operator-framework/operator-sdk/pkg/leader • Leader-with-lease • controller-runtime(client-go) ͕ఏڙ͍ͯ͠Δ࣮૷ • github.com/kubernetes-sigs/controller-runtime/pkg/leaderelection

Slide 21

Slide 21 text

Leader Election ͷ࣮૷ • Leader-for-life • NotReady Node ʹ Pod ͕͋ΔͱϦʔμʔͷબग़͕஗͘ͳΔ (pod-eviction-timeout, default: 5m ʹґଘ͢Δ) • Split Brain ͷϦεΫͳ͠ • Leader-with-lease • Split Brain ͷϦεΫ͕͋Δ • NotReady Node ʹ Pod ͕͋ͬͯ΋Ϧʔμʔͷ੾Γସ͕͑ૣ͍

Slide 22

Slide 22 text

Leader-for-life • OwnerReference ͕Ϧʔμʔͷ Pod Ͱ͋Δ ConfigMap Λ࡞੒͠, ϩοΫ͢Δ • Pod ͕࡟আ͞ΕΔͱ Kubernetes ͷΨϕʔδίϨΫγϣϯͷ ࢓૊ΈʹΑΓ ConfigMap ΋ࣗಈతʹ࡟আ͞ΕΔͨΊ, ଞͷ Pod ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ

Slide 23

Slide 23 text

Leader-for-life

Slide 24

Slide 24 text

Leader-for-life

Slide 25

Slide 25 text

API Server Controller Pod A Watch Controller Pod B Leader-for-life ConfigMap Lock Create ConfigMap PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE"

Slide 26

Slide 26 text

API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap Lock Create ConfigMap PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE"

Slide 27

Slide 27 text

API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap Lock Create ConfigMap PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE"

Slide 28

Slide 28 text

API Server Controller Pod A Controller Pod B Leader-for-life

Slide 29

Slide 29 text

API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap Lock PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE# Create ConfigMap Watch

Slide 30

Slide 30 text

API Server Controller Pod A Controller Pod B Leader-for-life ConfigMap Lock PXOFS3FGFSFODFT BQJ7FSTJPOW LJOE1PE OBNF$POUSPMMFS1PE# Create ConfigMap Watch

Slide 31

Slide 31 text

Leader-with-lease • ConfigMap ·ͨ͸ Endpoints Λ༻͍ͯϩοΫ͢Δ • Ϧʔεظ͕ؒઃఆ͞Ε͓ͯΓ, Ϧʔμʔ͸ఆظతʹϦʔεͷߋ৽Λߦ͏ • ԿΒ͔ͷཧ༝ͰϦʔε͕ظݶ੾Εʹͳͬͨࡍʹ͸ ଴ػ͍ͯͨ͠ Controller ͕ϦʔμʔΛ֫ಘ͢Δ͜ͱ͕Ͱ͖Δ

Slide 32

Slide 32 text

Leader-with-lease

Slide 33

Slide 33 text

Leader-with-lease

Slide 34

Slide 34 text

Leader-with-lease

Slide 35

Slide 35 text

API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record Check lease record

Slide 36

Slide 36 text

Check lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record observedTime: 2020-01-28T05:33:18Z holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

Slide 37

Slide 37 text

observedTime: 2020-01-28T05:33:18Z Check lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record Controller Pod A ConfigMap Lock Update lease record Ϧʔμʔ͸ఆظతʹϦʔεΛߋ৽͢Δ (ߋ৽͢Δͱ renewTime ͕ॻ͖׵ΘΔ) holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

Slide 38

Slide 38 text

Check lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch Update lease record PCTFSWFE5JNF5; Controller Pod A ConfigMap Lock Update lease record observedTime: 2020-01-28T05:33:18Z Controller ͸಺෦Ͱ observedTime ͱ͍͏஋Λ͓࣋ͬͯΓ ൑ఆʹ࢖༻͢Δ holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

Slide 39

Slide 39 text

Check lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

Slide 40

Slide 40 text

Check lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch IPMEFS*EFOUJUZ1PE" MFBTF%VSBUJPO4FDPOET BDRVJSF5JNF5; SFOFX5JNF5; MFBEFS5SBOTJUJPOT Check lease record Controller Pod B ConfigMap Lock holderIdentity: Pod A leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80 PCTFSWFE5JNF5; observedTime + leaseDurationSeconds < now ͳΒ ԿΒ͔ͷཧ༝ͰϦʔε͕ߋ৽͞Ε͍ͯͳ͍ͨΊ ࣗ෼͕ϦʔμʔʹͳΖ͏ͱߋ৽ΛࢼΈΔ

Slide 41

Slide 41 text

Update lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch holderIdentity: Pod B leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

Slide 42

Slide 42 text

Update lease record API Server Controller Pod A Controller Pod B Leader-with-lease ConfigMap Lock Watch Check lease record holderIdentity: Pod B leaseDurationSeconds: 60 acquireTime: 2020-01-27T03:44:58Z renewTime: 2020-01-28T05:33:18Z leaderTransitions: 80

Slide 43

Slide 43 text

·ͱΊ • Kubernetes Ͱ͸ Object Λ෼ࢄϩοΫʹ࢖༻͢Δ͜ͱͰ Leader Election Λ࣮ݱ͍ͯ͠Δ • ࡉ͔͍ίʔυͷ࿩ͱ͔͸͜͜ʹॻ͖·ͨ͠ • https://d-kuro.github.io/posts/kubernetes-leader-election/

Slide 44

Slide 44 text

Reference • Configuring leader election - Operator SDK • https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-leader-election.html • େମ஻ͬͨ͜ͱ͕ॻ͍ͯ͋Δ • Gopherize.me - A Gopher pic that's as unique as you • https://gopherize.me/ • Gopher ܅͕࡞ΕΔαΠτ • community/icons at master · kubernetes/community • https://github.com/kubernetes/community/tree/master/icons • Kubernetes ͷΞΠίϯηοτ

Slide 45

Slide 45 text

Thank You!