賢く「散らす」ための Topology Spread Constraints #k8sjp / Kubernetes Meetup Tokyo 25th

by y_taka_23

Slide 1

Slide 1 text

賢く「散らす」ための Topology Spread Constraints チェシャ猫 (@y_taka_23) Kubernetes Meetup Tokyo #25 (2019/11/13) #k8sjp

Slide 2

Slide 2 text

Scheduling = Pod の配置戦略 #k8sjp

Slide 3

Slide 3 text

複数 Node への Pod の配置 Node Pod Pod Node Pod Pod Node Node #k8sjp

Slide 4

Slide 4 text

複数 Node への Pod の配置 Availability Zone A Node Pod Pod Node Pod Pod Availability Zone B Node Node #k8sjp

Slide 5

Slide 5 text

Kubernetes の Topology ● Node を論理的にグルーピング ○ Label を付与することで表現 ○ 運用上は「まとめて故障する単位」に対応 ● Well-Known Labels も利用可能 ○ Cloud Provider が自動で付与 ○ failure-domain.beta.kubernetes.io/region ○ failure-domain.beta.kubernetes.io/zone #k8sjp

Slide 6

Slide 6 text

Zone 間の Pod 分散 Availability Zone A Node [zone=zoneA, node=node1] Pod Pod Node [zone=zoneA, node=node2] Availability Zone B Node [zone=zoneB, node=node3] Pod Pod Node [zone=zoneB, node=node4] #k8sjp

Slide 7

Slide 7 text

Node 間の Pod 分散 Availability Zone A Node [zone=zoneA, node=node1] Pod Node [zone=zoneA, node=node2] Availability Zone B Node [zone=zoneB, node=node3] Pod Node [zone=zoneB, node=node4] Pod Pod #k8sjp

Slide 8

Slide 8 text

既存の「散らす」機能 ● DaemonSet ○ 全 Node に 1 Pod ずつ配置されてしまう ○ 大きなクラスタでは無駄が多すぎる ● Pod Anti Affinity ○ 自分自身と反発するような設定を記述 ○ 各 Topology にちょうど 1 Pod しか配置できない ○ ローリングアップデートできない #k8sjp

Slide 9

Slide 9 text

Topology Spread Constraints #k8sjp

Slide 10

Slide 10 text

Topology Spread Constraints ● v1.16 で α 版機能として導入 ○ 従来よりも「柔軟な単位」で分散が可能に ○ Feature Gate: EvenPodsSpread=true ● Topology 間で Pod を分散させる ○ 許容できる Pod 個数の差を指定 ○ カウント対象となる Pod は Label で指定 ○ 強制か、あくまでも優先に留めるかは選択可能 #k8sjp

Slide 11

Slide 11 text

Topology Spread Constraints kind: Pod apiVersion: v1 metadata: name: mypod labels: foo: bar spec: constiners: ... topologySpreadConstraints: - topologyKey: zone maxSkew: 1 labelSelector: matchLabels: foo: bar whenUnsatisfiable: DoNotSchedule #k8sjp

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Zone に対して maxSkew: 1 Availability Zone A (2 Pods) Node [zone=zoneA, node=node1] Pod [foo=bar] Node [zone=zoneA, node=node2] Pod [foo=bar] Availability Zone B (1 Pods) Node [zone=zoneB, node=node3] Pod [foo=bar] Node [zone=zoneB, node=node4] #k8sjp

Slide 17

Slide 17 text

Zone に対して maxSkew: 1 Availability Zone A (2 Pods) Node [zone=zoneA, node=node1] Pod [foo=bar] Node [zone=zoneA, node=node2] Pod [foo=bar] Availability Zone B (2 Pods) Node [zone=zoneB, node=node3] Pod [foo=bar] Node [zone=zoneB, node=node4] Pod [foo=bar] #k8sjp

Slide 18

Slide 18 text

Slide 19

Slide 19 text

複数の Constraints を指定 spec: topologySpreadConstraints: - topologyKey: zone maxSkew: 1 labelSelector: matchLabels: foo: bar whenUnsatisfiable: DoNotSchedule - topologyKey: node maxSkew: 1 labelSelector: matchLabels: foo: bar whenUnsatisfiable: DoNotSchedule #k8sjp

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Zone & Node で maxSkew: 1 Availability Zone A (3 Pods) Node [zone=zoneA, node=node1] Pod [foo=bar] Pod [foo=bar] Node [zone=zoneA, node=node2] Pod [foo=bar] Availability Zone B (2 Pods) Node [zone=zoneB, node=node3] Pod [foo=bar] Pod [foo=bar] #k8sjp

Slide 23

Slide 23 text

Zone & Node で maxSkew: 1 Availability Zone A (4 Pods) Node [zone=zoneA, node=node1] Pod [foo=bar] Pod [foo=bar] Node [zone=zoneA, node=node2] Pod [foo=bar] Availability Zone B (2 Pods) Node [zone=zoneB, node=node3] Pod [foo=bar] Pod [foo=bar] Pod [foo=bar] #k8sjp

Slide 24

Slide 24 text

Zone & Node で maxSkew: 1 Availability Zone A (3 Pods) Node [zone=zoneA, node=node1] Pod [foo=bar] Pod [foo=bar] Node [zone=zoneA, node=node2] Pod [foo=bar] Availability Zone B (3 Pods) Node [zone=zoneB, node=node3] Pod [foo=bar] Pod [foo=bar] Pod [foo=bar] #k8sjp

Slide 25

Slide 25 text

配置できる Node がない！ #k8sjp

Slide 26

Slide 26 text

矛盾する Constraints の解消 spec: topologySpreadConstraints: - topologyKey: zone maxSkew: 1 labelSelector: matchLabels: foo: bar whenUnsatisfiable: ScheduleAnyway - topologyKey: node maxSkew: 1 labelSelector: matchLabels: foo: bar whenUnsatisfiable: DoNotSchedule #k8sjp

Slide 27

Slide 27 text

Slide 28

Slide 28 text

矛盾する Constraints の解消 spec: topologySpreadConstraints: - topologyKey: zone maxSkew: 2 labelSelector: matchLabels: foo: bar whenUnsatisfiable: DoNotSchedule - topologyKey: node maxSkew: 1 labelSelector: matchLabels: foo: bar whenUnsatisfiable: DoNotSchedule 許容できる差の範囲を緩める #k8sjp

Slide 29

Slide 29 text

Zone & Node で maxSkew: 2 Availability Zone A (4 Pods) Node [zone=zoneA, node=node1] Pod [foo=bar] Pod [foo=bar] Node [zone=zoneA, node=node2] Pod [foo=bar] Availability Zone B (2 Pods) Node [zone=zoneB, node=node3] Pod [foo=bar] Pod [foo=bar] Pod [foo=bar] #k8sjp

Slide 30

Slide 30 text

Zone & Node で maxSkew: 2 Availability Zone A (4 Pods) Node [zone=zoneA, node=node1] Pod [foo=bar] Pod [foo=bar] Node [zone=zoneA, node=node2] Pod [foo=bar] Availability Zone B (3 Pods) Node [zone=zoneB, node=node3] Pod [foo=bar] Pod [foo=bar] Pod [foo=bar] Pod [foo=bar] #k8sjp

Slide 31

Slide 31 text

まとめ ● Kubernetes の Topology ○ Node を論理的にグルーピング ● Topology Spread Constraint ○ Topology に対して「散らす」設定が可能に ○ Topology ごと障害が起きた際の可用性が向上 ● 運用上の注意点 ○ DoNotScheduler はうっかりデッドロックを招く #k8sjp

Slide 32

Slide 32 text

【PR】Scheduling 徹底解説 https://speakerdeck.com/ytaka23/cloudnative-days-tokyo-2019 #k8sjp

Slide 33

Slide 33 text

More Spreading, More Reliable! Presented by チェシャ猫 (@y_taka_23) #k8sjp