Slide 1
Slide 1 text
OpenShift-SDNと
NetworkPolicy
Manabu Ori
@orimanabu
1
2019-05-28
Red Hat Tech Night 2019.05
Slide 2
Slide 2 text
CNIとは
2
● CNIとは
○ Kubernetes でコンテナのネットワークインタフェースを設定
するための仕様
● やること
○ Linux コンテナが作成された際のネットワーク接続性の確保
○ コンテナが 削除された際のリソース解放を行う
Slide 3
Slide 3 text
OpenShift-SDN
3
● OpenShiftのデフォルトのCNIプラグイン
● OpenShift-SDN以外に、一応サードパーティのCNIプラグインも
使えます
○ Flannel (host-gw)
○ VMware NSX-T
○ Nuage VSP
○ Kuryr
○ Juniper Contrail
○ Calico
○ ...
Slide 4
Slide 4 text
OpenShift-SDN
4
● 3つの動作モード
○ ovs-subnet
■ ぶっとおし
○ ovs-multitenant
■ 同じプロジェクト内は疎通できる、異なるプロジェクトの
Podとは通信できない
○ ovs-networkpolicy
■ NetworkPolicy v1
● Open vSwitch (OVS)を活用
● VXLANでオーバーレイ
Slide 5
Slide 5 text
パケットの流れ
5
br0
vxlan0 tun0
veth6f5cb1b4
kernel
iptables
eth3
eth0
httpd
4
1 2
node2
172.16.99/0/24
vxlan0
kernel
iptables
eth3
1 2
node1
server1
server1-
1-6qc4j
172.16.99.42
172.16.99.41 10.129.0.1
10.129.0.13
tun0
172.30.255.14
10.130.0.1
br0
eth0
httpd
client1-
1-zn9ff
10.130.0.17
14
Slide 6
Slide 6 text
VNID
6
[ori@ocp311-master1 RHTN]$ oc get netnamespaces
NAME NETID EGRESS IPS
default 0 []
kube-public 536622 []
kube-system 7695582 []
management-infra 14065074 []
openshift 2031527 []
openshift-console 2954107 []
openshift-infra 12640971 []
openshift-logging 13439836 []
openshift-node 3244486 []
openshift-sdn 14688704 []
openshift-web-console 7072175 []
proj1 4610606 []
proj2 10513584 []
0x465a2e
a06cb0
Slide 7
Slide 7 text
node1のflow entry
7
table=0, priority=400, ip,in_port=2,nw_src=10.130.0.1 actions=goto_table:30
table=0, priority=300, ct_state=-trk,ip actions=ct(table=0)
table=0, priority=300, ip,in_port=2,nw_src=10.130.0.0/23,nw_dst=10.128.0.0/14 actions=goto_table:25
table=0, priority=250, ip,in_port=2,nw_dst=224.0.0.0/4 actions=drop
table=0, priority=200, arp,in_port=1,arp_spa=10.128.0.0/14,arp_tpa=10.130.0.0/23 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
table=0, priority=200, ip,in_port=1,nw_src=10.128.0.0/14 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
table=0, priority=200, ip,in_port=1,nw_dst=10.128.0.0/14 actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
table=0, priority=200, arp,in_port=2,arp_spa=10.130.0.1,arp_tpa=10.128.0.0/14 actions=goto_table:30
table=0, priority=200, ip,in_port=2 actions=goto_table:30
table=0, priority=150, in_port=1 actions=drop
table=0, priority=150, in_port=2 actions=drop
table=0, priority=100, arp actions=goto_table:20
table=0, priority=100, ip actions=goto_table:20
table=0, priority=0 actions=drop
table=10, priority=100, tun_src=172.16.99.31 actions=goto_table:30
table=10, priority=100, tun_src=172.16.99.21 actions=goto_table:30
table=10, priority=100, tun_src=172.16.99.42 actions=goto_table:30
table=10, priority=0 actions=drop
table=20, priority=100, arp,in_port=14,arp_spa=10.130.0.17,arp_sha=00:00:0a:82:00:11/00:00:ff:ff:ff:ff actions=load:0x465a2e->NXM_NX_REG0[],goto_table:21
table=20, priority=100, arp,in_port=18,arp_spa=10.130.0.21,arp_sha=00:00:0a:82:00:15/00:00:ff:ff:ff:ff actions=load:0xa06cb0->NXM_NX_REG0[],goto_table:21
table=20, priority=100, ip,in_port=14,nw_src=10.130.0.17 actions=load:0x465a2e->NXM_NX_REG0[],goto_table:21
table=20, priority=100, ip,in_port=18,nw_src=10.130.0.21 actions=load:0xa06cb0->NXM_NX_REG0[],goto_table:21
table=20, priority=0 actions=drop
table=21, priority=200, ip,nw_dst=10.128.0.0/14 actions=ct(commit,table=30)
table=21, priority=0 actions=goto_table:30
table=25, priority=100, ip,nw_src=10.130.0.17 actions=load:0x465a2e->NXM_NX_REG0[],goto_table:30
table=25, priority=100, ip,nw_src=10.130.0.21 actions=load:0xa06cb0->NXM_NX_REG0[],goto_table:30
table=25, priority=0 actions=drop
table=30, priority=300, arp,arp_tpa=10.130.0.1 actions=output:2
table=30, priority=300, ip,nw_dst=10.130.0.1 actions=output:2
table=30, priority=300, ct_state=+rpl,ip,nw_dst=10.130.0.0/23 actions=ct(table=70,nat)
table=30, priority=200, arp,arp_tpa=10.130.0.0/23 actions=goto_table:40
table=30, priority=200, ip,nw_dst=10.130.0.0/23 actions=goto_table:70
table=30, priority=100, arp,arp_tpa=10.128.0.0/14 actions=goto_table:50
table=30, priority=100, ip,nw_dst=10.128.0.0/14 actions=goto_table:90
table=30, priority=100, ip,nw_dst=172.30.0.0/16 actions=goto_table:60
table=30, priority=50, ip,in_port=1,nw_dst=224.0.0.0/4 actions=goto_table:120
table=30, priority=25, ip,nw_dst=224.0.0.0/4 actions=goto_table:110
table=30, priority=0, ip actions=goto_table:100
table=30, priority=0, arp actions=drop
table=40, priority=100, arp,arp_tpa=10.130.0.17 actions=output:14
table=40, priority=100, arp,arp_tpa=10.130.0.21 actions=output:18
table=40, priority=0 actions=drop
table=50, priority=100, arp,arp_tpa=10.131.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:172.16.99.31->tun_dst,output:1
table=50, priority=100, arp,arp_tpa=10.128.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:172.16.99.21->tun_dst,output:1
table=50, priority=100, arp,arp_tpa=10.129.0.0/23 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:172.16.99.42->tun_dst,output:1
table=50, priority=0 actions=drop
table=60, priority=200 actions=output:2
table=60, priority=0 actions=drop
table=70, priority=100, ip,nw_dst=10.130.0.17 actions=load:0x465a2e->NXM_NX_REG1[],load:0xe->NXM_NX_REG2[],goto_table:80
table=70, priority=100, ip,nw_dst=10.130.0.21 actions=load:0xa06cb0->NXM_NX_REG1[],load:0x12->NXM_NX_REG2[],goto_table:80
table=70, priority=0 actions=drop
table=80, priority=300, ip,nw_src=10.130.0.1 actions=output:NXM_NX_REG2[]
table=80, priority=200, ct_state=+rpl,ip actions=output:NXM_NX_REG2[]
table=80, priority=50, reg1=0xa4fb4b actions=output:NXM_NX_REG2[]
table=80, priority=50, reg1=0xa7d717 actions=output:NXM_NX_REG2[]
table=80, priority=50, reg1=0x465a2e actions=output:NXM_NX_REG2[]
table=80, priority=50, reg1=0xa06cb0 actions=output:NXM_NX_REG2[]
table=80, priority=0 actions=drop
Slide 8
Slide 8 text
client1 → server1
on node1
8
● table=0, ip actions=goto_table:20
● table=20, ip,in_port=14,nw_src=10.130.0.17 actions=load:0x465a2e->NXM_NX_REG0[],goto_table:21
● table=21, ip,nw_dst=10.128.0.0/14 actions=ct(commit,table=30)
● table=30, ip,nw_dst=10.128.0.0/14 actions=goto_table:90
● table=90, ip,nw_dst=10.129.0.0/23
actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:172.16.99.42->tun_dst,output:1
Slide 9
Slide 9 text
client1 → server1
on node2
9
● table=0, ip,in_port=1,nw_src=10.128.0.0/14
actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
● table=10, tun_src=172.16.99.41 actions=goto_table:30
● table=30, ip,nw_dst=10.129.0.0/23 actions=goto_table:70
● table=70, ip,nw_dst=10.129.0.13
actions=load:0x465a2e->NXM_NX_REG1[],load:0x8->NXM_NX_REG2[],goto_table:80
● table=80, reg1=0x465a2e actions=output:NXM_NX_REG2[]
Slide 10
Slide 10 text
NetworkPolicy適用 (1)
10
● table=0, ip,in_port=1,nw_src=10.128.0.0/14
actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
● table=10, tun_src=172.16.99.41 actions=goto_table:30
● table=30, ip,nw_dst=10.129.0.0/23 actions=goto_table:70
● table=70, ip,nw_dst=10.129.0.13
actions=load:0x465a2e->NXM_NX_REG1[],load:0x8->NXM_NX_REG2[],goto_table:80
● table=80, actions=drop
kind: NetworkPolicy
apiVersion:
networking.k8s.io/v1
metadata:
name: deny-by-default
spec:
podSelector:
ingress: []
Flow rules for [client1 → server1] on node2
Slide 11
Slide 11 text
NetworkPolicy適用 (2)
11
● table=0, ip,in_port=1,nw_src=10.128.0.0/14
actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
● table=10, tun_src=172.16.99.41 actions=goto_table:30
● table=30, ip,nw_dst=10.129.0.0/23 actions=goto_table:70
● table=70, ip,nw_dst=10.129.0.13
actions=load:0x465a2e->NXM_NX_REG1[],load:0x8->NXM_NX_REG2[],goto_table:80
● table=80, reg0=0x465a2e,reg1=0x465a2e actions=output:NXM_NX_REG2[]
kind: NetworkPolicy
apiVersion:
networking.k8s.io/v1
metadata:
name:
allow-same-namespace
spec:
podSelector:
ingress:
- from:
- podSelector: {}
Flow rules for [client1 → server1] on node2
Slide 12
Slide 12 text
NetworkPolicy適用 (3)
12
● table=0, ip,in_port=1,nw_src=10.128.0.0/14
actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
● table=10, tun_src=172.16.99.41 actions=goto_table:30
● table=30, ip,nw_dst=10.129.0.0/23 actions=goto_table:70
● table=70, ip,nw_dst=10.129.0.13
actions=load:0x465a2e->NXM_NX_REG1[],load:0x8->NXM_NX_REG2[],goto_table:80
● table=80,
ip,reg0=0x465a2e,reg1=0x465a2e,nw_src=10.130.0.17,nw_dst=10.129.0.13
actions=output:NXM_NX_REG2[]
kind: NetworkPolicy
apiVersion:
networking.k8s.io/v1
metadata:
name:
allow-server1-from-client
1
spec:
podSelector:
matchLabels:
app: server1
ingress:
- from:
- podSelector:
matchLabels:
app: client1
Flow rules for [client1 → server1] on node2
Slide 13
Slide 13 text
NetworkPolicy適用 (4)
13
● table=0, ip,in_port=1,nw_src=10.128.0.0/14
actions=move:NXM_NX_TUN_ID[0..31]->NXM_NX_REG0[],goto_table:10
● table=10, tun_src=172.16.99.41 actions=goto_table:30
● table=30, ip,nw_dst=10.129.0.0/23 actions=goto_table:70
● table=70, ip,nw_dst=10.129.0.13
actions=load:0x465a2e->NXM_NX_REG1[],load:0x8->NXM_NX_REG2[],goto_table:80
● table=80, ip,reg0=0x465a2e,reg1=0xa06cb0,nw_dst=10.129.0.14
actions=output:NXM_NX_REG2[]
kind: NetworkPolicy
apiVersion:
networking.k8s.io/v1
metadata:
name:
allow-from-proj1-client1
spec:
podSelector:
matchLabels:
app: server2
ingress:
- from:
- namespaceSelector:
matchLabels:
name: proj1
Flow rules for [client1 → server1] on node2
Slide 14
Slide 14 text
OpenShift-SDNにおける
NetworkPolicyの注意点
14
● NetworkPolicyオブジェクトに紐付かないPodは全通し
● OpenShift-SDNでサポートするNetworkPolicyはv1 APIのみ
○ Ingressのみサポート
○ 以下は使えない
■ Egress
■ IPBlock
■ namespaceSelectorとpodSelectorの両方指定
● podSelectorでPod個別に指定すると、マッチするPod(のIPアド
レス)ごとにFlow entryが増える
○ なるべくnamespaceSelector、もしくは空のpodSelectorで
指定する
○ 細かいPod間の制御は最低限にする
Slide 15
Slide 15 text
CONFIDENTIAL Designator
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat
Red Hat is the world’s leading provider of enterprise
open source software solutions. Award-winning
support, training, and consulting services make
Red Hat a trusted adviser to the Fortune 500.
Thank you
15