Tweet
Tweet

Slide 1

Slide 1 text

Kubernetes Network security

Slide 2

Slide 2 text

@eloycoto - Software engineer at Red Hat - Working now in 3Scale API Management - Former Isovalent software engineer - Working full time on Cilium - Former VoIP software dev at Foehn Coms.

Slide 3

Slide 3 text

Kubernetes services Payment API GW Identity Provider Backend Billing

Slide 4

Slide 4 text

Back to Kubernetes Network

Slide 5

Slide 5 text

Kubernetes architecture KubeProxy Scheduler APIServer CoreDNS Kubelet Kubelet Kubelet Kubelet KubeProxy KubeProxy KubeProxy K8S1 Master K8SN K8SN

Slide 6

Slide 6 text

APIServer Container Network interface Kubelet Kubectl apply New Container IP: X.X.X.X Kubernetes CNI

Slide 7

Slide 7 text

Kubernetes architecture Kubelet KubeProxy Scheduler CNI DS APIServer Kubelet KubeProxy Kubelet KubeProxy Kubelet KubeProxy K8S1 Master K8SN K8SN CNI DS CNI DS CNI DS

Slide 8

Slide 8 text

Network communication: Direct routing K8S2 10.10.2.10 DR POD: BA 10.10.2.11 POD: BB 10.10.2.12 POD: BC 10.10.2.13 K8S1 10.10.1.10 DR POD: BA 10.10.1.11 POD: BB 10.10.1.12 POD: BC 10.10.1.13

Slide 9

Slide 9 text

Network communication: Encapsulation K8S2 192.168.1.12 DR POD: BA 10.10.2.11 POD: BB 10.10.2.12 POD: BC 10.10.2.13 K8S1 192.168.1.11 DR POD: BA 10.10.1.11 POD: BB 10.10.1.12 POD: BC 10.10.1.13 VXLAN/GENEVE

Slide 10

Slide 10 text

Tell me about what CNI I should use!

Slide 11

Slide 11 text

- Native BPF/XDP from start - Overlay (VXLan or Geneve) or direct routing - Identity based network policies - L3/L4/L7 Policies - Egress DNS policies - Use a lot of kernel features (Some kernel maintainers working on the project) - Socket acceleration for sidecars proxies. - A lot of traction in the community. - IPSEC communication between endpoints - Multi cluster communication Cilium

Slide 12

Slide 12 text

- The main feature is BGP peering connections - Also provides a way to use VXLan - New version will have BPF and socket acceleration support. - L3/L4 policies based on IPtables - L7 policies using Istio - Network policies based on SPIFFE - Integration with Openstack/Mesos Calico

Slide 13

Slide 13 text

Weave - VXLan encapsulation - L3/L4 policies based on IPtables - Use Kernel fast datapath features - IPSec encryption between pods.

Slide 14

Slide 14 text

BPF hooks Pod TCP Network Veth Veth Iptables Normal connect flow Connect() LLVM Bytecode + verifier + JIT Source code (c) Pod Connect() BPF LSM Hook EACCESS BPF connect() hook

Slide 15

Slide 15 text

FAST Datapath Pod Router OVS Pod Router OVS User Space Kernel Space K8S 1 K8S 2 Network

Slide 16

Slide 16 text

Ethernet Ethernet Socket TCP/IP Ethernet Loopback Ethernet Socket TCP/IP IPtables Ethernet POD Loopback Socket acceleration Socket TCP/IP Ethernet Socket TCP/IP IPtables Ethernet ETH0 Socket TCP/IP IPtables Ethernet ETH0 Socket TCP/IP IPtables Ethernet POD

Slide 17

Slide 17 text

Socket acceleration Socket Socket TCP/IP IPtables Ethernet ETH0 Socket TCP/IP IPtables Ethernet ETH0 Socket Loopback Socket POD Socket POD Cilium Cilium

Slide 18

Slide 18 text

Network policies

Slide 19

Slide 19 text

$ iptables -A INPUT -p tcp \ -d 10.10.1.10/32 --dport 22 \ -s 10.10.2.10/32 \ -j ACCEPT

Slide 20

Slide 20 text

Right now we have 8576 entries. Before removing them the first time we had >30k entries. While observing the iptables we learned that every 1-2 seconds a new rule is added.

Slide 21

Slide 21 text

Kubernetes network policy apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: test-network-policy namespace: default spec: podSelector: matchLabels: role: db policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: role: backend ports: - protocol: TCP port: 6379

Slide 22

Slide 22 text

Network policy spec EGRESS ports (port, protocol) to ipBlock namespaceSelector podSelector INGRESS ports (port, protocol) from ipBlock namespaceSelector podSelector POD podSelector

Slide 23

Slide 23 text

Kubernetes Network policies Backend (Role: backend) Frontend (Role: frontend) Auth (Role: auth) Identity (Role: identity) DB (role: db) Egress

Slide 24

Slide 24 text

Namespace isolation NS: billing NS: monitoring NS: backend Service Service Service Pod Pod Pod Pod Pod Pod Pod Pod Pod Compromised

Slide 25

Slide 25 text

Namespace isolation kind: NetworkPolicy spec: namespaceSelector: matchLabels: project: myproject policyTypes: - Ingress - Egress ingress: - from: - namespaceSelector: matchLabels: project: myproject egress: - to: - namespaceSelector: matchLabels: project: myproject

Slide 26

Slide 26 text

Namespace isolation NS: billing NS: monitoring NS: backend Service Service Service Pod Pod Pod Pod Pod Pod Pod Pod Pod Compromised

Slide 27

Slide 27 text

And that, my friends, is what Kubernetes offers!

Slide 28

Slide 28 text

Cilium L7 policies to rescue! apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy Metadata: {name: "rule1"} spec: endpointSelector: matchLabels: {app: "service"} ingress: - fromEndpoints: - matchLabels: env: prod toPorts: - ports: - port: "80" Protocol: tcp rules: http: - method: "GET" path: "/public"

Slide 29

Slide 29 text

Cilium L7 policies traffic flow! TCP: L3/L4 Client pod Server pod TCP: L7 GET /private GET /foo GET /public

Slide 30

Slide 30 text

Cilium L7 policies toPorts: - ports: - port: "9092" protocol: TCP rules: kafka: - role: "consume" topic: "empire-announce" Kafka Protocol toPorts: - ports: - port: "9042" protocol: TCP rules: l7proto: cassandra l7: - query_action: "select" query_table: "system\\..*" Cassandra Protocol Memcached Protocol AWS metadata

Slide 31

Slide 31 text

Cilium Egress DNS policies egress: - toPorts: - ports: { port: 53, protocol: ANY} rules: dns: - matchName: "cilium.io" - matchPattern: "*.cilium.io" - matchPattern: "*.api.cilium.io" - toFQDNs: - matchName: "cilium.io" - matchName: "service1.api.cilium.io" - matchPattern: "special*service.api.cilium.io" toPorts: - ports: - port: "80" protocol: TCP

Slide 32

Slide 32 text

DNS www.a.com Cilium DNS egress rule! Client pod www.a.com 1.2.3.4 1.2.3.4 Allowed for pod client TTL 60

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Istio MTLS traffic apiVersion: "authentication.istio.io/v1alpha1" kind: "Policy" metadata: name: "default" namespace: "ns1" spec: peers: - mtls: {}

Slide 36

Slide 36 text

Istio Origin authentication origins: - jwt: issuer: "https://accounts.google.com" jwksUri: "https://www.googleapis.com/oauth2/v3/certs" trigger_rules: - excluded_paths: - exact: /health

Slide 37

Slide 37 text

Future reading Cilium BPF and XDP: Thomas Graf slides https://www.slideshare.net/ThomasGraf5 Cilium blog: https://cilium.io/blog Istio: https://istio.io/docs/reference/config/istio.authentication.v1alpha1/ SPIFFE: https://spiffe.io/ Community days recording: https://www.youtube.com/watch?v=H5IlmYmEDKk

Slide 38

Slide 38 text

Graciñas!! @eloycoto

Slide 39

Slide 39 text

References Thomas Graf slides Weave Net slides Calico Cover photo