Kubernetes Network Policies

1. Kubernetes Network security
2. @eloycoto - Software engineer at Red Hat - Working now in 3Scale API Management - Former Isovalent software engineer - Working full time on Cilium - Former VoIP software dev at Foehn Coms.
3. Kubernetes services Payment API GW Identity Provider Backend Billing
4. Back to Kubernetes Network
5. Kubernetes architecture KubeProxy Scheduler APIServer CoreDNS Kubelet Kubelet Kubelet Kubelet KubeProxy KubeProxy KubeProxy K8S1 Master K8SN K8SN
6. APIServer Container Network interface Kubelet Kubectl apply New Container IP: X.X.X.X Kubernetes CNI
7. Kubernetes architecture Kubelet KubeProxy Scheduler CNI DS APIServer Kubelet KubeProxy Kubelet KubeProxy Kubelet KubeProxy K8S1 Master K8SN K8SN CNI DS CNI DS CNI DS
8. Network communication: Direct routing K8S2 10.10.2.10 DR POD: BA 10.10.2.11 POD: BB 10.10.2.12 POD: BC 10.10.2.13 K8S1 10.10.1.10 DR POD: BA 10.10.1.11 POD: BB 10.10.1.12 POD: BC 10.10.1.13
9. Network communication: Encapsulation K8S2 192.168.1.12 DR POD: BA 10.10.2.11 POD: BB 10.10.2.12 POD: BC 10.10.2.13 K8S1 192.168.1.11 DR POD: BA 10.10.1.11 POD: BB 10.10.1.12 POD: BC 10.10.1.13 VXLAN/GENEVE
10. Tell me about what CNI I should use!
11. - Native BPF/XDP from start - Overlay (VXLan or Geneve) or direct routing - Identity based network policies - L3/L4/L7 Policies - Egress DNS policies - Use a lot of kernel features (Some kernel maintainers working on the project) - Socket acceleration for sidecars proxies. - A lot of traction in the community. - IPSEC communication between endpoints - Multi cluster communication Cilium
12. - The main feature is BGP peering connections - Also provides a way to use VXLan - New version will have BPF and socket acceleration support. - L3/L4 policies based on IPtables - L7 policies using Istio - Network policies based on SPIFFE - Integration with Openstack/Mesos Calico
13. Weave - VXLan encapsulation - L3/L4 policies based on IPtables - Use Kernel fast datapath features - IPSec encryption between pods.
14. BPF hooks Pod TCP Network Veth Veth Iptables Normal connect flow Connect() LLVM Bytecode + verifier + JIT Source code (c) Pod Connect() BPF LSM Hook EACCESS BPF connect() hook
15. FAST Datapath Pod Router OVS Pod Router OVS User Space Kernel Space K8S 1 K8S 2 Network
16. Ethernet Ethernet Socket TCP/IP Ethernet Loopback Ethernet Socket TCP/IP IPtables Ethernet POD Loopback Socket acceleration Socket TCP/IP Ethernet Socket TCP/IP IPtables Ethernet ETH0 Socket TCP/IP IPtables Ethernet ETH0 Socket TCP/IP IPtables Ethernet POD
17. Socket acceleration Socket Socket TCP/IP IPtables Ethernet ETH0 Socket TCP/IP IPtables Ethernet ETH0 Socket Loopback Socket POD Socket POD Cilium Cilium
18. Network policies
19. $ iptables -A INPUT -p tcp \ -d 10.10.1.10/32 --dport 22 \ -s 10.10.2.10/32 \ -j ACCEPT
20. Right now we have 8576 entries. Before removing them the first time we had >30k entries. While observing the iptables we learned that every 1-2 seconds a new rule is added.
21. Kubernetes network policy apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: test-network-policy namespace: default spec: podSelector: matchLabels: role: db policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: role: backend ports: - protocol: TCP port: 6379
22. Network policy spec EGRESS ports (port, protocol) to ipBlock namespaceSelector podSelector INGRESS ports (port, protocol) from ipBlock namespaceSelector podSelector POD podSelector
23. Kubernetes Network policies Backend (Role: backend) Frontend (Role: frontend) Auth (Role: auth) Identity (Role: identity) DB (role: db) Egress
24. Namespace isolation NS: billing NS: monitoring NS: backend Service Service Service Pod Pod Pod Pod Pod Pod Pod Pod Pod Compromised
25. Namespace isolation kind: NetworkPolicy spec: namespaceSelector: matchLabels: project: myproject policyTypes: - Ingress - Egress ingress: - from: - namespaceSelector: matchLabels: project: myproject egress: - to: - namespaceSelector: matchLabels: project: myproject
26. Namespace isolation NS: billing NS: monitoring NS: backend Service Service Service Pod Pod Pod Pod Pod Pod Pod Pod Pod Compromised
27. And that, my friends, is what Kubernetes offers!
28. Cilium L7 policies to rescue! apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy Metadata: {name: "rule1"} spec: endpointSelector: matchLabels: {app: "service"} ingress: - fromEndpoints: - matchLabels: env: prod toPorts: - ports: - port: "80" Protocol: tcp rules: http: - method: "GET" path: "/public"
29. Cilium L7 policies traffic flow! TCP: L3/L4 Client pod Server pod TCP: L7 GET /private GET /foo GET /public
30. Cilium L7 policies toPorts: - ports: - port: "9092" protocol: TCP rules: kafka: - role: "consume" topic: "empire-announce" Kafka Protocol toPorts: - ports: - port: "9042" protocol: TCP rules: l7proto: cassandra l7: - query_action: "select" query_table: "system\\..*" Cassandra Protocol Memcached Protocol AWS metadata
31. Cilium Egress DNS policies egress: - toPorts: - ports: { port: 53, protocol: ANY} rules: dns: - matchName: "cilium.io" - matchPattern: "*.cilium.io" - matchPattern: "*.api.cilium.io" - toFQDNs: - matchName: "cilium.io" - matchName: "service1.api.cilium.io" - matchPattern: "special*service.api.cilium.io" toPorts: - ports: - port: "80" protocol: TCP
32. DNS www.a.com Cilium DNS egress rule! Client pod www.a.com 1.2.3.4 1.2.3.4 Allowed for pod client TTL 60
35. Istio MTLS traffic apiVersion: "authentication.istio.io/v1alpha1" kind: "Policy" metadata: name: "default" namespace: "ns1" spec: peers: - mtls: {}
36. Istio Origin authentication origins: - jwt: issuer: "https://accounts.google.com" jwksUri: "https://www.googleapis.com/oauth2/v3/certs" trigger_rules: - excluded_paths: - exact: /health
37. Future reading Cilium BPF and XDP: Thomas Graf slides https://www.slideshare.net/ThomasGraf5 Cilium blog: https://cilium.io/blog Istio: https://istio.io/docs/reference/config/istio.authentication.v1alpha1/ SPIFFE: https://spiffe.io/ Community days recording: https://www.youtube.com/watch?v=H5IlmYmEDKk
38. Graciñas!! @eloycoto
39. References Thomas Graf slides Weave Net slides Calico Cover photo