[KubeCon EU 2026, Isovalent Booth Session] Who Owns This IP? Debugging L2 in Kubernetes

Transcript

1. Who Owns This IP? Debugging L2 in Kubernetes Donia Chaiehloudj, Software Engineer & Community
2. © 2025 Isovalent. All Rights Reserved. 2 The Situation You just deployed your first bare-metal Kubernetes cluster • Here a kind cluster • You deploy an app, create a LoadBalancer service • Kubernetes assigns an external IPv6 address • You try to curl it from outside the cluster…
3. © 2025 Isovalent. All Rights Reserved. 3 Let's Debug
4. © 2025 Isovalent. All Rights Reserved. 4 Let's Debug First instinct: is the app even running? • kubectl get pods → all Running ✓ • curl from inside a pod → works ✓ • curl from a cluster node → works ✓ • The problem is outside the cluster
5. © 2025 Isovalent. All Rights Reserved. 5 Is It a Routing Problem? Maybe the IP isn't routed correctly? • ip addr on the nodes: the external IP doesn't exist anywhere • Ping from the client: no response, not even "unreachable" • tcpdump on the client: packets go out, nothing comes back • The IP is a ghost: it lives in etcd, nowhere else
6. © 2025 Isovalent. All Rights Reserved. 6 Who on this network knows that IP exists? • Every IP needs an owner: a MAC address • Before sending a packet, your client asks: "Who has this IP?" a. IPv4 → ARP (Address Resolution Protocol) b. IPv6 → NDP (Neighbor Discovery Protocol) → If nobody answers, traffic never leaves the client
7. © 2025 Isovalent. All Rights Reserved. 7 Neighbour Solicitation The Missing Answer…
8. © 2025 Isovalent. All Rights Reserved. 8 Why Cloud Never Taught Us This On AWS / GCP / Azure, someone answers NDP for you silently • Cloud load balancers have IPs routed at the cloud network layer • No ARP/NDP needed, it’s fully abstracted • On bare metal: Kubernetes assigns the IP but nobody tells the network • The gap is silent, it looks like the app is broken when it isn't
9. © 2025 Isovalent. All Rights Reserved. 9 Cilium L2 Announcements Cilium makes a node claim the service IP on the L2 network • Works with any bare-metal or on-prem setup • Uses leader election: one node announces per service IP • Responds to NDP (IPv6) and ARP (IPv4) solicitations • No external load balancer needed
10. © 2025 Isovalent. All Rights Reserved. 10 Neighbor Advertisement
11. © 2025 Isovalent. All Rights Reserved. 11 Cilium L2 What You Need 1. CiliumLoadBalancerIPPool: the range to assign IPs from 2. CiliumL2AnnouncementPolicy: which IPs to announce, on which nodes, on which interfaces 3. A LoadBalancer Service: with a label matching the policy selector
12. © 2025 Isovalent. All Rights Reserved. 12 Cilium L2 What You Need apiVersion: "cilium.io/v2" kind: CiliumLoadBalancerIPPool metadata: name: l2-announcement-v6 spec: blocks: # This CIDR defines the range of external IPs Cilium will assign to LoadBalancer services # Must be routable on your L2 network (the Docker bridge for KinD) # Using fc00:f853:ccd:e793::/64 network, allocating from ::f000 range to avoid node IPs (::1-::ff) - cidr: "fc00:f853:ccd:e793::f000:0/112" 1. CiliumLoadBalancerIPPool: the range to assign IPs from
13. © 2025 Isovalent. All Rights Reserved. 13 Cilium L2 What You Need apiVersion: "cilium.io/v2alpha1" kind: CiliumL2AnnouncementPolicy metadata: name: announce-v6 spec: # Trigger announcement for LoadBalancer service external IPs loadBalancerIPs: true interfaces: # The network interface on worker nodes that will send NDP advertisements - eth0 nodeSelector: # Only worker nodes should announce (exclude control-plane) # This prevents the control-plane from advertising IPs it doesn't handle matchExpressions: - key: node-role.kubernetes.io/control-plane operator: DoesNotExist 2. CiliumL2AnnouncementPolicy: which IPs to announce, on which nodes, on which interfaces
14. © 2025 Isovalent. All Rights Reserved. 14 Cilium L2 What You Need apiVersion: v1 kind: Service metadata: name: httpd labels: # This label is matched by the L2AnnouncementPolicy announcement: l2 spec: # LoadBalancer type triggers external IP allocation from the CiliumLoadBalancerIPPool type: LoadBalancer selector: app: httpd ports: - port: 80 targetPort: 80 protocol: TCP 3. A LoadBalancer Service: with a label matching the policy selector
15. © 2025 Isovalent. All Rights Reserved. 15 Cilium L2 Apply the fix
16. © 2025 Isovalent. All Rights Reserved. 16 Takeaways • A LoadBalancer service IP is meaningless without L2 advertisement on bare metal • NDP/ARP is the missing link between Kubernetes and your network • Cilium L2 Announcements fills that gap, no extra infra required • One policy, one IP pool → external connectivity on any bare-metal cluster
17. © 2025 Isovalent. All Rights Reserved. 17 Takeaways • A LoadBalancer service IP is meaningless without L2 advertisement on bare metal • NDP/ARP is the missing link between Kubernetes and your network • Cilium L2 Announcements fills that gap, no extra infra required • One policy, one IP pool → external connectivity on any bare-metal cluster 🙏 Nico Vibert for diagrams and inspo
18. © 2025 Isovalent. All Rights Reserved. 18