Slide 1

Slide 1 text

Who Owns This IP? Debugging L2 in Kubernetes Donia Chaiehloudj, Software Engineer & Community

Slide 2

Slide 2 text

© 2025 Isovalent. All Rights Reserved. 2 The Situation You just deployed your first bare-metal Kubernetes cluster ● Here a kind cluster ● You deploy an app, create a LoadBalancer service ● Kubernetes assigns an external IPv6 address ● You try to curl it from outside the cluster…

Slide 3

Slide 3 text

© 2025 Isovalent. All Rights Reserved. 3 Let's Debug

Slide 4

Slide 4 text

© 2025 Isovalent. All Rights Reserved. 4 Let's Debug First instinct: is the app even running? ● kubectl get pods → all Running ✓ ● curl from inside a pod → works ✓ ● curl from a cluster node → works ✓ ● The problem is outside the cluster

Slide 5

Slide 5 text

© 2025 Isovalent. All Rights Reserved. 5 Is It a Routing Problem? Maybe the IP isn't routed correctly? ● ip addr on the nodes: the external IP doesn't exist anywhere ● Ping from the client: no response, not even "unreachable" ● tcpdump on the client: packets go out, nothing comes back ● The IP is a ghost: it lives in etcd, nowhere else

Slide 6

Slide 6 text

© 2025 Isovalent. All Rights Reserved. 6 Who on this network knows that IP exists? ● Every IP needs an owner: a MAC address ● Before sending a packet, your client asks: "Who has this IP?" a. IPv4 → ARP (Address Resolution Protocol) b. IPv6 → NDP (Neighbor Discovery Protocol) → If nobody answers, traffic never leaves the client

Slide 7

Slide 7 text

© 2025 Isovalent. All Rights Reserved. 7 Neighbour Solicitation The Missing Answer…

Slide 8

Slide 8 text

© 2025 Isovalent. All Rights Reserved. 8 Why Cloud Never Taught Us This On AWS / GCP / Azure, someone answers NDP for you silently ● Cloud load balancers have IPs routed at the cloud network layer ● No ARP/NDP needed, it’s fully abstracted ● On bare metal: Kubernetes assigns the IP but nobody tells the network ● The gap is silent, it looks like the app is broken when it isn't

Slide 9

Slide 9 text

© 2025 Isovalent. All Rights Reserved. 9 Cilium L2 Announcements Cilium makes a node claim the service IP on the L2 network ● Works with any bare-metal or on-prem setup ● Uses leader election: one node announces per service IP ● Responds to NDP (IPv6) and ARP (IPv4) solicitations ● No external load balancer needed

Slide 10

Slide 10 text

© 2025 Isovalent. All Rights Reserved. 10 Neighbor Advertisement

Slide 11

Slide 11 text

© 2025 Isovalent. All Rights Reserved. 11 Cilium L2 What You Need 1. CiliumLoadBalancerIPPool: the range to assign IPs from 2. CiliumL2AnnouncementPolicy: which IPs to announce, on which nodes, on which interfaces 3. A LoadBalancer Service: with a label matching the policy selector

Slide 12

Slide 12 text

© 2025 Isovalent. All Rights Reserved. 12 Cilium L2 What You Need apiVersion: "cilium.io/v2" kind: CiliumLoadBalancerIPPool metadata: name: l2-announcement-v6 spec: blocks: # This CIDR defines the range of external IPs Cilium will assign to LoadBalancer services # Must be routable on your L2 network (the Docker bridge for KinD) # Using fc00:f853:ccd:e793::/64 network, allocating from ::f000 range to avoid node IPs (::1-::ff) - cidr: "fc00:f853:ccd:e793::f000:0/112" 1. CiliumLoadBalancerIPPool: the range to assign IPs from

Slide 13

Slide 13 text

© 2025 Isovalent. All Rights Reserved. 13 Cilium L2 What You Need apiVersion: "cilium.io/v2alpha1" kind: CiliumL2AnnouncementPolicy metadata: name: announce-v6 spec: # Trigger announcement for LoadBalancer service external IPs loadBalancerIPs: true interfaces: # The network interface on worker nodes that will send NDP advertisements - eth0 nodeSelector: # Only worker nodes should announce (exclude control-plane) # This prevents the control-plane from advertising IPs it doesn't handle matchExpressions: - key: node-role.kubernetes.io/control-plane operator: DoesNotExist 2. CiliumL2AnnouncementPolicy: which IPs to announce, on which nodes, on which interfaces

Slide 14

Slide 14 text

© 2025 Isovalent. All Rights Reserved. 14 Cilium L2 What You Need apiVersion: v1 kind: Service metadata: name: httpd labels: # This label is matched by the L2AnnouncementPolicy announcement: l2 spec: # LoadBalancer type triggers external IP allocation from the CiliumLoadBalancerIPPool type: LoadBalancer selector: app: httpd ports: - port: 80 targetPort: 80 protocol: TCP 3. A LoadBalancer Service: with a label matching the policy selector

Slide 15

Slide 15 text

© 2025 Isovalent. All Rights Reserved. 15 Cilium L2 Apply the fix

Slide 16

Slide 16 text

© 2025 Isovalent. All Rights Reserved. 16 Takeaways ● A LoadBalancer service IP is meaningless without L2 advertisement on bare metal ● NDP/ARP is the missing link between Kubernetes and your network ● Cilium L2 Announcements fills that gap, no extra infra required ● One policy, one IP pool → external connectivity on any bare-metal cluster

Slide 17

Slide 17 text

© 2025 Isovalent. All Rights Reserved. 17 Takeaways ● A LoadBalancer service IP is meaningless without L2 advertisement on bare metal ● NDP/ARP is the missing link between Kubernetes and your network ● Cilium L2 Announcements fills that gap, no extra infra required ● One policy, one IP pool → external connectivity on any bare-metal cluster 🙏 Nico Vibert for diagrams and inspo

Slide 18

Slide 18 text

© 2025 Isovalent. All Rights Reserved. 18