eBPF at Mobile Network Data Plane

Japan Community Day at KubeCon + CloudNativeCon Japan 2025で発表した資料です。

イベントについて詳しくはこちらでご覧ください
https://community.cncf.io/events/details/cncf-cloud-native-community-japan-presents-japan-community-day-at-kubecon-cloudnativecon-japan-2025/

Transcript

1. eBPF at Mobile Network Data Plane 2025/06/15 Takeru Hayasaka <t-hayasaka@bbsakura.net> Japan Community Day at KubeCon + CloudNativeCon Japan 2025
2. Who am I? - Takeru Hayasaka - Senior Software Engineer / Mobile Core Development - MobileCore, SRv6-MUP - Mainly using eBPF(XDP) for PGW-U - Co-Organizer eBPF Meetup Japan
3. Why we adopted eBPF?
4. - Mobile networks have a system called the “Mobile Core”, which connects UEs to the Internet. - At first, we assigned one dedicated VM per user for the user-plane (PGW-U). - However, this architecture incurred high resource costs due to per-user VM allocation. Legacy Architecture in 2018 Roaming Network PGW-U VM PGW-U VM UE （e.g. smartphone) … UserNetworkA (e.g. The Internet) UserNetworkZ (e.g. Cloud Network) …
5. - We thought: “What if a single VM could serve multiple users? That would significantly reduce costs.” - The requirements to realize this: - High performance - Low development and operational costs - Low resource usage - Existing approaches like DPDK and Netmap couldn’t fully meet these requirements... Re-architecture and Requirements High-Performance PGW-U VM💪 Roaming Network e.g. The Internet & Cloud Network UE （e.g. smartphone) UserNetwork A-Z
6. - High-Performance - XDP processes packets at the driver level, before the network stack, enabling high-speed handling. - It offers performance close to DPDK, with simpler integration. - Low Cost(dev/ops/resource) - Coexists with the Linux stack, enabling reuse of existing infrastructure and reducing dev cost. - The eBPF verifier allows even C beginners to write safe code. - Unit tests using BPF_PROG_RUN improve development efficiency. - No dedicated CPU core needed, resulting in better power and core usage efficiency than DPDK. What We Expected from XDP
7. Practical Use Cases
8. PGW-U Architecture • Uplink: Decapsulates GTP packets from UE and forwards to the Internet • Downlink: Adds GTP headers and sends packets to UE via the mobile network • Other: Performs per-UE routing and applies traffic control policies The Internet & Cloud (Operator network) UE IP TCP IP ETH GTP/ UDP IP TCP ETH PGW-U Encap Uplink => Decap <=Downlink Dot1q Dot1q XDP TC-eBPF eBPF Maps FIB Table Application
9. Operational Challenges
10. - At the time, XDP was still evolving and had several limitations. - VLAN processing in the Linux stack didn’t support XDP metadata, which caused issues when integrating with TC. - -> Our patch was submitted upstream - [v3] net: Fix missing meta data in skb with vlan packet - virtio_net also lacked XDP metadata support - -> We implemented the necessary changes and contributed them upstream. - [bpf-next,v6,2/2] virtio_net: add XDP meta data support - Since driver behavior varied across environments, we established a consistent kernel and driver validation framework using virtme. XDP Was Still Evolving
11. Tools for Real-world Ops - To capture packets after XDP, we introduced Cloudflare’s xdpcap - Since XDP operates before the protocol stack, we used tools/libraries to store packets in eBPF Maps for capture. - We also built custom CLI tools to inspect eBPF Maps easily. - Combined with jq, it allows easy state inspection and investigation directly from the terminal.
12. Lessons Learned & Future Outlook
13. Lessons Learned & Future Outlook - XDP excels at simple processing - It delivers high throughput for lightweight tasks like Encap/Decap based on table lookups. - However, it has limitations with complex workloads - For example, QoS control, jumbo frame handling, and fragmentation are challenging. - Some protocols, such as GTP-U, make RSS-based load balancing difficult. While DPDK offers NIC control APIs, these are often unavailable on Linux. Still, alternatives like CPUMAP can be effective within eBPF. - Design can mitigate weaknesses - Rather than forcing XDP to handle everything, offload complex tasks to hardware or external systems. - Understanding XDP's characteristics in advance is key to determining if it's the right tool for the job. - Rich ecosystem and developer support - Official resources like samples in the Linux source tree and xdp-tutorial are great starting points. - If you're using a Go-based control plane, check out my goxdp-template for loading and managing XDP programs. cf. https://github.com/takehaya/goxdp-template
14. Enabling a Connected Future.