XDPerf: A High-Performance Traffic Generator Built with WASM and eBPF

https://events.linuxfoundation.org/kubecon-cloudnativecon-japan/co-located-events/japan-community-day-schedule/?id=1261494
https://github.com/takehaya/xdperf

Transcript

1. XDPerf: A High-Performance Traffic Generator Built with WASM and eBPF 2026/07/28 Takeru Hayasaka <t-hayasaka@bbsakura.net> kubecon japan 2026 japan community day https://github.com/takehaya/xdperf
2. Whoami • 早坂 彪流 (Hayasaka Takeru|@takemioIO) • Senior Software Enginner at SAKURA internet Inc. ◦ Currentry on secondment to BBSakura Networks.Inc • Develops and operates mobile core and virtualization infrastructure ◦ Previously wrote game console firmware at a game company :-) • My favorite eBPF Helper Func is bpf_fib_lookup • Co-Organizer of the eBPF Japan Meetup 2
3. Agenda • Background: Why did we create XDPerf? • Tech background: XDP and live frames mode • Architecture • How it works: the template engine • Performance: measurements at 100G • Demo • Lessons learned 3
4. Background: Why did we create XDPerf? 4
5. XDPerf: High-Performance Traffic Gen with XDP - An XDP-based packet generator for high-throughput load testing - WebAssembly (Wasm) plugins - Define packet templates and variation rules - Written in any language that compiles to Wasm (Rust, TinyGo, ...) - Go application - A single Go binary embeds the Wasm runtime (wazero) and manages all eBPF interaction - Generates packet byte arrays with the plugin's template and registers them into eBPF maps XDP in the kernel does the sending. The packets you want to throw, you write in Wasm https://github.com/takehaya/xdperf 5
6. Traffic generators cluster at two extremes The gap near DPDK-class pps, without the DPDK setup XDPerf performance （PPS) pktgen pktgen-dpdk TRex MoonGen NIC takeover, hugepages, NIC binding (TRex/DPDK family) netperf iperf3 socket, stock kernel (socket/iperf family) setup cost/env dependent 6
7. Why the existing tools fall short socket / iperf family • Socket-based, so nothing below TRex / DPDK family • L4 is reachable • Cannot craft custom headers anything else during tests) • such as VLAN, tunnels, or SRv6 • Not built for Mpps-class pps testing • Needs hugepages and vfio/uio driver binding • TRex drags in a Python environment and its dependencies Coarse control over flow diversity Takes over the NIC (unusable for • Hard to bring into CI or short-lived test environments 7
8. What the tool needed to be Issue Requirement Sockets cannot reach below L4 Craft L2/L3 freely, at Mpps to 100 Mpps class Driver binding and NIC takeover are heavy Keep the driver, share the NIC; back to normal right after the test Heavy dependencies keep it out of CI A single dependency-free binary Nothing satisfied all three at once, so I built it 8
9. XDPerf show brief curl -fsSL https://raw.githubusercontent.com/takehaya/xdperf/main/scripts/install_xdperf.sh | sudo bash sudo xdperf probe --device eth0 # check XDP / live frames support sudo xdperf run --device eth0 --duration 30s --pps 100k --plugin simpleudp.go • Transmission uses XDP live frames (explained in the tech background section ) • Packet definitions use Wasm plugins (explained from the architecture section ) 9
10. Tech background: XDP and live frames mode 10
11. What is eBPF? • A mechanism for safely running small programs inside the Linux kernel • The eBPF verifier proves each program safe before it runs, JIT-compiled to native code (it cannot crash the kernel) • Hook points exist for networking, tracing, and security. XDP is one of the networking hooks 11
12. XDP Architecture Packet arrives The return code decides what happens NIC driver XDP_PASS XDP_TX / REDIRECT XDP hook (eBPF program) before skb: no alloc, no stack yet Kernel stack => socket normal receive path Transmitted straight from the NIC bounce / forward, skipping the stack XDP_DROP dropped on the spot Fast because it stays in the driver layer. But all of this is the receive side 12
13. BPF_PROG_RUN live frames: making syscalls disappear Conventional: socket TX User space BPF_PROG_RUN live frames (kernel 5.18+) User space call sendmsg() per packet bpf(BPF_PROG_TEST_RUN, repeat=N) syscalls × packet count 1 syscall (batched, per-cpu) Kernel In-kernel loop protocol stack => driver run XDP prog N times, XDP_TX really sends ×N ndo_xdp_xmit NIC NIC 13
14. Probe subcommand $ sudo ./out/bin/xdperf probe --device enp138s0f0np0 2026-07-20T11:31:46Z INFO Probing XDP capabilities {"device": "enp138s0f0np0"} 2026-07-20T11:31:47Z INFO Device {"name": "enp138s0f0np0"} 2026-07-20T11:31:47Z INFO XDP Support {"supported": true, "driver_mode": true, "generic_mode": true, "offload_mode": false} 2026-07-20T11:31:47Z INFO BPF_PROG_RUN Support {"live_frame_mode": true} 2026-07-20T11:31:47Z INFO Summary: This device is fully compatible with xdperf • driver_mode is native XDP. This is where the performance is (ice / mlx5 / i40e / ixgbe / virtio_net / veth, etc.) • generic_mode is the skb-based fallback. It works, but slowly • live_frame_mode needs kernel 5.18 or later. Required for transmission • --json output makes it usable for environment checks in CI 14
15. Prior implementations Implementation Notes xdp-trafficgen The canonical implementation by the author of live frames mode itself. Achieves roughly 8 Mpps per core. IPv6-only, with source ports variable inside the XDP program. Its probe design informed ours. eunomia-bpf tutorial A teaching implementation of a BPF_PROG_RUN packet generator. Great for learning the mechanism higebu/xdperf Go + cilium/ebpf. IPv4/IPv6 UDP, configurable parallelism and batch-size. The namesake and starting point of this project (now archived) All of them send essentially a fixed packet. variation is limited to things like port sweeps… 15
16. Live frames sends one fixed packet, repeatedly Single flow (fixed packet) Multiflow (sweep src port / IP) Packets with an identical 5-tuple Packets with varying 5-tuples RSS on the DUT NIC RSS on the DUT NIC Queue 0 CPU0: 100% Queue 1 Queue N Queue 0 Queue 1 Queue N idle idle CPU0 CPU1 CPUN Load piles onto a single DUT core Spread across all cores = real performance 16
17. Where should the flexibility live? Approach Assessment Verdict (a) Generate whole packets inside eBPF Fights the verifier, every new protocol means kernel-side changes Rejected on paper (b) Pre-build full Implemented and measured. packets, copy each one full-packet memcpy became the out bottleneck Rejected by measurement (c) Write only diffs onto a base packet Adopted🙆 Generation runs once at startup; the send path writes a few bytes Why (b) failed: copying every packet in full saturates memory bandwidth. Live frames takes the syscalls away, and memcpy hands the win right back 17
18. Architecture 18
19. The big picture Wasm plugin (TinyGo / Go / Rust / ...) Skeleton + variation rules: the only part users write packet template (JSON) Can be slow runs once at startup owns flexibility xdperf host (Go + wazero) expands template into diff entries / precomputes checksum meta writes eBPF maps (base_packet_map / diff_map) eBPF/XDP program (xdp_tx) base + diffs + checksum: per-CPU, no lock contention BPF_PROG_RUN live frames loop (1 syscall) Fas runs t per packet owns performance NIC (unmodified upstream driver) ice / mlx5 / i40e / virtio_net / veth ... : no binding, no takeover 19
20. XDPerf: User land Architecture Go Application eBPF Maps pkt_map Plugin Source Code Wazero (Wasm Runtime) Compiled Plugin Wasm Binary Metrics Server Load & Run Define Packet Template (randomize ports, etc) OLTP(optional) Main Logic Send Template Generate Packet ByteArray Send TX packets in Live Frame Mode per CPU Register Per Core Thread for Run Run XDP bpf(2) eBPF Maps stats_map terminal (stdio) show terminal Show Stats Data (tx/rx pkt&byte count) Fetch 20
21. XDPerf: Kernel land Architecture TX path (sender) BPF_PROG_RUN live frames loop 1 syscall / batch, per-CPU thread pkt_map lookup base_packet_map diff_map / checksum_meta stats_map ① fetch diff_entry per-CPU round robin ② copy base skipped if same base & len ③ apply diffs a few bytes written ④ checksum cached → skip / first: tail call XDP_TX TX loaded: xdp_tx program buffer recycled: previous packet still in place RX path (receiver / swap-resp echo mode) XDP_TX ③ swap resp swap MAC / IP / port tx/rx packets & bytes loaded: xdp_rx program (PERCPU) (attach also allocates XDP TX queues) update (PERCPU) ② stats++ recv packets / bytes ① parse & match dst port filter RX 21
22. Why Wasm • Packet definitions should be code, not config files (you need loops, branches, randomness, state) • But the host should not grow runtime dependencies (the flip side of getting burned by TRex's Python stack) Wasm satisfies both at once • Zero dependencies: the artifact is a single .wasm file • Sandboxed: a plugin cannot break the host • Language-agnostic: TinyGo / Go / Rust / C, and so on 22
23. Why wazero Runtime Written in Go embedding CGO wazero pure Go native not needed wasmtime-go Rust via wrapper required wasmer-go Rust via wrapper required WasmEdge C++ via SDK required wasm-micro-runtime C via bindings required • wazero is the only one without CGO. That preserves the single binary and cross-compilation, and that was the deciding factor • I did not benchmark execution speed. Wasm runs exactly once at startup, so speed was never a selection criterion 23
24. The plugin API //go:wasmexport plugin_init //go:wasmexport plugin_process // the actual packet generation //go:wasmexport plugin_cleanup • The host writes config (JSON) into Wasm memory; the plugin returns a template (JSON) In the other direction, plugins can call host functions (pkg/guest) • NeighborResolve(ip, iface): resolves a destination IP to a MAC. Reads the kernel neighbor cache; if empty, sends one UDP datagram to trigger ARP/ND, then reads again • Log / ReportMetric: plugin output into the host's logger and metrics • Plus constants for header lengths and checksum offsets, and ChecksumSpec builders…etc 24
25. How it works: The template engine 25
26. Template engine (1): declaring the variation Declaration: vary 2B at offset 34 over 1024-1124, sequential Base packet (just one in the map) Ethernet IPv4 14B 20B csum src port offset 34 csum Payload Amber cells are checksums, affected by rewrites (next slide) Host expands into diff entries port=1024 port=1025 port=1026 … port=1124 One entry = a 2-byte diff + offset (==not a full 64B packet) Per packet, XDP writes diff[i] onto the base and transmits One base; only bytes of diffs grow. Easy on memory bandwidth and caches! 26
27. The template, in actual plugin code res := guest.GeneratorProcessResponse{ TemplateType: guest.GeneratorTemplateTypeVariable, Base: the one base packet in the map VariablePacketTemplate: guest.PacketVariantSet{ Variants: []guest.PacketVariant{{ Base: guest.BasePacket{Data: packetBytes, Length: maxLen}, ByteStart / ByteSize: the 2B src-port cell (offset 34) Params: []guest.VariableParams{{ ByteStart: srcPortOffset, // = 34 for Eth/IPv4/UDP ByteSize: 2, ByteRange: guest.TemplateRange{Start: 1024, End: 1124}, PatternType: guest.ValuePatternTypeSequential, }}, Checksums: guest.IPv4UDPChecksumSpecs( uint16(srcPortOffset) - guest.IPv4HeaderLen), }}, Pattern: guest.VariantSelectionModeSequential, },} ByteRange + PatternType: sweep 1024-1124, sequential Checksums: the amber csum cells SDK computes offsets No magic numbers: offsets and checksum specs come from the SDK 27
28. Template engine (2): applying diffs in eBPF ① Fetch diff_entry ② Copy base ③ Apply diffs per-CPU round robin skipped if same base & len a few bytes written ④csum cached? NO (first time only) YES Hot path (2nd packet on) XDP_TX just ①-④ + XDP_TX no compute, no tail call Tail call: checksum compute full recompute (variable len) or incremental (bpf_csum_diff) next packet takes the hot path result cached into diff slots => csum_cached=1 28
29. The hardest part was checksums Rewriting the 2-byte breaks the UDP csum Ethernet IPv4 src/dst IP IP csum src port UDP csum Payload IP csum covers the whole IPv4 header UDP csum covers UDP header + payload + pseudo-header (incl. src/dst IP) Which diff breaks which checksum differs per field → The Go host computes this mapping at startup, embedded as an affects_csum bitmask • A full recompute is packet length and hits the verifier's instruction limits, worked around with bpf_loop • Incremental updates (bpf_csum_diff) need the pre-rewrite value. XDP_TX recycles buffers, so old values are captured from the live buffer on first use • The IPv6 pseudo-header needs the final destination. For SRv6, walk the SRH and use segment[0] (RFC 8754) 29
30. Supported protocols and extensibility Plugin What it does simpleudp UDP with 802.1Q VLAN. The default imixudp Weighted IMIX (multiple variants) gtpv1u GTPv1-U (5G/4G), with PSC/QFI vxlan VXLAN, VNI sweep and inner Ethernet SRv6 mode: L3VPN(inner v4),inner v6, L2VPN • For E2E testing there are also ARP, LLDP, ICMP, TCP, IPv6, EAPOL, MPLS, SRv6, and raw • Supporting a new protocol means writing a Wasm plugin; no eBPF changes needed 30
31. Performance 31
32. Example: Round-Trip Test with MAC/IP Address Swapping Uplink Downlink Client Host IP ETH ETH IP UDP Received the packet that was sent! Server Host eth0 eth0 UDP Mac/IP Address Swapd sudo .xdperf run --device eth0 \ --plugin simpleudp.go \ --count 128 --parallelism 24 --infinite --batch-size 64 \ --cfg '{"src_ip":"192.168.1.1","dst_ip":"192.168.1.2","dst_port":10001,"p ayload_size":22,"is_arp_resolve":true, "src_port_sweep_start":1024,"src_port_sweep_end":1124}' Swap MAC/IP on receive and send back! sudo xdperf --device eth0 --send=false --recv=true --swap-resp 32
33. Test environment • Kernel: Linux 7.2-rc (client), 6.8 (server) • CPU: Intel Xeon Platinum 8362 @ 2.80 GHz ◦ 32 cores / 64 threads / 1 NUMA node • Memory: 64 GB • NIC: Intel E810-C 100GbE (ice driver 21.5.9) • 100GbE direct cable, back-to-back (no switch) Sender settings: --batch-size 64, packet pool 128, mitigations=off iommu=pt (see the IOMMU slide). All numbers are receive-side counters 33
34. 116.5 Mpps at 64B, wire rate from 256B up Mpps 160 100G wire rate One-way RX 128B 256B Round-trip RX 120 80 40 0 64B 512B Frame size (on-wire) 1024B 1280B 1500B 34
35. Cores do not scale linearly forever Mpps 180 Linear ideal (1 core x N) Measured (64B round-trip RX, kernel 6.15) 150 Peak 110.7 Mpps (24 cores) 120 90 60 30 0 1 2 4 8 16 24 32 TX cores (parallelism) 35
36. No performance on real hardware. The culprit: IOMMU 72.58% native_queued_spin_lock_slowpath -> cache_tag_flush_range_np -> intel_iommu_iotlb_sync_map -> iommu_dma_map_page -> __ice_xmit_xdp_ring • Intel IOMMU (VT-d) flushed the IOTLB on every packet's DMA mapping, and everything spun on that lock • Fixed with iommu=pt (passthrough), bypassing DMA address translation • Measured impact (kernel 7.2): intel_iommu=on 42.3 Mpps -> iommu=pt 115.8 Mpps (2.74x) • The virtio-net VM never showed this because VT-d was not in effect there 36
37. It also works where containers live (veth) Measured on a netns pair over veth (64B) Setup Mpps default veth (1 queue) 7.4 multi-queue veth (8 queues) 61.3 • Default veth serializes receive processing on a single queue, which caps throughput • Adding queues scales nearly linearly (36.7 Mpps at 1500B) 37
38. Demo 38
39. Demo 1. The install one-liner (a few seconds) 2. xdperf probe --device eth0 to confirm support 3. On the peer: xdperf run --send=false --recv --swap-resp (echo) 4. Send in Both mode and watch recv/s tick at tens of Mpps 39
40. Demo: Install 40
41. Demo: probe cmd 41
42. Demo: perf start 42
43. Demo: perf continue 1.send(xmit) 3.response 2.recv 43
44. Demo: perf end 44
45. Lessons learned 45
46. Multi-kernel CI is essential for eBPF tools • Verifier behavior differs across kernel versions ◦ 6.1 tracks variable offsets poorly; worked around with doubled bounds checks and __noinline ◦ 7.2 needed a build fix for the BPF stack-arguments change • vimto runs the CI matrix from 6.1 to 7.2 vimto -kernel :6.1 -- go test ./pkg/coreelf/ -run TestBpf vimto -kernel :6.6 -- go test ./pkg/coreelf/ -run TestBpf vimto -kernel :6.12 -- go test ./pkg/coreelf/ -run TestBpf vimto -kernel :6.18 -- go test ./pkg/coreelf/ -run TestBpf … Working on your kernel guarantees nothing about the others 46
47. When building on new kernel features, upstream contribution is essential • The batch parameter for live frames was missing from cilium/ebpf. Too niche: nobody had needed it • So I sent the PR and it was merged (cilium/ebpf #1914) • Batch support cut syscalls further: 6.1 → 12.3 Mpps on the virtio-net VM, a 2x A tool on bleeding-edge kernel features walks ahead of its libraries. Build the missing piece, give it back 47
48. Summary 48
49. Summary 1. 100G wire rate / 116.5 Mpps without DPDK, on unmodified upstream drivers 2. Arbitrary packets via Wasm plugins and the diff-based template engine 3. A single binary, installed with one line! a. curl -fsSL https://raw.githubusercontent.com/takehaya/xdperf/main/scripts/install_xdperf.sh | sudo bash https://github.com/takehaya/xdperf 49
50. Enabling a Connected Future. 50