Which container ate my bandwidth?

Which Container Ate My Bandwidth? An eBPF Answer

Your VM's network traffic suddenly spikes.

Monitoring shows the increase, but not who's responsible. Docker stats gives partial answers. Traditional infrastructure dashboards tell you that bandwidth is being consumed, but not which container is consuming it or where the traffic is going.

This talk explores how eBPF can provide container-level network attribution directly from the Linux kernel.

Using cgroup_skb programs written in Go, I built a container network observer that tracks every byte sent and received by individual containers, broken down by remote IP address. No sidecars. No agents inside containers. No Docker API polling. Just kernel-level visibility and real-time network accounting.

We'll cover:

• Why existing observability tools struggle with per-container network attribution
• How eBPF cgroups provide accurate network visibility at the container boundary
• The architecture behind a production-ready monitoring tool written entirely in Go
• Challenges encountered while working with eBPF maps, cgroups, and container runtimes
• Lessons learned from turning a real infrastructure incident into a practical debugging tool

If you've ever looked at a bandwidth graph and wondered, "Which container actually caused this?", this session will show how to find the answer directly from the kernel.

Transcript

1. Gopher Africa · TALK Which container ate my bandwidth? An eBPF answer; built in Go, attached to cgroups, read from the kernel. SPEAKER Daniel Benjamin PROJECT docker-net-ebpf
2. PART ZERO Meet Netwatch
3. COLD OPEN · THE INCIDENT “ Unusual outbound network activity detected on VM. GOOGLE CLOUD · SECURITY ALERT DANIEL BENJAMIN · @PAPIDB 02
5. THE INCIDENT · THE QUESTION FIRST REACTION: IS THIS PHISHING? · IT WAS NOT. Which container is responsible for this traffic — and where is it going? DANIEL BENJAMIN · @PAPIDB 03
6. PART ONE I. The usual tools.
7. I · THE USUAL TOOLS What we reached for first. Each told us something. None of them answered the question. i docker stats Per-container CPU and memory. No remote IPs, no protocols, no byte counts on the wire. ii htop WordPress was spawning a lot of processes; a signal, not an answer. iii Prometheus + cAdvisor Traffic showed up on eth0; not on the container that caused it. iv Application logs A few thousand lines in, you are mostly guessing. DANIEL BENJAMIN · @PAPIDB 05
8. I · THE USUAL TOOLS Signals, not attribution. WHAT THE TOOLS COULD SHOW Traffic was high. One VM was the outlier. A specific container looked busy. Some process counts were unusual. WHAT NONE OF THEM COULD SHOW Which container · which remote IP. Ingress vs egress, separately. Exact byte counts on the wire. Traffic on the container not eth0. DANIEL BENJAMIN · @PAPIDB 06
9. I · THE USUAL TOOLS Why high-level tooling falls short. They share the same shape and the same blind spot. i They poll the Docker API. Asking the orchestrator what it thinks happened, not what did. ii They scrape summarized metrics. Pre-aggregated counters. The per-flow detail is already gone. iii They depend on agents. Heavy sidecars, slow rollout, blind spots when an agent isn't there. iv They lose attribution context. By the time data reaches the dashboard, the container link is gone. DANIEL BENJAMIN · @PAPIDB 07
10. PART TWO II. The kernel already knows.
11. Intro · A short detour What is eBPF? Tiny programs you load into the running Linux kernel — verified, sandboxed, JIT-compiled, attached to hook points. No recompile, no kernel module, no reboot. i Origin Berkeley Packet Filter, 1992: a tiny filter VM for tcpdump. Extended into eBPF in 2014 to run almost anywhere in the kernel. ii The contract A verifier proves your program halts, never reads invalid memory, and respects the kernel before it runs. If it doesn't, it's rejected. iii The interface Programs read and write maps, shared key-value structures. That's how user space talks to your kernel-side code. DANIEL BENJAMIN · @PAPIDB 05
12. II · THE KERNEL ALREADY KNOWS eBPF solution Write a small program. Have the kernel verify it. Run it at a hook point. Read results from a shared map. USER SPACE Write a tiny C program. Compile to BPF bytecode. Load via bpf() syscall. Read maps from Go. → VE RI FIE D · AT TA C HE D KERNEL Verifier checks safety. JIT to native code. Attach to a hook — here, the container cgroup. Runs on every packet. DANIEL BENJAMIN · @PAPIDB 09
13. II · THE KERNEL ALREADY KNOWS cgroup_skb: the right hook. Every container is a cgroup. Every packet crosses one of two well-defined hook points. Remote peer 93.184.216.34 CONTAINER CGROUP /sys/fs/cgroup/ docker/<id> Two BPF programs attach here, one for each direction. They see every packet, with full attribution. CGROUP/SKB · INGRESS Inbound bytes. CGROUP/SKB · EGRESS Outbound bytes. Process inside container DANIEL BENJAMIN · @PAPIDB 10
14. II · THE KERNEL ALREADY KNOWS The architecture. Four layers. The kernel does the counting; Go does the sense-making. KERNEL · 01 cgroup_skb programs ingress.c egress.c per-packet hook KERNEL · 02 BPF maps HASH<flow_key, stats> LRU eviction shared kernel ↔ user USER SPACE · 03 Go reader cilium/ebpf periodic map poll container resolver USER SPACE · 04 Exporters stdout · TUI Prometheus JSONL · SQLite DANIEL BENJAMIN · @PAPIDB 11
15. II · THE KERNEL ALREADY KNOWS Counting bytes in the kernel. // Runs on every egress packet from this container's cgroup. SEC("cgroup_skb/egress") int count_egress(struct -_sk_buff *skb) { flow_key_t key = { skb->saddr, skb->daddr, skb->protocol }; flow_stat_t init = { .bytes = skb->len, .packets = 1 }; flow_stat_t *s = bpf_map_lookup_elem (&flows, &key); if (s) { __sync_fetch_and_add (&s->bytes, skb->len); __sync_fetch_and_add(&s->packets, 1 ); } else { bpf_map_update_elem (&flows, &key, &init, BPF_ANY); } return 1; // observe, don't filter } DANIEL BENJAMIN · @PAPIDB 12
16. II · THE KERNEL ALREADY KNOWS Bridging to Go. // Load, attach, iterate. No C userspace — pure Go. obj -= flowsObjects{} loadFlowsObjects(&obj, nil ) link. AttachCgroup (link.CgroupOptions{ Path: "/sys/fs/cgroup/docker/<id>" , Attach: ebpf.AttachCGroupInetEgress, Program: obj.CountEgress, }) for range time.Tick (time.Second) { var key flowKey; var stat flowStat iter -= obj.Flows. Iterate () for iter.Next (&key, &stat) { emit(resolveContainer (key), key.Daddr, stat.Bytes) } } DANIEL BENJAMIN · @PAPIDB 13
17. PART THREE III. What the tutorials skip.
18. III · WHAT THE TUTORIALS SKIP The boring blockers. Days of debugging that no blog post will earn you sympathy for. i Missing kernel headers. No compile, no clue why. Different across every distro. ii asm/types.h not found. A one-line include path that swallows an afternoon. iii Cross-arch compilation. Build on arm64, target amd64. Every flag matters. CO-RE and BTF help — but only after you've fought the toolchain hard enough to recognize what they actually solve. DANIEL BENJAMIN · @PAPIDB 16
19. III · WHAT THE TUTORIALS SKIP Endianness will bite you. The kernel hands you bytes in network order. Print them naïvely and the IP looks scrambled. NETWORK BYTE ORDER · BIG-ENDIAN 93.184.216.34 5D B8 D8 22 How the packet arrives. How the kernel stores it in the map. HOST BYTE ORDER · LITTLE-ENDIAN (X86) 34.216.184.93 22 D8 B8 5D What you'll see if you forget ntohl() on the Go side. DANIEL BENJAMIN · @PAPIDB 17
20. III · WHAT THE TUTORIALS SKIP Containers to cgroups isn't clean. Three lookups, two filesystems, and a runtime quirk before you can name what you're counting. i CONTAINER ID a3f9c2e1b7… (the friendly name) ii CGROUP PATH /sys/fs/cgroup/system.slice/docker-a3f9c2e1b7.scope iii CGROUP INODE 3221225557 (what the kernel actually keys on) iv REVERSE LOOKUP cache · invalidate · resolve → container name DANIEL BENJAMIN · @PAPIDB 18
21. PART FOUR IV. When it clicked.
22. IV · WHEN IT CLICKED Real-time attribution. docker-net-ebpf · live $ sudo ./netbpf watch --interval 1s [ok] attached cgroup_skb/ingress · 7 containers [ok] attached cgroup_skb/egress · 7 containers CONTAINER REMOTE PROTO IN OUT wordpress-1 192.0.71.4 tcp · external 1.2 MB 218 kB wordpress-1 203.0.113.27 tcp · external 42 kB 9.1 MB wordpress-1 127.0.0.11 udp · docker dns 3 kB 3 kB postgres 10.0.0.12 tcp · internal 84 kB 31 kB redis 10.0.0.14 tcp · internal 2 kB 2 kB DANIEL BENJAMIN · @PAPIDB 20
23. PART FIVE V. Why this matters.
24. V · WHY THIS MATTERS The abstractions keep stacking. Managed platforms You don't own the host. You don't even know the host. LAYER 05 Sidecars & proxies Service-mesh metrics — for service-mesh traffic. LAYER 04 Orchestration Kubernetes knows pods, not packets. LAYER 03 Containers Process isolation. Network namespaces. veth pairs. LAYER 02 The Linux kernel Sees every packet. Always has. We just stopped asking. LAYER 01 DANIEL BENJAMIN · @PAPIDB 22
25. V · WHY THIS MATTERS EVERY INCIDENT, EVENTUALLY What is actually happening on this machine, right now? eBPF is not magic. It's a stable answer to that one question: Read straight from the layer that already knows. DANIEL BENJAMIN · @PAPIDB 23
26. V · WHY THIS MATTERS Where this goes next. From a debugging tool into something you can operate. Prometheus exporter Per-container, per-flow series; alert on what matters. JSONL · SQLite sinks Replayable history without standing up a TSDB. Traffic deltas & aggregations By IP, by domain, by minute; the views the incident needed. Multi-arch Docker distribution One command on amd64 and arm64 hosts. Sidecar deployment Drop in next to a workload, no host changes. A doctor mode Confirm if you can run the codebase natively DANIEL BENJAMIN · @PAPIDB 24
27. V · WHY THIS MATTERS What you take away. Three answers, worth more than the one this talk started with. i When eBPF is actually useful. Attribution that the layers above genuinely cannot give you. ii When existing tooling is enough. Most of the time. Prometheus, cAdvisor and stats earn their keep. iii How to start in Go. cilium/ebpf, one cgroup, one map. No prior eBPF experience required. DANIEL BENJAMIN · @PAPIDB 25
28. FIN Thank you. Questions, war stories, and PRs all welcome. SPEAKER Daniel Benjamin HANDLE @papidb PROJECT github.com/papidb/ docker-net-ebpf