Slide 1

Slide 1 text

eBPF Meets the GPUξ‚’ The Future of AI Infrastructure Observability Donia Chaiehloudj, Software Engineer & Community KCD Provence organiser 🌞

Slide 2

Slide 2 text

Agenda ● GPU 101 ● What is eBPF ● The observability gap ● A landscape of eBPF tools for GPU Observability ● The future of eBPF for AI Infrastructure ● Practical Toolkit @doniacld

Slide 3

Slide 3 text

When was last time you were on call? @doniacld

Slide 4

Slide 4 text

When was last time you were on call? ⚠ ALERT: llm-inference-api P99 latency: 2.4s β†’ 9.8s (+408%) βœ“ CPU: 34% avg βœ“ Memory: 62% used βœ“ Network: 1.2 Gbps βœ“ Disk I/O: 45 MB/s GPU Dashboard: └─ Utilization: 87% └─ Temperature: 76Β°C └─ Power: 285W / 350W @doniacld

Slide 5

Slide 5 text

AI Infrastructure Stack @doniacld

Slide 6

Slide 6 text

AI Workloads on GPUs: Training Workflow Goal: Build the model Duration: Days Batch: 512-2048 NNCL: Neural Network Coding Layer @doniacld

Slide 7

Slide 7 text

AI Workloads on GPUs: Training Workflow Goal: Build the model Duration: Days Batch: 512-2048 @doniacld NNCL: Neural Network Coding Layer

Slide 8

Slide 8 text

AI Workloads on GPUs: Inference Workflow Goal: Serve predictions Duration: ms per request Batch: 1-32 requests @doniacld

Slide 9

Slide 9 text

Training vs Inference Bottlenecks Training: ● πŸ”₯ Compute: Matrix multiply dominates GPU cycles ● πŸ”„ Communication: NCCL AllReduce syncing gradients between GPUs ● πŸ’Ύ Memory: Storing activations + gradients fills VRAM ● πŸ“ I/O: Loading training batches from disk Inference ● πŸ’Ύ Memory bandwidth: Reading KV cache every decode step ● ⏳ Sequential decode: One token at a time, can't parallelize ● πŸ’Ύ VRAM capacity: Long context = giant KV cache β†’ OOM ● ⚑ Prefill latency: Large prompts take time to process @doniacld

Slide 10

Slide 10 text

The Visibility Gap GPU executions we canΚΌt see: ● GPU kernel execution ● Wrap scheduling ● VRAM pressure ● Memory transfers ● NNCL communication ● Tensor Code usage Vendor profilers a. NVIDIA's Nsight b. CUPTI Cons ● 10 to 30% overhead ● Run a profiling session and analyse results offline ● No correlation between CPU-side events and GPU events @doniacld

Slide 11

Slide 11 text

The Visibility Gap @doniacld Some GFX vendors do offer some additional tooling Intel GPU Ntop screenshot

Slide 12

Slide 12 text

GPU Observability Tools @doniacld

Slide 13

Slide 13 text

GPU Observability Tools 🧱 Visibility Wall @doniacld

Slide 14

Slide 14 text

What is ? @doniacld Makes the Linux kernel programmable

Slide 15

Slide 15 text

Without @doniacld

Slide 16

Slide 16 text

With @doniacld

Slide 17

Slide 17 text

Overview of @doniacld Program App Linux Kernel Userspace syscalls

Slide 18

Slide 18 text

@doniacld Program App Linux Kernel Userspace syscalls event Overview of

Slide 19

Slide 19 text

@doniacld Program App Linux Kernel Userspace syscalls event Events HookPoints) ● tracepoints ● kprobes (kernel) ● uprobes (user) ● TC Traffic Control) ● XDP Express Datapath) Overview of

Slide 20

Slide 20 text

is safe @doniacld Program App Linux Kernel Userspace syscalls

Slide 21

Slide 21 text

@doniacld Program App Linux Kernel Userspace syscalls eBPF verifier is safe

Slide 22

Slide 22 text

@doniacld Program App Linux Kernel Userspace syscalls eBPF verifier eBPF verifier guarantees: ● No Infinite Loops ● Memory Safety ● Kernel Integrity is safe

Slide 23

Slide 23 text

@doniacld Program App Linux Kernel Userspace syscalls eBPF verifier approved eBPF verifier guarantees: ● No Infinite Loops ● Memory Safety ● Kernel Integrity is safe

Slide 24

Slide 24 text

@doniacld Program App Linux Kernel Userspace syscalls eBPF verifier eBPF JIT Compiler approved eBPF verifier guarantees: ● No Infinite Loops ● Memory Safety ● Kernel Integrity is safe and performant

Slide 25

Slide 25 text

@doniacld Program App Linux Kernel Userspace syscalls eBPF verifier eBPF JIT Compiler approved eBPF verifier guarantees: ● No Infinite Loops ● Memory Safety ● Kernel Integrity eBPF Just In Time Compiler: ● eBPF bytecode ⟢ native machine code ● ~ same speed as native kernel code is safe and performant

Slide 26

Slide 26 text

@doniacld App Linux Kernel Userspace Program syscalls Maps Use Cases: ● kernel β†’ userspace ● userspace β†’ kernel ● kernel β†’ kernel Program write read Communicating with Write/Read

Slide 27

Slide 27 text

@doniacld Program pod Linux Kernel Userspace event pod pods One kernel for all the containers syscalls Why for Kubernetes?

Slide 28

Slide 28 text

@doniacld Why for Kubernetes?

Slide 29

Slide 29 text

Three Layers of Observability @doniacld

Slide 30

Slide 30 text

Four Layers of Observability Layer 1: Runtime Observability Layer 2: Inference-Aware Observability Layer 3: Infrastructure Correlation @doniacld

Slide 31

Slide 31 text

Layer 1ξ‚’ GPU Runtime Observability @doniacld

Slide 32

Slide 32 text

GPUprobe: Tracing CUDA runtime calls with eBPF @doniacld

Slide 33

Slide 33 text

GPUprobe: Tracing CUDA runtime calls with eBPF @doniacld

Slide 34

Slide 34 text

GPUprobe: Tracing CUDA Workloads with eBPF Traditional observability stops at: ● CPU usage ● Process metrics ● GPU utilisation % But tells us nothing about: ● CUDA runtime latency ● kernel launch timing ● memory allocation pressure ● synchronization bottlenecks ● inference execution flow @doniacld

Slide 35

Slide 35 text

GPUprobe: Tracing CUDA Workloads with eBPF Traditional observability stops at: ● CPU usage ● Process metrics ● GPU utilisation % But tells us nothing about: ● CUDA runtime latency ● kernel launch timing ● memory allocation pressure ● synchronization bottlenecks ● inference execution flow @doniacld

Slide 36

Slide 36 text

GPUprobe: Tracing CUDA Workloads with eBPF GPUprobe uses eBPF uprobes/kprobes to trace: ● CUDA runtime APIs ● kernel launches ● memory allocations ● synchronization events ● GPU driver interactions Without modifying the application @doniacld

Slide 37

Slide 37 text

GPUprobe: Tracing CUDA Workloads with eBPF sudo bpftrace -e ' uprobe:/usr/lib/x86_64-linux-gnu/libcuda.so:cuLaunchKernel { @kernel_launches[comm] = count(); printf("CUDA kernel launch: process=%s pid=%d\n", comm, pid); } uprobe:/usr/lib/x86_64-linux-gnu/libcudart.so:cudaMemcpy { @memcpy_calls[comm] = count(); printf("CUDA memory copy: process=%s pid=%d\n", comm, pid); } tracepoint:syscalls:sys_exit_ioctl /comm -= "python"/ { @ioctl_latency = hist(nsecs); } ' CUDA kernel launch: process=python pid=1842 CUDA memory copy: process=python pid=1842 CUDA kernel launch: process=python pid=1842 CUDA kernel launch: process=python pid=1842 @kernel_launches[python]: 3421 @memcpy_calls[python]: 891 @ioctl_latency: [1K, 2K) 120 [2K, 4K) 387 [4K, 8K) 912 [8K, 16K) 240 [16K, 32K) 42 @doniacld

Slide 38

Slide 38 text

GPU Time Operation Tracing Demo @doniacld

Slide 39

Slide 39 text

@doniacld GPU Time Operation Tracing Demo gpudemo@DESKTOP-N3HOOD6:~/demo1$ python3 gpu_test.py ================================================== Testing GPU with PyTorch ================================================== CUDA available: True GPU: NVIDIA GeForce RTX 4090 Creating tensors and moving to GPU--. Running matrix multiplication on GPU--. Iteration 1: result shape torch.Size([2000, 2000]) Iteration 2: result shape torch.Size([2000, 2000]) Iteration 3: result shape torch.Size([2000, 2000]) Iteration 4: result shape torch.Size([2000, 2000]) Iteration 5: result shape torch.Size([2000, 2000]) Moving result back to CPU--.

Slide 40

Slide 40 text

@doniacld GPU Time Operation Tracing Demo gpudemo@DESKTOP-N3HOOD6:~/demo1$ sudo bpftrace cuda_trace.bt Attaching 3 probes... GPU MALLOC | PID: 15641 | Process: python3 | Time: 169232197289074 ns GPU MALLOC | PID: 15641 | Process: python3 | Time: 169232214662144 ns GPU MALLOC | PID: 15641 | Process: python3 | Time: 169233217615951 ns GPU MALLOC | PID: 15641 | Process: python3 | Time: 169233225441813 ns GPU MALLOC | PID: 15641 | Process: python3 | Time: 169233727676077 ns

Slide 41

Slide 41 text

Layer 2ξ‚’ Inference-Aware Observability @doniacld

Slide 42

Slide 42 text

eInfer: What is it? eInfer shows: ● why inference latency spikes ● token generation stalls ● decode bottlenecks ● request-level behavior Source: β€œeInfer: Unlocking Fine-Grained Tracing for Distributed LLM Inference with eBPF” paper (https://dl.acm.org/doi/pdf/10.1145/3748355.3748372) eInfer correlates LLM Inference with GPU Activity: LLM requests ↔ CUDA Runtime activities ↔ System Kernel signals @doniacld

Slide 43

Slide 43 text

eInfer: How does it work? @doniacld

Slide 44

Slide 44 text

eInfer: How to use it? # Attach to running inference server sudo ./einfer -p # Output: per-request traces Request ID: abc123 β”œβ”€ Queue: 5ms β”œβ”€ Prefill: 45ms (512 tokens) β”‚ └─ 15 CUDA kernels β”œβ”€ Decode: 1.2s (100 tokens) β”‚ β”œβ”€ Token 1-50: 8ms avg β”‚ └─ Token 51-100: 15ms avg ⚠ β”‚ └─ KV cache realloc at token 64 β”œβ”€ TTFT (Time To First Token): 50ms └─ Total: 1.25s @doniacld

Slide 45

Slide 45 text

GPU Memory Transfer Tracing Demo @doniacld

Slide 46

Slide 46 text

@doniacld GPU Memory Transfer Tracing Demo gpudemo@DESKTOP-N3HOOD6:~/demo2$ python3 gpu_test2.py CUDA available: True GPU: NVIDIA GeForce RTX 4090 [Phase 1] Allocating GPU tensors of different sizes--. Small tensor: torch.Size([100, 100]) (~40 KB) Medium tensor: torch.Size([1000, 1000]) (~4 MB) Large tensor: torch.Size([2000, 2000]) (~16 MB) [Phase 2] Explicit host->device memory transfers--. Transferred 3.81 MB to GPU [Phase 3] Running GPU computations (kernel launches)--. Iteration 1: matmul result shape torch.Size([1000, 1000]) Iteration 2: matmul result shape torch.Size([1000, 1000]) Iteration 3: matmul result shape torch.Size([1000, 1000]) [Phase 4] Transferring results back to CPU (device->host)--. Transferred 3.81 MB to CPU [Phase 5] GPU-to-GPU memory copy--. Copied 15.26 MB within GPU

Slide 47

Slide 47 text

@doniacld GPU Memory Transfer Tracing Demo gpudemo@DESKTOP-N3HOOD6:~/demo2$ sudo bpftrace cuda_trace2.bt Attaching 7 probes--. GPU MALLOC | PID: 15459 | Process: python3 | Size: 2097152 bytes (2 MB) GPU MEMCPY ASYNC | PID: 15459 | Process: python3 | HOST->GPU | Size: 40000 bytes (0 MB) GPU MALLOC | PID: 15459 | Process: python3 | Size: 20971520 bytes (20 MB) GPU MEMCPY ASYNC | PID: 15459 | Process: python3 | HOST->GPU | Size: 4000000 bytes (3 MB) GPU MEMCPY ASYNC | PID: 15459 | Process: python3 | HOST->GPU | Size: 16000000 bytes (15 MB) GPU MALLOC | PID: 15459 | Process: python3 | Size: 20971520 bytes (20 MB) GPU MEMCPY ASYNC | PID: 15459 | Process: python3 | HOST->GPU | Size: 4000000 bytes (3 MB) GPU MEMCPY ASYNC | PID: 15459 | Process: python3 | GPU->HOST | Size: 4000000 bytes (3 MB) GPU MALLOC | PID: 15459 | Process: python3 | Size: 16777216 bytes (16 MB) GPU MEMCPY ASYNC | PID: 15459 | Process: python3 | GPU->GPU | Size: 16000000 bytes (15 MB)

Slide 48

Slide 48 text

Remediation Guide to Inference Bottlenecks

Slide 49

Slide 49 text

Layer 3ξ‚’ Infrastructure Correlation @doniacld

Slide 50

Slide 50 text

Layer 3ξ‚’ Infrastructure Correlation Traditional GPU telemetry shows: ● utilization ● memory ● throughput Useful… but isolated. @doniacld

Slide 51

Slide 51 text

Traditional GPU telemetry shows: ● utilization ● memory ● throughput Useful… but isolated. Correlated infrastructure signals ● Which pod owns the GPU? ● Which request caused the spike? ● Is the GPU waiting on NCCL? ● Is networking slowing training? ● Is CPU scheduling starving inference? Layer 3ξ‚’ Infrastructure Correlation @doniacld

Slide 52

Slide 52 text

Traditional GPU telemetry shows: ● utilization ● memory ● throughput Useful… but isolated. Correlated infrastructure signals ● Which pod owns the GPU? ● Which request caused the spike? ● Is the GPU waiting on NCCL? ● Is networking slowing training? ● Is CPU scheduling starving inference? eBPF telemetries correlates: ● processes ● containers ● syscalls ● networking ● scheduling ● GPU runtime activity Layer 3ξ‚’ Infrastructure Correlation @doniacld

Slide 53

Slide 53 text

eACGMξ‚’ What is it? eACGM: eBPF-based Automated Comprehensive Governance and Monitoring framework for AI/ML system It correlates GPU metrics with infrastructure signals src: eACGMξ‚’ Non-instrumented Performance Tracing and Anomaly Detection towards Machine Learning Systems (https://arxiv.org/abs/2506.02007) @doniacld

Slide 54

Slide 54 text

eACGMξ‚’ How does it work? @doniacld

Slide 55

Slide 55 text

eACGMξ‚’ How does it work? @doniacld

Slide 56

Slide 56 text

eACGMξ‚’ How does it work? @doniacld

Slide 57

Slide 57 text

eACGMξ‚’ How does it work? @doniacld

Slide 58

Slide 58 text

eACGMξ‚’ Usage # Attach to running training process sudo ./eacgm -p # Or launch new process sudo ./eacgm python train.py # Output: Chrome trace JSON # Perfetto UI: ui.perfetto.dev @doniacld

Slide 59

Slide 59 text

Infrastructure Correlation Remediations Guide @doniacld

Slide 60

Slide 60 text

Future Vision @doniacld

Slide 61

Slide 61 text

eGPUξ‚’ Inject eBPF Into Running GPU Kernels

Slide 62

Slide 62 text

eGPUξ‚’ Inject eBPF Into Running GPU Kernels Injects eBPF-like bytecode directly into GPU kernel execution Technique: PTX Injection ● PTX = NVIDIA's intermediate GPU assembly language ● eGPU translates eBPF β†’ PTX ● Injects instrumentation into running CUDA kernels

Slide 63

Slide 63 text

eGPUξ‚’ Inject eBPF Into Running GPU Kernels Injects eBPF-like bytecode directly into GPU kernel execution Technique: PTX Injection ● PTX = NVIDIA's intermediate GPU assembly language ● eGPU translates eBPF β†’ PTX ● Injects instrumentation into running CUDA kernels Status: πŸ”¬ Research (published ACM HCDS'25) Supports: NVIDIA GPUs (PTX injection) Repo: github.com/eunomia-bpf/bpftime Limitations: ● NVIDIA-specific (PTX is NVIDIA's IL) ● Requires kernel recompilation/JIT injection

Slide 64

Slide 64 text

eGPUξ‚’ Usage sudo ./egpu attach --kernel matmul_kernel --pid 42891 matmul_kernel (grid 256x256, block 32x32) β”œβ”€ Execution time per warp: β”‚ β”œβ”€ Warp 0-23: 8.2ms avg βœ“ β”‚ β”œβ”€ Warp 24-27: 14.8ms avg ⚠ (+80% slower) β”‚ └─ Warp 28-31: 15.1ms avg ⚠ β”‚ β”œβ”€ Branch divergence: β”‚ β”œβ”€ Warp 0-23: 2% divergent branches β”‚ └─ Warp 24-31: 38% divergent branches ⚠ β”‚ β”œβ”€ Memory access: β”‚ β”œβ”€ Coalesced: 73% of loads β”‚ β”œβ”€ Uncoalesced: 27% of loads (3.2x slower) β”‚ └─ Bank conflicts: 142 detected β”‚ β”œβ”€ Register pressure: β”‚ β”œβ”€ Registers per thread: 64 / 64 (maxed out) β”‚ β”œβ”€ Spills to local memory: 18 per thread β”‚ └─ Occupancy limited to: 50% (register bound) β”‚ └─ Root cause: Conditional path at line 89 causes warp 24-31 to execute 2x loop iterations. Registers spilling to VRAM.

Slide 65

Slide 65 text

eGPUξ‚’ Remediation Guide sudo ./egpu attach --kernel matmul_kernel --pid 42891 matmul_kernel (grid 256x256, block 32x32) β”œβ”€ Execution time per warp: β”‚ β”œβ”€ Warp 0-23: 8.2ms avg βœ“ β”‚ β”œβ”€ Warp 24-27: 14.8ms avg ⚠ (+80% slower) β”‚ └─ Warp 28-31: 15.1ms avg ⚠ β”‚ β”œβ”€ Branch divergence: β”‚ β”œβ”€ Warp 0-23: 2% divergent branches β”‚ └─ Warp 24-31: 38% divergent branches ⚠ β”‚ β”œβ”€ Memory access: β”‚ β”œβ”€ Coalesced: 73% of loads β”‚ β”œβ”€ Uncoalesced: 27% of loads (3.2x slower) β”‚ └─ Bank conflicts: 142 detected β”‚ β”œβ”€ Register pressure: β”‚ β”œβ”€ Registers per thread: 64 / 64 (maxed out) β”‚ β”œβ”€ Spills to local memory: 18 per thread β”‚ └─ Occupancy limited to: 50% (register bound) β”‚ └─ Root cause: Conditional path at line 89 causes warp 24-31 to execute 2x loop iterations. Registers spilling to VRAM. Warp β€’ Refactor conditional at line 89 β€’ Move branch outside kernel β€’ OR use predication instead of if/else Uncoalesced loads β€’ Transpose input matrix layout (row-major β†’ col-major) β€’ OR adjust thread β†’ data mapping Register spills β€’ Split kernel into 2 passes β€’ OR compile with --maxrregcount=48 β€’ OR reduce loop unrolling

Slide 66

Slide 66 text

AgentSight: AIξ‚ˆSystem-Wide Observability AgentSight uses eBPF to correlate: ● AI agent activity ● system behavior ● process execution ● network activity ● filesystem access AI Agent ↓ LLM Runtime / APIs ↓ Kernel & System Events ↓ Processes β€’ Files β€’ Network β€’ GPU AI systems increasingly orchestrate: ● tools ● APIs ● shell commands ● infrastructure workflows https://github.com/eunomia-bpf/agentsight AgentSight: System-Level Observability for AI Agents Using eBPF (https://arxiv.org/abs/2508.02736) @doniacld

Slide 67

Slide 67 text

Key Takeaways GPUs are infrastructure now AI workloads introduce new operational failure modes and observability gaps. eBPF observes the boundaries CUDA runtimes, kernel transitions, networking, and orchestration layers are already observable today. Correlation matters more than metrics The challenge is no longer collecting more telemetry β€” it is connecting signals across the AI infrastructure stack. The ecosystem is still early Most GPU observability tooling is emerging, which makes this a great time to experiment, contribute, and shape the ecosystem. @doniacld

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

Thank you and credits What did you think of the talk? πŸ˜ƒ 🧠 σ°·»? Resources ● Isovalent Lab eBPF Getting Started (https://isovalent.com/labs/ebpf-getting-started/) ● Linux Plumbers Conference Talks (https://lpc.events/event/19/program) ● GPUprobe (https://dev.to/ethgraham/snooping-on-your-gpu-using-ebpf-to-build-zero-instrumentation-cuda-monitoring-2hh1) ● eInfer (https://dl.acm.org/doi/10.1145/3748355.3748372) ● eACGM (github.com/shady1543/eACGM) ● eBPF Γ— AI/LLMs: The Convergence of System Observability and Artificial Intelligence, Yusheng Zheng (https://eunomia.dev/GPTtrace/) Go further ● gpu_ext: GPU scheduling (arxiv.org/abs/2512.12615) ● NCCLbpf: +27% AllReduce throughput (arxiv.org/abs/2603.11438) ● eGPU: PTX injection (doi.org/10.1145/3723851.3726984) ● SysOM-AI: GPUs at Alibaba, daysβ†’10min diagnosis (arxiv.org/abs/2603.29235) @doniacld