[GopherCon 2026] Inside Out: Observing Go from Code to Kernel

Inside Out: Observing Go from Code to Kernel Donia Chaiehloudj,
Software Engineer & Community GopherCon 2026, Seattle © 2024 Isovalent. All Rights Reserved. 1

A bit about myself • • • • Software Engineer
& OSS Pollinator 🐝 at Isovalent @Cisco Co-wrote a book to learn Go KCD Provence Organiser (CFP open) 󰏃 🥐 🥖 Know me for my talk about my henhouse and TinyGo © 2025 Isovalent. All Rights Reserved. 2

Sometimes… something goes wrong © 2025 Isovalent. All Rights Reserved.
4

Letʼs fix the issue! © 2025 Isovalent. All Rights Reserved.
5

DON’T WORRY I GOT YOUR BACK © 2025 Isovalent. All
Rights Reserved. 6

Start where the symptom is visible © 2025 Isovalent. All
Rights Reserved. 8

Start where the symptom is visible © 2025 Isovalent. All
Rights Reserved. 9

Pillars of Observability metrics © 2025 Isovalent. All Rights Reserved.
logs traces 11

Pillars of Observability metrics © 2025 Isovalent. All Rights Reserved.
logs traces 12

Original code profile, err = fetchProfile(ctx) // ... recommendations, err
= fetchRecmdt(ctx) // ... inventory, err = fetchInventory(ctx) // ... © 2025 Isovalent. All Rights Reserved. 17

Code after the fix group, ctx := errgroup.WithContext(ctx) // one
sub request fails -> main request fails group.Go(func() error { var err error profile, err = fetchProfile(ctx) return err }) // add a goroutine to the group group.Go(func() error { var err error recommendations, err = fetchRecmdt(ctx) return err }) group.Go(func() error { var err error inventory, err = fetchInventory(ctx) return err }) if err := group.Wait(); err != nil { return err } © 2025 Isovalent. All Rights Reserved. // wait for the all goroutines 18

Missing cancellation goroutine still running © 2025 Isovalent. All Rights
Reserved. 21

Narrow down to the Go runtime © 2025 Isovalent. All
Rights Reserved. 23

2. Inside the Go Runtime © 2024 Isovalent. All Rights
Reserved. 24

Inside the runtime: example starts workers GET /work Client Backend
Service Worker 200 OK Prometheus instrumentation © 2025 Isovalent. All Rights Reserved. 25

pprof code instrumentation import ( "net/http" _ "net/http/pprof" ) go
func() { log.Println(http.ListenAndServe( "localhost:6061", nil, )) }() // add blank import // expose pprof profiles metrics Access pprof metrics go tool pprof http://localhost:6061/debug/pprof/profile © 2025 Isovalent. All Rights Reserved. 28

Original code results := make(chan Result) go func() { result
:= doWork(context.Background()) results <- result // leaks: blocks forever }() select { case result := <-results: writeResult(w, result) case <-r.Context().Done(): return } © 2025 Isovalent. All Rights Reserved. 31

Code after the fix ctx := r.Context() results := make(chan
Result) go func() { result, err := doWork(ctx) if err != nil { return } // use request context select { case results <- result: case <-ctx.Done(): return } }() // exit when context is cancelled select { case result := <-results: writeResult(w, result) case <-ctx.Done(): return } © 2025 Isovalent. All Rights Reserved. 32

Other pprof use cases: retained allocation A tiny slice can
retain a huge array payload := make([]byte, 100<<20) token := payload[:100] cache.Store(key, token) © 2025 Isovalent. All Rights Reserved. // referencing payload slice 35

retain a huge array payload := make([]byte, 100<<20) token := payload[:100] cache.Store(key, token) // referencing payload slice heap profile inuse_space bytes still retained inuse_objects objects still alive © 2025 Isovalent. All Rights Reserved. 36

retain a huge array payload := make([]byte, 100<<20) token := payload[:100] cache.Store(key, token) // referencing payload slice heap profile inuse_space bytes still retained inuse_objects objects still alive Make a copy tokenCopy := append([]byte(nil), token...) © 2025 Isovalent. All Rights Reserved. // copy the slice’s values 37

Other pprof use cases: expensive allocation Regexp computation for _,
line := range lines { re := regexp.MustCompile(`...`) matches := re.FindAllString(line, -1) process(matches) } © 2025 Isovalent. All Rights Reserved. 38

Other pprof use cases: expensive allocation Regexp computation allocations profile
for _, line := range lines { re := regexp.MustCompile(`...`) matches := re.FindAllString(line, -1) process(matches) } alloc_space total bytes allocated over time alloc_objects total objects allocated over time © 2025 Isovalent. All Rights Reserved. 39

Other pprof use cases: expensive allocation Regexp computation allocations profile
for _, line := range lines { re := regexp.MustCompile(`...`) matches := re.FindAllString(line, -1) process(matches) } alloc_space total bytes allocated over time alloc_objects total objects allocated over time Outside of a loop var re = regexp.MustCompile(`...`) for _, line := range lines { matches := re.FindAllString(line, -1) process(matches) } © 2025 Isovalent. All Rights Reserved. 40

3. Beyond the process with eBPF © 2024 Isovalent. All
Rights Reserved. 42

Beyond the process with eBPF: example POST /compute { “jobs”:16
} Client Backend Service Launching 16 goroutines 200 OK © 2025 Isovalent. All Rights Reserved. 43

Request baseline / Demo • • • Baseline average latency
for 2 jobs: 1s Under load latency for 16 jobs: 8s Number of available CPUs: 11 © 2025 Isovalent. All Rights Reserved. 44

Where is the CPU going? $ go tool pprof cpu.pprof

Where is the CPU going? $ go tool pprof cpu.pprof
Duration: 15.11s, Total samples = 3.52s (23.29%) Only 23,29% of the CPU is in use during the 15s of the requests by this application. We have an Off-CPU problem. © 2025 Isovalent. All Rights Reserved. 49

CPU Configuration and Throttling Requests started Requests completed Quota Exceeded
T#10 … (20ms) T#10 … (20ms) T#10. (10ms) … … … T#3 … (20ms) Threads Throttled T#3 … (20ms) Threads Throttled T#3. (10ms) T#2 … (20ms) Idle for 80ms … waiting next period T#2 … (20ms) Idle for 80m … waiting next period T#2. (10ms) T#1 … (20ms) 0 20 T#1 … (20ms) T#1. (10ms) 100 200 Time (ms) src: https://medium.com/@alexandru.lazarev/cpu-limits-in-kubernetes-why-your-pod-is-idle-but-still-throttled-a-deep-dive-into-what-really-136c0cdd62ff © 2025 Isovalent. All Rights Reserved. 61

Thanks! 🧪 Demo Repository: github.com/doniacld/gophercon26-obs-code2kernel 📘Resources: • Profiling Go programs
with pprof: https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/ • eBPF docs: https://docs.ebpf.io/ • off-CPU analysis blog post: https://www.brendangregg.com/offcpuanalysis.html 🔗 Want to connect? Donia Chaiehloudj / @doniacld © 2025 Isovalent. All Rights Reserved. 70

[GopherCon 2026] Inside Out: Observing Go from...

[GopherCon 2026] Inside Out: Observing Go from Code to Kernel

More Decks by Donia Chaiehloudj

Other Decks in Technology

Featured

Transcript