Share
Embed
Transcript
70 slides
-
Inside Out: Observing Go from Code to Kernel Donia Chaiehloudj,
Software Engineer & Community GopherCon 2026, Seattle © 2024 Isovalent. All Rights Reserved. 1 -
A bit about myself • • • • Software Engineer
& OSS Pollinator 🐝 at Isovalent @Cisco Co-wrote a book to learn Go KCD Provence Organiser (CFP open) 🥐 🥖 Know me for my talk about my henhouse and TinyGo © 2025 Isovalent. All Rights Reserved. 2 -
DON’T WORRY I GOT YOUR BACK © 2025 Isovalent. All
Rights Reserved. 6 -
Observability gives us evidence to investigate a system. © 2025
Isovalent. All Rights Reserved. 7 -
Start where the symptom is visible © 2025 Isovalent. All
Rights Reserved. 8 -
Start where the symptom is visible © 2025 Isovalent. All
Rights Reserved. 9 -
Original code profile, err = fetchProfile(ctx) // ... recommendations, err
= fetchRecmdt(ctx) // ... inventory, err = fetchInventory(ctx) // ... © 2025 Isovalent. All Rights Reserved. 17 -
Code after the fix group, ctx := errgroup.WithContext(ctx) // one
sub request fails -> main request fails group.Go(func() error { var err error profile, err = fetchProfile(ctx) return err }) // add a goroutine to the group group.Go(func() error { var err error recommendations, err = fetchRecmdt(ctx) return err }) group.Go(func() error { var err error inventory, err = fetchInventory(ctx) return err }) if err := group.Wait(); err != nil { return err } © 2025 Isovalent. All Rights Reserved. // wait for the all goroutines 18 -
Narrow down to the Go runtime © 2025 Isovalent. All
Rights Reserved. 23 -
Inside the runtime: example starts workers GET /work Client Backend
Service Worker 200 OK Prometheus instrumentation © 2025 Isovalent. All Rights Reserved. 25 -
pprof code instrumentation import ( "net/http" _ "net/http/pprof" ) go
func() { log.Println(http.ListenAndServe( "localhost:6061", nil, )) }() // add blank import // expose pprof profiles metrics Access pprof metrics go tool pprof http://localhost:6061/debug/pprof/profile © 2025 Isovalent. All Rights Reserved. 28 -
Original code results := make(chan Result) go func() { result
:= doWork(context.Background()) results <- result // leaks: blocks forever }() select { case result := <-results: writeResult(w, result) case <-r.Context().Done(): return } © 2025 Isovalent. All Rights Reserved. 31 -
Code after the fix ctx := r.Context() results := make(chan
Result) go func() { result, err := doWork(ctx) if err != nil { return } // use request context select { case results <- result: case <-ctx.Done(): return } }() // exit when context is cancelled select { case result := <-results: writeResult(w, result) case <-ctx.Done(): return } © 2025 Isovalent. All Rights Reserved. 32 -
Other pprof use cases: retained allocation A tiny slice can
retain a huge array payload := make([]byte, 100<<20) token := payload[:100] cache.Store(key, token) © 2025 Isovalent. All Rights Reserved. // referencing payload slice 35 -
Other pprof use cases: retained allocation A tiny slice can
retain a huge array payload := make([]byte, 100<<20) token := payload[:100] cache.Store(key, token) // referencing payload slice heap profile inuse_space bytes still retained inuse_objects objects still alive © 2025 Isovalent. All Rights Reserved. 36 -
Other pprof use cases: retained allocation A tiny slice can
retain a huge array payload := make([]byte, 100<<20) token := payload[:100] cache.Store(key, token) // referencing payload slice heap profile inuse_space bytes still retained inuse_objects objects still alive Make a copy tokenCopy := append([]byte(nil), token...) © 2025 Isovalent. All Rights Reserved. // copy the slice’s values 37 -
Other pprof use cases: expensive allocation Regexp computation for _,
line := range lines { re := regexp.MustCompile(`...`) matches := re.FindAllString(line, -1) process(matches) } © 2025 Isovalent. All Rights Reserved. 38 -
Other pprof use cases: expensive allocation Regexp computation allocations profile
for _, line := range lines { re := regexp.MustCompile(`...`) matches := re.FindAllString(line, -1) process(matches) } alloc_space total bytes allocated over time alloc_objects total objects allocated over time © 2025 Isovalent. All Rights Reserved. 39 -
Other pprof use cases: expensive allocation Regexp computation allocations profile
for _, line := range lines { re := regexp.MustCompile(`...`) matches := re.FindAllString(line, -1) process(matches) } alloc_space total bytes allocated over time alloc_objects total objects allocated over time Outside of a loop var re = regexp.MustCompile(`...`) for _, line := range lines { matches := re.FindAllString(line, -1) process(matches) } © 2025 Isovalent. All Rights Reserved. 40 -
3. Beyond the process with eBPF © 2024 Isovalent. All
Rights Reserved. 42 -
Beyond the process with eBPF: example POST /compute { “jobs”:16
} Client Backend Service Launching 16 goroutines 200 OK © 2025 Isovalent. All Rights Reserved. 43 -
Request baseline / Demo • • • Baseline average latency
for 2 jobs: 1s Under load latency for 16 jobs: 8s Number of available CPUs: 11 © 2025 Isovalent. All Rights Reserved. 44 -
Where is the CPU going? $ go tool pprof cpu.pprof
© 2025 Isovalent. All Rights Reserved. 46 -
Where is the CPU going? $ go tool pprof cpu.pprof
Duration: 15.11s, Total samples = 3.52s (23.29%) Only 23,29% of the CPU is in use during the 15s of the requests by this application. We have an Off-CPU problem. © 2025 Isovalent. All Rights Reserved. 49 -
What is ? Makes the Linux kernel programmable @doniacld ©
2024 Isovalent. All Rights Reserved. 51 -
Overview of Userspace App Events: • File is opening •
Network packet is received syscalls Linux Kernel @doniacld © 2024 Isovalent. All Rights Reserved. event Program 52 -
Off-CPU flamegraph of the service ℹHow to read Each box
= time spent waiting on CPU © 2025 Isovalent. All Rights Reserved. 60 -
CPU Configuration and Throttling Requests started Requests completed Quota Exceeded
T#10 … (20ms) T#10 … (20ms) T#10. (10ms) … … … T#3 … (20ms) Threads Throttled T#3 … (20ms) Threads Throttled T#3. (10ms) T#2 … (20ms) Idle for 80ms … waiting next period T#2 … (20ms) Idle for 80m … waiting next period T#2. (10ms) T#1 … (20ms) 0 20 T#1 … (20ms) T#1. (10ms) 100 200 Time (ms) src: https://medium.com/@alexandru.lazarev/cpu-limits-in-kubernetes-why-your-pod-is-idle-but-still-throttled-a-deep-dive-into-what-really-136c0cdd62ff © 2025 Isovalent. All Rights Reserved. 61 -
Allocate CPU intentionally # Before services: app: cpus: "0.25" ©
2025 Isovalent. All Rights Reserved. 62 -
Allocate CPU intentionally # Before services: app: cpus: "0.25" ©
2025 Isovalent. All Rights Reserved. # After services: app: cpus: "2.0" 63 -
Other eBPF Observability use cases • Network issues • Disk/IO
issues © 2025 Isovalent. All Rights Reserved. 67 -
You may already be using eBPF © 2025 Isovalent. All
Rights Reserved. 68 -
Takeaways t? re? 🔎 pprof } ? © 2025 Isovalent.
All Rights Reserved. 69 -
Thanks! 🧪 Demo Repository: github.com/doniacld/gophercon26-obs-code2kernel 📘Resources: • Profiling Go programs
with pprof: https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/ • eBPF docs: https://docs.ebpf.io/ • off-CPU analysis blog post: https://www.brendangregg.com/offcpuanalysis.html 🔗 Want to connect? Donia Chaiehloudj / @doniacld © 2025 Isovalent. All Rights Reserved. 70