Finding memory leaks in Go programs

GoSF Meetup 07/23/2014

Transcript

1. Finding memory leaks in Go programs Oleg Shaldybin July 23, 2014 GoSF Meetup
2. About • Software Engineer at Apcera • Worked at VMware on Cloud Foundry and large-scale VM management solution • Founding member of CF BOSH • @olegshaldybin • oleg@apcera.com
3. Memory leaks in Go? • Garbage-collected language • Good idioms for resource management (defers) • Still prone to memory leaks (just like Java, Ruby, or pretty much any language out there) • Anything that’s still reachable by Go program won’t be GC’ed • Defers can be mis-used, acquired resources might not be released properly • Possible to use C with cgo, need to be careful when allocating memory from C-land
4. Leak detection • Code reviews and overall code hygiene help but some things are not obvious and/or hard to track. • Measuring and understanding memory consumption is important. • pprof: sampling profiler, comes with standard library, easy to use, helps a lot. • When leaks are detected, it’s usually a process running in some real environment, need to be careful not to break it, limited control over execution.
5. Example: misused defer • defer has a function scope • deferred function will be kept around until original function returns • reference to tracker kept forever • quite easy to avoid by wrapping into a function or not using defer
6. Example: misused defer • defer has a function scope • reference to tracker kept forever • quite easy to avoid by wrapping into a function or not using defer
7. Using pprof • Instrument your program:
 
 import  _  “net/http/pprof”
 http.ListenAndServe(":8080",  nil)   • pprof is a sampling profiler. Only tracks a small percentage of allocations and approximates real counts. OK to use in production.
8. Using pprof • You can profile a running process without affecting its behavior. • Find out if your process already has profiling endpoint exposed:
 
 >  lsof  -­‐Pp  $(pidof  myprocess)  |  grep  LISTEN   • Dive in with ‘go tool pprof’:
 
 >  go  tool  pprof  http://myhost:8080/debug/pprof/heap   • Or save heap profile locally so you can compare it later:
 
 >  curl  -­‐s  http://myhost:8080/debug/pprof/heap  >  base.heap
9. Heap profiling modes • -base /path/to/heap.profile allows to compare current profile with some base (useful to see what allocations are made over time). • Use -inuse_space mode when you want to display an amount of memory in use. • Use -inuse_objects mode to display a number of objects in use.
10. >  go  tool  pprof  \          -­‐base  base.heap  \          /path/to/your/binary  \            /tmp/current.heap
11. >  go  tool  pprof  -­‐inuse_objects  \          -­‐base  base.heap  \          /path/to/your/binary  \            /tmp/current.heap
12. >  top      #  Show  where  most  allocations  happened   >  list    #  Show  annotated  source   >  web      #  Display  profile  graph  in  a  browser
14. Lots of defers get accumulated Someone is holding onto *http.Request?
15. Example: pub-sub • request pattern using pub-sub with NATS
17. What’s allocated here?
18. It’s a Subscription struct: we never unsubscribe, so they keep accumulating.
22. All allocated objects (including GC’d) Objects accumulated since our base heap profile was taken, still in use.
23. Example: cgo • using C standard library directly from Go • explicit memory allocation and no GC to help us free this memory
24. Example: cgo • using C standard library directly from Go • explicit memory allocation and no GC to help us free this memory
25. Example: cgo • pprof can’t help • Valgrind can’t help • defer is useful • better to avoid making caller care about freeing memory • use helpers to copy values into Go types before returning them (e.g. C.GoString, C.GoBytes)
26. Profiling can help in a lot of ways • Not just memory leaks: heap profiling also helps you understand what’s going on with memory allocations in your program and see where most of them happen, providing an opportunity for optimization. • Not just heap profiling: CPU profiling is also possible and works pretty much the same way (go tool pprof). See http://blog.golang.org/ profiling-go-programs for details. • Most non-trivial programs are hard to reason about, so measuring and profiling are a way to go, otherwise a lot of time will be spent optimizing something that’s not really a bottleneck. • Heap inspector: it’s useful to understand what is actually on the heap at any point in time. Might be possible with WriteHeapDump (added in 1.3).
27. Also important: Go Memory Model • Important to understand in order to avoid hard-to-debug situations in concurrent programs. • A simple set of rules defining “happens-before” and “happens-after” relationship between reads and writes in different goroutines. • Usually not a source of memory leaks: it is possible however that code you’re counting on to run at certain point won’t run because it can’t observe an effect in a different goroutine required to trigger a certain code path. • http://golang.org/ref/mem
28. Questions?