How To Find Memory Leaks in GAUDI

1. Memory Leaks How to find In GAUDI Applications Sasha Mazurov bug #93508: Moore Memory leak
2. Tools MemoryAuditor Google__HeapProfiler Google__HeapCheker Auditors
3. Valgrind? No Auditor == No Control • Could not skip an initialization/finalization phase • Could not skip N events and stop after M events • Slower than Google tools • Google__HeapChecker do the same
4. Step 1: MemoryAuditor from Configurables import MemoryAuditor Moore().EnableAuditor = [ MemoryAuditor() ] EventSelector().PrintFreq = 1 SUCCESS Reading Event record 75698. Record number within stream 1: 75698 INFO Memory usage has changed after Velo2CandidatesVertexDisplVertex Execute virtual size = 1572.61 MB resident set size = 1209.75 MB INFO Memory usage has changed after Hlt1VertexDisplVertexStreamer Execute virtual size = 2445.82 MB resident set size = 2078.48 MB Detect suspicious events
5. Step 2: Google__HeapProfiler In which function the leak happens? from Configurables import Google__HeapProfiler Moore().SkipEvents = 75697 auditor = Google__HeapProfiler() auditor.DoFullEventProfile = True auditor.FullEventNSampleEvents = 3 Moore().EnableAuditor = [ auditor ] $> LD_PRELOAD=/path/to/libtcmalloc_and_profiler.so gaudirun.py moore.py
6. Profiler output: Dump files FULL-Events1To6001.0001.heap FULL-Events1To6001.0002.heap FULL-Events1To6001.0003.heap FULL-Events1To6001.0004.heap ... Dump the memory usage data each 1Gb memory allocation/deallocation cycle.
7. Analyze I: Text output (1) $> pprof –text `which python` FULL-Events1To6001.0001.heap (2) $> pprof –text `which python` FULL-Events1To6001.0005.heap boost:pool::malloc_need_resize use 577.0 MB of 598.4 MB total in use boost:pool::malloc_need_resize use 698.1 MB of 750 MB total in use
9. Analyze II: Call chain $> pprof –gv `which python` FULL-Events1To6001.0001.heap
10. Leak in UpgradeVertices Bonus: the fix reduced a CPU consumption
11. Links http://code.google.com/p/gperftools/ Gaudi / GaudiProfiling / GoogleAuditor.cpp $LCG_external_area/tcmalloc/1.7p1 http://amazurov.ru/cern/mooremem/
12. Thanks to Vava, Marco, Chris, Victor, Vanya, Patrick, Vanya, Gerhard, Markus