Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PyDataLondon2014 Lighting Talk on IPython Memory Usage Analyzer

PyDataLondon2014 Lighting Talk on IPython Memory Usage Analyzer

3d644406158b4d440111903db1f62622?s=128

ianozsvald

August 08, 2014
Tweet

Transcript

  1. www.morconsulting.c Understanding RAM usage in IPython Ian Ozsvald @IanOzsvald MorConsulting.com

  2. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 “High Performance Python” • “Practical

    Performant Programming for Humans” • We needed something lighter...
  3. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 @fpedregosa's memory_profiler

  4. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 Interactive memory reporting IPython 2.1.0

    ­­ An enhanced Interactive Python. In [1]: %run ­i ipython_memory_usage.py In [2]: a=np.ones(1e7) 'a=np.ones(1e7)' used 76.2305 MiB RAM in 0.32s, peaked 0.00 MiB above current, total RAM usage 125.61 MiB In [3]: del a 'del a' used ­76.2031 MiB RAM in 0.10s, peaked 0.00 MiB above current, total RAM usage 49.40 MiB
  5. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 Watch hidden temporaries # approx

    750MB per matrix In [2]: a=np.ones(1e8); b=np.ones(1e8); c=np.ones(1e8) 'a=np.ones(1e8); b=np.ones(1e8); c=np.ones(1e8)' used 2288.8750 MiB RAM in 1.02s, peaked 0.00 MiB above current, total RAM usage 2338.06 MiB In [3]: d=a*b+c 'd=a*b+c' used 762.9453 MiB RAM in 0.91s, peaked 667.91 MiB above current, total RAM usage 3101.01 MiB
  6. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 Understand cache activity In [2]:

    %run ­i ipython_memory_usage_perf.py In [3]: ones_c = np.ones((1e4,1e4)); v=np.ones(1e4) # C memory layout In [5]: %timeit v*ones_c[:,0] # cache­unfriendly Fortran­ordered layout 1000 loops, best of 3: 211 µs per loop Used 0.1445 MiB RAM in 1.02s, ... perf value for cache­misses averages to 230,318/second In [6]: %timeit v*ones_c[0,:] # cache­friendly C­ordered layout 100000 loops, best of 3: 14.2 µs per loop Used 0.0000 MiB RAM in 5.99s, ... perf value for cache­misses averages to 5,591/second
  7. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 Thank You • RAM –

    watch the envelope, understand e.g. sparse arrays • Caches – get geeky for fine tuning • github.com/ianozsvald/ • ipython_memory_usage • Ian@IanOzsvald.com @IanOzsvald • ModelInsight.io / MorConsulting.com