PyDataLondon2014 Lighting Talk on IPython Memory Usage Analyzer

Transcript

1. www.morconsulting.c Understanding RAM usage in IPython Ian Ozsvald @IanOzsvald MorConsulting.com
2. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 “High Performance Python” • “Practical Performant Programming for Humans” • We needed something lighter...
3. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 @fpedregosa's memory_profiler
4. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 Interactive memory reporting IPython 2.1.0 ­­ An enhanced Interactive Python. In [1]: %run ­i ipython_memory_usage.py In [2]: a=np.ones(1e7) 'a=np.ones(1e7)' used 76.2305 MiB RAM in 0.32s, peaked 0.00 MiB above current, total RAM usage 125.61 MiB In [3]: del a 'del a' used ­76.2031 MiB RAM in 0.10s, peaked 0.00 MiB above current, total RAM usage 49.40 MiB
5. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 Watch hidden temporaries # approx 750MB per matrix In [2]: a=np.ones(1e8); b=np.ones(1e8); c=np.ones(1e8) 'a=np.ones(1e8); b=np.ones(1e8); c=np.ones(1e8)' used 2288.8750 MiB RAM in 1.02s, peaked 0.00 MiB above current, total RAM usage 2338.06 MiB In [3]: d=a*b+c 'd=a*b+c' used 762.9453 MiB RAM in 0.91s, peaked 667.91 MiB above current, total RAM usage 3101.01 MiB
6. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 Understand cache activity In [2]: %run ­i ipython_memory_usage_perf.py In [3]: ones_c = np.ones((1e4,1e4)); v=np.ones(1e4) # C memory layout In [5]: %timeit v*ones_c[:,0] # cache­unfriendly Fortran­ordered layout 1000 loops, best of 3: 211 µs per loop Used 0.1445 MiB RAM in 1.02s, ... perf value for cache­misses averages to 230,318/second In [6]: %timeit v*ones_c[0,:] # cache­friendly C­ordered layout 100000 loops, best of 3: 14.2 µs per loop Used 0.0000 MiB RAM in 5.99s, ... perf value for cache­misses averages to 5,591/second
7. Ian@MorConsulting.com @IanOzsvald PyDataLondon February 2014 Thank You • RAM – watch the envelope, understand e.g. sparse arrays • Caches – get geeky for fine tuning • github.com/ianozsvald/ • ipython_memory_usage • Ian@IanOzsvald.com @IanOzsvald • ModelInsight.io / MorConsulting.com