Python Memory Management

Python Memory Management

A simple talk about simple problems with Python memory allocator


Tomek Paczkowski

November 28, 2013


  1. Python Memory Management Tomasz Paczkowski Pykonik, 28.11.2013

  2. Case Study • Long lived web process • Periodically allocates

    boatloads of memory • Never releases it
  3. Distilled code def main():! big = alloc(100000)! small = alloc(1)!

    del big! # memory not released
  4. [Demo time]

  5. Diagnose:
 Memory Fragmentation big small small big

  6. Solution • Make better use of memory • Subprocess •

    jemalloc* via LD_PRELOAD
  7. Fun with Python allocator • Python does not use malloc

    directly — too costly for small objects • Instead implements more sophisticated allocator on top of malloc
  8. Pools for integers ints = range(5*1000*1000)! del ints! import gc;

  9. Free lists • For handful of most common types Python

    keeps unused objects in so called free lists • Those are most significantly: lists, dictionaries, frames • Speeds up code execution immensely
  10. [Demo time]

  11. Conclusions • Sometimes memory leak is not what it seems

    • glibc malloc is not the best of breed • do memory intensive work in subprocess • be mindful when using C extensions
  12. Thanks. Questions?