Python Memory Management

Transcript

1. Python Memory Management Tomasz Paczkowski Pykonik, 28.11.2013

2. Case Study • Long lived web process • Periodically allocates

   boatloads of memory • Never releases it

3. Distilled code def main():! big = alloc(100000)! small = alloc(1)!

   del big! # memory not released

4. [Demo time]

5. Diagnose:
 Memory Fragmentation big small small big

6. Solution • Make better use of memory • Subprocess •

   jemalloc* via LD_PRELOAD

7. Fun with Python allocator • Python does not use malloc

   directly — too costly for small objects • Instead implements more sophisticated allocator on top of malloc

8. Pools for integers ints = range(5*1000*1000)! del ints! import gc;

   gc.collect(2)

9. Free lists • For handful of most common types Python

   keeps unused objects in so called free lists • Those are most significantly: lists, dictionaries, frames • Speeds up code execution immensely

10. [Demo time]

11. Conclusions • Sometimes memory leak is not what it seems

    • glibc malloc is not the best of breed • do memory intensive work in subprocess • be mindful when using C extensions

12. Thanks. Questions?