Python Memory Management

A simple talk about simple problems with Python memory allocator

Transcript

1. Python Memory Management Tomasz Paczkowski Pykonik, 28.11.2013
2. Case Study • Long lived web process • Periodically allocates boatloads of memory • Never releases it
3. Distilled code def main():! big = alloc(100000)! small = alloc(1)! del big! # memory not released
4. [Demo time]
5. Diagnose:
 Memory Fragmentation big small small big
6. Solution • Make better use of memory • Subprocess • jemalloc* via LD_PRELOAD
7. Fun with Python allocator • Python does not use malloc directly — too costly for small objects • Instead implements more sophisticated allocator on top of malloc
8. Pools for integers ints = range(5*1000*1000)! del ints! import gc; gc.collect(2)
9. Free lists • For handful of most common types Python keeps unused objects in so called free lists • Those are most significantly: lists, dictionaries, frames • Speeds up code execution immensely
10. [Demo time]
11. Conclusions • Sometimes memory leak is not what it seems • glibc malloc is not the best of breed • do memory intensive work in subprocess • be mindful when using C extensions
12. Thanks. Questions?