Python Memory
Management
Tomasz Paczkowski
Pykonik, 28.11.2013
Slide 2
Slide 2 text
Case Study
• Long lived web process
• Periodically allocates boatloads of memory
• Never releases it
Slide 3
Slide 3 text
Distilled code
def main():!
big = alloc(100000)!
small = alloc(1)!
del big!
# memory not released
Slide 4
Slide 4 text
[Demo time]
Slide 5
Slide 5 text
Diagnose:
Memory Fragmentation
big
small
small
big
Slide 6
Slide 6 text
Solution
• Make better use of memory
• Subprocess
• jemalloc* via LD_PRELOAD
Slide 7
Slide 7 text
Fun with Python allocator
• Python does not use malloc directly — too costly
for small objects
• Instead implements more sophisticated allocator
on top of malloc
Slide 8
Slide 8 text
Pools for integers
ints = range(5*1000*1000)!
del ints!
import gc; gc.collect(2)
Slide 9
Slide 9 text
Free lists
• For handful of most common types Python keeps
unused objects in so called free lists
• Those are most significantly: lists, dictionaries,
frames
• Speeds up code execution immensely
Slide 10
Slide 10 text
[Demo time]
Slide 11
Slide 11 text
Conclusions
• Sometimes memory leak is not what it seems
• glibc malloc is not the best of breed
• do memory intensive work in subprocess
• be mindful when using C extensions