Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Python Memory Management
Search
Tomek Paczkowski
November 28, 2013
Programming
0
62
Python Memory Management
A simple talk about simple problems with Python memory allocator
Tomek Paczkowski
November 28, 2013
Tweet
Share
More Decks by Tomek Paczkowski
See All by Tomek Paczkowski
Dispelling py.test magic
oinopion
0
800
Python Fest lightning talk
oinopion
0
88
Contributing to Django
oinopion
2
560
Django and Time Zones: New in the Trunk
oinopion
4
210
Other Decks in Programming
See All in Programming
効率的な開発手段として VRTを活用する
ishkawa
0
130
Node-RED を(HTTP で)つなげる MCP サーバーを作ってみた
highu
0
120
A2A プロトコルを試してみる
azukiazusa1
2
1.3k
“いい感じ“な定量評価を求めて - Four Keysとアウトカムの間の探求 -
nealle
1
9k
AIともっと楽するE2Eテスト
myohei
0
310
Modern Angular with Signals and Signal Store:New Rules for Your Architecture @enterJS Advanced Angular Day 2025
manfredsteyer
PRO
0
210
技術同人誌をMCP Serverにしてみた
74th
1
630
AIと”コードの評価関数”を共有する / Share the "code evaluation function" with AI
euglena1215
1
150
LT 2025-06-30: プロダクトエンジニアの役割
yamamotok
0
730
「テストは愚直&&網羅的に書くほどよい」という誤解 / Test Smarter, Not Harder
munetoshi
0
150
たった 1 枚の PHP ファイルで実装する MCP サーバ / MCP Server with Vanilla PHP
okashoi
1
230
VS Code Update for GitHub Copilot
74th
2
630
Featured
See All Featured
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Statistics for Hackers
jakevdp
799
220k
How STYLIGHT went responsive
nonsquared
100
5.6k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
It's Worth the Effort
3n
185
28k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
Fireside Chat
paigeccino
37
3.5k
Optimizing for Happiness
mojombo
379
70k
Speed Design
sergeychernyshev
32
1k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Rails Girls Zürich Keynote
gr2m
94
14k
Transcript
Python Memory Management Tomasz Paczkowski Pykonik, 28.11.2013
Case Study • Long lived web process • Periodically allocates
boatloads of memory • Never releases it
Distilled code def main():! big = alloc(100000)! small = alloc(1)!
del big! # memory not released
[Demo time]
Diagnose: Memory Fragmentation big small small big
Solution • Make better use of memory • Subprocess •
jemalloc* via LD_PRELOAD
Fun with Python allocator • Python does not use malloc
directly — too costly for small objects • Instead implements more sophisticated allocator on top of malloc
Pools for integers ints = range(5*1000*1000)! del ints! import gc;
gc.collect(2)
Free lists • For handful of most common types Python
keeps unused objects in so called free lists • Those are most significantly: lists, dictionaries, frames • Speeds up code execution immensely
[Demo time]
Conclusions • Sometimes memory leak is not what it seems
• glibc malloc is not the best of breed • do memory intensive work in subprocess • be mindful when using C extensions
Thanks. Questions?