Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Python Memory Management
Search
Tomek Paczkowski
November 28, 2013
Programming
0
58
Python Memory Management
A simple talk about simple problems with Python memory allocator
Tomek Paczkowski
November 28, 2013
Tweet
Share
More Decks by Tomek Paczkowski
See All by Tomek Paczkowski
Dispelling py.test magic
oinopion
0
750
Python Fest lightning talk
oinopion
0
88
Contributing to Django
oinopion
2
530
Django and Time Zones: New in the Trunk
oinopion
4
210
Other Decks in Programming
See All in Programming
DevinとCursorから学ぶAIエージェントメモリーの設計とMoatの考え方
itarutomy
1
450
Amazon Bedrock Multi Agentsを試してきた
tm2
1
180
AWS Lambda functions with C# 用の Dev Container Template を作ってみた件
mappie_kochi
0
210
サーバーゆる勉強会 DBMS の仕組み編
kj455
1
330
.NETでOBS Studio操作してみたけど…… / Operating OBS Studio by .NET
skasweb
0
130
オニオンアーキテクチャを使って、 Unityと.NETでコードを共有する
soi013
0
380
チームの立て直し施策をGoogleの 『効果的なチーム』と見比べてみた
maroon8021
0
160
月刊 競技プログラミングをお仕事に役立てるには
terryu16
1
1.2k
ISUCON14感想戦で85万点まで頑張ってみた
ponyo877
1
770
為你自己學 Python
eddie
0
530
PHPUnitしか使ってこなかった 一般PHPerがPestに乗り換えた実録
mashirou1234
0
450
asdf-ecspresso作って 友達が増えた話 / Fujiwara Tech Conference 2025
koluku
0
1.5k
Featured
See All Featured
RailsConf 2023
tenderlove
29
980
Why Our Code Smells
bkeepers
PRO
335
57k
Statistics for Hackers
jakevdp
797
220k
Documentation Writing (for coders)
carmenintech
67
4.6k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
28
2.2k
We Have a Design System, Now What?
morganepeng
51
7.4k
Designing Experiences People Love
moore
139
23k
Six Lessons from altMBA
skipperchong
27
3.6k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
1.3k
StorybookのUI Testing Handbookを読んだ
zakiyama
28
5.4k
Designing on Purpose - Digital PM Summit 2013
jponch
117
7.1k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
Transcript
Python Memory Management Tomasz Paczkowski Pykonik, 28.11.2013
Case Study • Long lived web process • Periodically allocates
boatloads of memory • Never releases it
Distilled code def main():! big = alloc(100000)! small = alloc(1)!
del big! # memory not released
[Demo time]
Diagnose: Memory Fragmentation big small small big
Solution • Make better use of memory • Subprocess •
jemalloc* via LD_PRELOAD
Fun with Python allocator • Python does not use malloc
directly — too costly for small objects • Instead implements more sophisticated allocator on top of malloc
Pools for integers ints = range(5*1000*1000)! del ints! import gc;
gc.collect(2)
Free lists • For handful of most common types Python
keeps unused objects in so called free lists • Those are most significantly: lists, dictionaries, frames • Speeds up code execution immensely
[Demo time]
Conclusions • Sometimes memory leak is not what it seems
• glibc malloc is not the best of breed • do memory intensive work in subprocess • be mindful when using C extensions
Thanks. Questions?