Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Python Memory Management
Search
Tomek Paczkowski
November 28, 2013
Programming
0
67
Python Memory Management
A simple talk about simple problems with Python memory allocator
Tomek Paczkowski
November 28, 2013
Tweet
Share
More Decks by Tomek Paczkowski
See All by Tomek Paczkowski
Dispelling py.test magic
oinopion
0
840
Python Fest lightning talk
oinopion
0
90
Contributing to Django
oinopion
2
560
Django and Time Zones: New in the Trunk
oinopion
4
210
Other Decks in Programming
See All in Programming
FindyにおけるTakumi活用と脆弱性管理のこれから
rvirus0817
0
480
ソフトウェアテスト徹底指南書の紹介
goyoki
1
150
[FEConf 2025] 모노레포 절망편, 14개 레포로 부활하기까지 걸린 1년
mmmaxkim
0
1.6k
go test -json そして testing.T.Attr / Kyoto.go #63
utgwkk
3
280
OSS開発者という働き方
andpad
5
1.7k
Go言語での実装を通して学ぶLLMファインチューニングの仕組み / fukuokago22-llm-peft
monochromegane
0
120
Updates on MLS on Ruby (and maybe more)
sylph01
1
180
CJK and Unicode From a PHP Committer
youkidearitai
PRO
0
110
Flutter with Dart MCP: All You Need - 박제창 2025 I/O Extended Busan
itsmedreamwalker
0
150
Oracle Database Technology Night 92 Database Connection control FAN-AC
oracle4engineer
PRO
1
440
1から理解するWeb Push
dora1998
7
1.8k
AWS発のAIエディタKiroを使ってみた
iriikeita
1
180
Featured
See All Featured
Done Done
chrislema
185
16k
Measuring & Analyzing Core Web Vitals
bluesmoon
9
580
Optimizing for Happiness
mojombo
379
70k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
Reflections from 52 weeks, 52 projects
jeffersonlam
352
21k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
The Power of CSS Pseudo Elements
geoffreycrofte
77
6k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Building Applications with DynamoDB
mza
96
6.6k
Thoughts on Productivity
jonyablonski
70
4.8k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Transcript
Python Memory Management Tomasz Paczkowski Pykonik, 28.11.2013
Case Study • Long lived web process • Periodically allocates
boatloads of memory • Never releases it
Distilled code def main():! big = alloc(100000)! small = alloc(1)!
del big! # memory not released
[Demo time]
Diagnose: Memory Fragmentation big small small big
Solution • Make better use of memory • Subprocess •
jemalloc* via LD_PRELOAD
Fun with Python allocator • Python does not use malloc
directly — too costly for small objects • Instead implements more sophisticated allocator on top of malloc
Pools for integers ints = range(5*1000*1000)! del ints! import gc;
gc.collect(2)
Free lists • For handful of most common types Python
keeps unused objects in so called free lists • Those are most significantly: lists, dictionaries, frames • Speeds up code execution immensely
[Demo time]
Conclusions • Sometimes memory leak is not what it seems
• glibc malloc is not the best of breed • do memory intensive work in subprocess • be mindful when using C extensions
Thanks. Questions?