Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Python Memory Management
Search
Tomek Paczkowski
November 28, 2013
Programming
0
62
Python Memory Management
A simple talk about simple problems with Python memory allocator
Tomek Paczkowski
November 28, 2013
Tweet
Share
More Decks by Tomek Paczkowski
See All by Tomek Paczkowski
Dispelling py.test magic
oinopion
0
800
Python Fest lightning talk
oinopion
0
88
Contributing to Django
oinopion
2
560
Django and Time Zones: New in the Trunk
oinopion
4
210
Other Decks in Programming
See All in Programming
Railsアプリケーションと パフォーマンスチューニング ー 秒間5万リクエストの モバイルオーダーシステムを支える事例 ー Rubyセミナー 大阪
falcon8823
4
1k
Result型で“失敗”を型にするPHPコードの書き方
kajitack
4
570
設計やレビューに悩んでいるPHPerに贈る、クリーンなオブジェクト設計の指針たち
panda_program
6
1.8k
PostgreSQLのRow Level SecurityをPHPのORMで扱う Eloquent vs Doctrine #phpcon #track2
77web
2
470
ペアプロ × 生成AI 現場での実践と課題について / generative-ai-in-pair-programming
codmoninc
0
780
システム成長を止めない!本番無停止テーブル移行の全貌
sakawe_ee
1
160
Rubyでやりたい駆動開発 / Ruby driven development
chobishiba
1
530
20250628_非エンジニアがバイブコーディングしてみた
ponponmikankan
0
630
deno-redisの紹介とJSRパッケージの運用について (toranoana.deno #21)
uki00a
0
170
ソフトウェア品質を数字で捉える技術。事業成長を支えるシステム品質の マネジメント
takuya542
0
790
今ならAmazon ECSのサービス間通信をどう選ぶか / Selection of ECS Interservice Communication 2025
tkikuc
21
3.8k
Composerが「依存解決」のためにどんな工夫をしているか #phpcon
o0h
PRO
1
250
Featured
See All Featured
How to train your dragon (web standard)
notwaldorf
94
6.1k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.9k
Agile that works and the tools we love
rasmusluckow
329
21k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
Side Projects
sachag
455
42k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Faster Mobile Websites
deanohume
307
31k
The World Runs on Bad Software
bkeepers
PRO
69
11k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
720
YesSQL, Process and Tooling at Scale
rocio
173
14k
Code Review Best Practice
trishagee
69
18k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
680
Transcript
Python Memory Management Tomasz Paczkowski Pykonik, 28.11.2013
Case Study • Long lived web process • Periodically allocates
boatloads of memory • Never releases it
Distilled code def main():! big = alloc(100000)! small = alloc(1)!
del big! # memory not released
[Demo time]
Diagnose: Memory Fragmentation big small small big
Solution • Make better use of memory • Subprocess •
jemalloc* via LD_PRELOAD
Fun with Python allocator • Python does not use malloc
directly — too costly for small objects • Instead implements more sophisticated allocator on top of malloc
Pools for integers ints = range(5*1000*1000)! del ints! import gc;
gc.collect(2)
Free lists • For handful of most common types Python
keeps unused objects in so called free lists • Those are most significantly: lists, dictionaries, frames • Speeds up code execution immensely
[Demo time]
Conclusions • Sometimes memory leak is not what it seems
• glibc malloc is not the best of breed • do memory intensive work in subprocess • be mindful when using C extensions
Thanks. Questions?