Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Python Memory Management
Search
Tomek Paczkowski
November 28, 2013
Programming
0
52
Python Memory Management
A simple talk about simple problems with Python memory allocator
Tomek Paczkowski
November 28, 2013
Tweet
Share
More Decks by Tomek Paczkowski
See All by Tomek Paczkowski
Dispelling py.test magic
oinopion
0
670
Python Fest lightning talk
oinopion
0
81
Contributing to Django
oinopion
2
490
Django and Time Zones: New in the Trunk
oinopion
4
200
Other Decks in Programming
See All in Programming
Findy - エンジニア向け会社紹介 / Findy Letter for Engineers
findyinc
2
81k
CSC307 Lecture 13
javiergs
PRO
0
150
HMSコンペ 11th Solution (team : kansai-kaggler)
t88
1
680
小さな開発会社を作った理由
polidog
0
1.9k
Rust.Nagoya #1
codemountains
0
170
20240706_CDKConf
takuyay0ne
0
1.2k
Temporalを取り巻く仕様を整理する
sajikix
0
120
しくじり先生 Image Matching Challenge 2024 編
goosehaaan
0
810
Introduction of Happy Eyeballs Version 2 (RFC8305) to the Socket library
coe401_
1
220
CSC307 Lecture 08
javiergs
PRO
0
330
iOSアプリでクリップボードにコピーしたことをユーザーに伝えるちょうど良いフィードバックを探す
ski
0
100
みんなのオブザーバビリティプラットフォームを作ってるんだがパフォーマンスがやばい #mackerelio #srenext
ne_sachirou
0
380
Featured
See All Featured
Visualization
eitanlees
139
14k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
12
3.8k
How to Ace a Technical Interview
jacobian
274
23k
Building Flexible Design Systems
yeseniaperezcruz
323
37k
How GitHub (no longer) Works
holman
305
140k
Thoughts on Productivity
jonyablonski
64
4.1k
For a Future-Friendly Web
brad_frost
173
9.2k
Teambox: Starting and Learning
jrom
130
8.6k
The Illustrated Children's Guide to Kubernetes
chrisshort
39
47k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
226
52k
Bootstrapping a Software Product
garrettdimon
PRO
304
110k
Optimizing for Happiness
mojombo
373
69k
Transcript
Python Memory Management Tomasz Paczkowski Pykonik, 28.11.2013
Case Study • Long lived web process • Periodically allocates
boatloads of memory • Never releases it
Distilled code def main():! big = alloc(100000)! small = alloc(1)!
del big! # memory not released
[Demo time]
Diagnose: Memory Fragmentation big small small big
Solution • Make better use of memory • Subprocess •
jemalloc* via LD_PRELOAD
Fun with Python allocator • Python does not use malloc
directly — too costly for small objects • Instead implements more sophisticated allocator on top of malloc
Pools for integers ints = range(5*1000*1000)! del ints! import gc;
gc.collect(2)
Free lists • For handful of most common types Python
keeps unused objects in so called free lists • Those are most significantly: lists, dictionaries, frames • Speeds up code execution immensely
[Demo time]
Conclusions • Sometimes memory leak is not what it seems
• glibc malloc is not the best of breed • do memory intensive work in subprocess • be mindful when using C extensions
Thanks. Questions?