Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Python Memory Management
Search
Tomek Paczkowski
November 28, 2013
Programming
0
52
Python Memory Management
A simple talk about simple problems with Python memory allocator
Tomek Paczkowski
November 28, 2013
Tweet
Share
More Decks by Tomek Paczkowski
See All by Tomek Paczkowski
Dispelling py.test magic
oinopion
0
700
Python Fest lightning talk
oinopion
0
83
Contributing to Django
oinopion
2
490
Django and Time Zones: New in the Trunk
oinopion
4
200
Other Decks in Programming
See All in Programming
オートマトン学習しろ / Do automata learning
makenowjust
3
130
ドメイン駆動設計を実践するために必要なもの
bikisuke
4
330
What we keep in mind when migrating from Serverless Framework to AWS CDK and AWS SAM
kasacchiful
1
140
Kotlin 2.0が与えるAndroid開発の進化
masayukisuda
1
330
いまから追い上げる、Jetpack Compose トレーニング
nyafunta9858
0
300
マイグレーションコード自作して File-Based Routing に自動移行!! ~250 ページの歴史的経緯を添えて~
cut0
1
260
LangGraphでのHuman-in-the-Loopの実装
os1ma
3
1k
A New Era of Testing
mannodermaus
2
310
The Shape of a Service Object
inem
0
510
月間4.5億回再生を超える大規模サービス TVer iOSアプリのリアーキテクチャ戦略 - iOSDC2024
techtver
PRO
1
820
僕が思い描くTypeScriptの未来を勝手に先取りする
yukukotani
9
2.4k
How to Break into Reading Open Source
kaspth
1
210
Featured
See All Featured
WebSockets: Embracing the real-time Web
robhawkes
59
7.3k
Bootstrapping a Software Product
garrettdimon
PRO
304
110k
Optimizing for Happiness
mojombo
375
69k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
28
1.6k
In The Pink: A Labor of Love
frogandcode
139
22k
Building a Modern Day E-commerce SEO Strategy
aleyda
36
6.8k
Product Roadmaps are Hard
iamctodd
PRO
48
10k
The Illustrated Children's Guide to Kubernetes
chrisshort
47
48k
Git: the NoSQL Database
bkeepers
PRO
425
64k
Being A Developer After 40
akosma
84
590k
Building Your Own Lightsaber
phodgson
101
6k
Embracing the Ebb and Flow
colly
83
4.4k
Transcript
Python Memory Management Tomasz Paczkowski Pykonik, 28.11.2013
Case Study • Long lived web process • Periodically allocates
boatloads of memory • Never releases it
Distilled code def main():! big = alloc(100000)! small = alloc(1)!
del big! # memory not released
[Demo time]
Diagnose: Memory Fragmentation big small small big
Solution • Make better use of memory • Subprocess •
jemalloc* via LD_PRELOAD
Fun with Python allocator • Python does not use malloc
directly — too costly for small objects • Instead implements more sophisticated allocator on top of malloc
Pools for integers ints = range(5*1000*1000)! del ints! import gc;
gc.collect(2)
Free lists • For handful of most common types Python
keeps unused objects in so called free lists • Those are most significantly: lists, dictionaries, frames • Speeds up code execution immensely
[Demo time]
Conclusions • Sometimes memory leak is not what it seems
• glibc malloc is not the best of breed • do memory intensive work in subprocess • be mindful when using C extensions
Thanks. Questions?