Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Python Memory Management
Search
Tomek Paczkowski
November 28, 2013
Programming
0
49
Python Memory Management
A simple talk about simple problems with Python memory allocator
Tomek Paczkowski
November 28, 2013
Tweet
Share
More Decks by Tomek Paczkowski
See All by Tomek Paczkowski
Dispelling py.test magic
oinopion
0
620
Python Fest lightning talk
oinopion
0
76
Contributing to Django
oinopion
2
460
Django and Time Zones: New in the Trunk
oinopion
4
180
Other Decks in Programming
See All in Programming
Goのmultiple errorsについて (2024年4月版)
syumai
3
830
Tailwind CSSを本気でカスタマイズする方法
fsubal
13
5.3k
Let's learn code review
riofujimon
1
300
CA.swift19 恋するAIアプリ開発の裏側
oskmr
0
360
GitHub Copilotのススメ
marcy731
1
200
Elm Form Validation
bkuhlmann
0
510
Apache Hive 4 on Treasure Data
ryukobayashi
0
310
データアナリストが行うDatabricksを活用したETLの自動化事例
shinoa
0
270
Java 22 Overview
kishida
1
180
ONE WEDGE_company_guide
1wedge_one
0
480
冗長なエラーログを削減し、スタックトレースを手に入れる / Reducing Verbose Error Logs and Obtaining Stack Traces
upamune
0
680
Random\Randomizer クラスで日常のあれこれを解決しよう! / Random\Randomizer class solves familiar trouble
cocoeyes02
0
240
Featured
See All Featured
Creatively Recalculating Your Daily Design Routine
revolveconf
210
11k
The Language of Interfaces
destraynor
151
23k
Atom: Resistance is Futile
akmur
259
25k
Producing Creativity
orderedlist
PRO
337
39k
Six Lessons from altMBA
skipperchong
21
3k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
60
14k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
352
28k
How To Stay Up To Date on Web Technology
chriscoyier
782
250k
Gamification - CAS2011
davidbonilla
76
4.6k
Rebuilding a faster, lazier Slack
samanthasiow
73
8.2k
From Idea to $5000 a Month in 5 Months
shpigford
377
45k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
34
8.9k
Transcript
Python Memory Management Tomasz Paczkowski Pykonik, 28.11.2013
Case Study • Long lived web process • Periodically allocates
boatloads of memory • Never releases it
Distilled code def main():! big = alloc(100000)! small = alloc(1)!
del big! # memory not released
[Demo time]
Diagnose: Memory Fragmentation big small small big
Solution • Make better use of memory • Subprocess •
jemalloc* via LD_PRELOAD
Fun with Python allocator • Python does not use malloc
directly — too costly for small objects • Instead implements more sophisticated allocator on top of malloc
Pools for integers ints = range(5*1000*1000)! del ints! import gc;
gc.collect(2)
Free lists • For handful of most common types Python
keeps unused objects in so called free lists • Those are most significantly: lists, dictionaries, frames • Speeds up code execution immensely
[Demo time]
Conclusions • Sometimes memory leak is not what it seems
• glibc malloc is not the best of breed • do memory intensive work in subprocess • be mindful when using C extensions
Thanks. Questions?