Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Garbage Collection in Python by Benjamin Peterson
Search
PyCon 2014
April 12, 2014
5
1.4k
Garbage Collection in Python by Benjamin Peterson
PyCon 2014
April 12, 2014
Tweet
Share
More Decks by PyCon 2014
See All by PyCon 2014
Postgres Performance for Humans by Craig Kerstiens
pycon2014
29
3.6k
Technical Onboarding, Training, and Mentoring by Kate Heddleston and Nicole Zuckerman
pycon2014
1
2.2k
"My big gay adventure. Making, releasing and selling an indie game made in python." by Luke Miller
pycon2014
2
1.5k
Farewell and Welcome Home, Python in Two Genders by Naomi_Ceder
pycon2014
1
700
Deliver Your Software in an Envelope by Augie Fackler and Nathaniel Manista
pycon2014
1
520
Hitchhikers Guide to Free and Open Source Participation by Elena Williams
pycon2014
6
1.1k
Localization Revisted (aka. Translations Evolved) by Ruchi Varshney
pycon2014
0
680
Smart Dumpster by Bradley E. Angell
pycon2014
0
480
Software Engineering for Hackers: Bridging the Two Solitudes by Tavish Armstrong
pycon2014
0
700
Featured
See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
504
140k
Thoughts on Productivity
jonyablonski
67
4.2k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
355
29k
Done Done
chrislema
181
16k
The Cost Of JavaScript in 2023
addyosmani
43
5.8k
Faster Mobile Websites
deanohume
304
30k
Music & Morning Musume
bryan
46
6.1k
Intergalactic Javascript Robots from Outer Space
tanoku
268
27k
Web development in the modern age
philhawksworth
205
10k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
105
48k
Being A Developer After 40
akosma
84
590k
How GitHub (no longer) Works
holman
311
140k
Transcript
Garbage Collection Benjamin Peterson
Outline ➔ How GC works in various Python implementations ◆
optimizations ➔ GC semantics subtleties
Part 1 Implementation Basics
Bias & Disclaimer
What is GC in Python? ➔ Unused objects are finalized
and deallocated. ➔ When? ◆ “eventually”... or never! ◆ not running out of memory is good
CPython
CPython: reference counting ➔ Every object has a count of
how many other objects want to keep it alive. ➔ New objects have ref count 1. ➔ When ref count is 0, the object can be deleted.
New object Preexisting objects
Preexisting objects New object
Preexisting objects Dead object
Preexisting objects Dead object Dead object
refcounting example
Reference Counting’s Major Flaw Reference Cycles
Preexisting objects
Preexisting objects Cycle keeps itself alive!
CPython’s cyclic GC ➔ Detects cycles unreachable from the program
and deletes them ➔ Runs every once and while on allocation ➔ on CPython
Preexisting objects Cyclic GC subtracts internal references
Preexisting objects Cycles are now deleted
None
PyPy review ➔ Interpreter written in RPython ➔ RPython translated
to low level language (C) ➔ Interpreter is abstracted from low-level details like GC
PyPy has pluggable GCs
PyPy GC ➔ GC is simply another low-level transform during
translation. ➔ GC algorithm itself is written in RPython. ➔ GC implementation can be selected at translation time. ➔ Current default GC: “minmark”
Mark and Sweep ➔ Step 1: Starting from known live
objects, recursively traverse objects, marking them as reachable. ➔ Step 2: Walk all allocated objects, deleting that ones that aren’t marked as alive. ➔ No need to worry about reference cycles.
Preexisting objects Unreachable object Marking
Preexisting objects Unreachable object Marking GC traverses references to C.
Preexisting objects Unreachable object Sweeping GC notices that D is
unreachable and deletes it.
Fancy PyPy GC Optimizations
Python allocates constantly
“High Infant Mortality”
The nursery ➔ Store newly allocated objects in a “nursery”
➔ Collect the “nursery” often and move surviving objects elsewhere (minor collection) ➔ Garbage collect old objects less frequently (major collection)
GC Pauses ➔ When the GC is running, the program
is not. ➔ Inconvenient for many long running programs like servers. ➔ A deal-breaker for real-time applications like video processing.
PyPy incremental GC ➔ In PyPy 2.2 ➔ Major collection
split into multiple passes, each lasting only a few milliseconds. ➔ “All in all it was relatively painless work.”
GC in PyPy summary ➔ Pluggable ➔ Generational ➔ Incremental
➔ Integrated with the JIT
Part 2 GC Semantics
kills kittens
Cycles with finalizers: a conundrum Which finalizer to run first?
Reference to the alive world.
What to do about cycles with ? ➔ CPython <
3.4: give up ➔ CPython >= 3.4: PEP 442 ➔ PyPy: sort finalizers into a “reasonable” order and run them
PEP 442 - “Safe Object Finalization” ➔ Step 1: Run
finalizers on unreachable cycles (arbitrary order). Resurrect any cycles that become reachable again. ➔ Step 2: Break references in remaining cycles.
Themes ➔ Garbage Collection is hard. ➔ (PyPy’s) GCs are
awesome!
Queries? (
[email protected]
) ➔ PyPy Blog (http://morepypy.blogspot.com) ➔ GC module documentation
➔ Wikipedia article on garbage collection ➔ The source ◆ ◆