Python Memory Management

These slides are from a talk I gave at PyATL on January 14, 2016. It is the compilation of many freely available sources on how python deals with memory, and I have cited the work of others where applicable (see references section).

Transcript

1. PYTHON MEMORY MANAGEMENT @crlane PyATL January 14, 2016
2. BUT...PYTHON MANAGES ITS OWN MEMORY
3. COUNTING REFERENCES
4. INSPECTING REFCOUNTS > > > a = [ ' p y t h o n ' , ' i s ' , ' c o o l ' ] > > > h e x ( i d ( a ) ) ' 0 x 1 0 8 8 8 3 5 a 8 ' > > > i m p o r t c t y p e s > > > d e f r e f c o u n t ( o b j ) : r e t u r n c t y p e s . c _ s i z e _ t . f r o m _ a d d r e s s ( i d ( o b j ) )
5. INSPECT A MEMORY ADDRESS > > > a _ r e f _ c o u n t = r e f c o u n t ( a ) > > > a _ r e f _ c o u n t c _ u l o n g ( 1 L )
6. BREAK REFERENCE COUNTING > > > b = a > > > a _ r e f _ c o u n t c _ u l o n g ( 2 L ) > > > a _ r e f _ c o u n t . v a l u e = 1 > > > a _ r e f _ c o u n t c _ u l o n g ( 1 L )
7. DIRTY, DIRTY HACKS > > > a [ ' p y t h o n ' , ' i s ' , ' c o o l ' ] > > > d e l b c = [ ' p l e a s e ' , ' d o ' , ' n o t ' , ' d o ' , ' t h i s ' ] > > > a [ ' p l e a s e ' , ' d o ' , ' n o t ' , ' d o ' , ' t h i s ' ] > > > h e x ( i d ( c ) ) = = h e x ( i d ( a ) ) T r u e
8. OBJECT LIFECYCLE c l a s s N o d e ( o b j e c t ) : d e f _ _ i n i t _ _ ( s e l f , v a l u e ) : s e l f . v a l u e = v a l u e n 1 = N o d e ( ' A B C ' ) n 2 = N o d e ( ' D E F ' ) n 3 = N o d e ( ' G H I ' )
9. OBJECT LIFECYCLE http://patshaughnessy.net/2013/10/24/visualizing-garbage-collection-in-ruby-and-python
10. REFCOUNT 0 http://patshaughnessy.net/2013/10/24/visualizing-garbage-collection-in-ruby-and-python
11. REASSIGNING LABELS http://patshaughnessy.net/2013/10/24/visualizing-garbage-collection-in-ruby-and-python
12. REFERENCE CYCLES http://patshaughnessy.net/2013/10/30/generational-gc-in-python-and-ruby
13. GARBAGE COLLECTION http://www.heiloftexas.com/productsItem.asp?intProductID=DuraPack%20Python
14. KEY POINTS Generational garbage collector Most objects are short lived Three linked lists of allocated objects Tunable frequency parameters
15. GENERATION 0 http://patshaughnessy.net/2013/10/30/generational-gc-in-python-and-ruby
16. GENERATION 1 http://patshaughnessy.net/2013/10/30/generational-gc-in-python-and-ruby
17. GC IS NOT FREE
18. GC standard library module provides visibility into garbage collection
19. GC > > > i m p o r t g c > > > g c . g e t _ r e f e r r e r s ( o b j ) . . . > > > g c . g e t _ r e f e r e n t s ( o b j ) > > > g c . c o l l e c t ( [ g e n e r a t i o n ] ) . . . > > > g c . g e t _ t h r e s h o l d ( ) ( 7 0 0 , 1 0 , 1 0 ) > > > g c . s e t _ t h r e s h o l d ( 1 0 0 , 5 , 5 ) > > > g c . s e t _ d e b u g ( g c . D E B U G _ L E A K ) # D E B U G _ S T A T S , e t a l .
20. OBJGRAPH https://mg.pov.lt/objgraph/_images/sample-backref-graph.png
21. OTHER CONSIDERATIONS _ _ d e l _ _ some types (e.g., int & strings) are special internal memory pools heap fragmentation
22. REFERENCES http://blog.hakril.net/articles/0-understanding-python-by-bre http://patshaughnessy.net/2013/10/24/visualizing-garbage-co ruby-and-python http://www.evanjones.ca/memoryallocator/ http://pymotw.com/2/gc/ http://mg.pov.lt/blog/hunting-python-memleaks https://mg.pov.lt/objgraph/ http://deeplearning.net/software/theano/tutorial/python-mem management.html http://www.slideshare.net/delimitry/python-gc http://www.digi.com/wiki/developer/index.php/Python_Garba