Tweet
Tweet

Slide 1

Slide 1 text

José Manuel Ortega @jmortegac Python Memory Management 101 Deeping in Garbage collector

Slide 2

Slide 2 text

About me ● @jmortegac ● http://jmortega.github.io ● https://www.linkedin.com/in/jmortega1/

Slide 3

Slide 3 text

Agenda ● Introduction to memory management ● Garbage collector and reference counting with python ● Review the gc module for configuring the python garbage collector ● Best practices for memory managment

Slide 4

Slide 4 text

Introduction to memory management ● Memory management is the process of efficiently allocating, de-allocating, and coordinating memory so that all the different processes run smoothly and can optimally access different system resources.

Slide 5

Slide 5 text

Python Objects in Memory

Slide 6

Slide 6 text

Python Memory Manager

Slide 7

Slide 7 text

Heap allocation

Slide 8

Slide 8 text

Heap allocation def main(): x=300 print(id(x)) w=fun(x) print(id(w)) def sqr(x): print (id(x)) z=x*x print(id(z)) return z if __name__ == "__main__": main()

Slide 9

Slide 9 text

Python Objects in Memory ● Each variable in Python acts as an object ● Python is a dynamically typed language which means that we do not need to declare types for variables.

Slide 10

Slide 10 text

Python Objects in Memory

Slide 11

Slide 11 text

Python Objects in Memory

Slide 12

Slide 12 text

Python Objects in Memory

Slide 13

Slide 13 text

Python Objects in Memory

Slide 14

Slide 14 text

id() method

Slide 15

Slide 15 text

id() function

Slide 16

Slide 16 text

id() function

Slide 17

Slide 17 text

Is Operator

Slide 18

Slide 18 text

Reference counting ● Python manages objects by using reference counting ● Reference counting works by counting the number of times an object is referenced by other objects in the application. ● When references to an object are removed, the reference count for an object is decremented.

Slide 19

Slide 19 text

Reference counting ● A reference is a container object pointing at another object. ● Reference counting is a simple technique in which objects are allocated when there is reference to them in a program

Slide 20

Slide 20 text

Reference counting ● when reference count increases? ○ x=1 ○ def(x): ○ list.append(x)

Slide 21

Slide 21 text

Reference counting

Slide 22

Slide 22 text

Reference counting

Slide 23

Slide 23 text

Reference counting

Slide 24

Slide 24 text

Reference counting ● Easy to implement ● Objects are immediately deleted when reference counter is 0 ✗ Not thread-safe ✗ Doesn’t detect cyclic references ✗ space overhead - reference count is stored for every object

Slide 25

Slide 25 text

Python Garbage collector ● The Python Garbage Collector (GC) runs during the program execution and is triggered if the reference count reduces to zero.

Slide 26

Slide 26 text

Garbage collector(GC) module

Slide 27

Slide 27 text

Python Garbage collector ● Reference Counting + Generational GC ● RefCount reaches zero, immediate deletion ● Deleted objects with cyclic references are deleted with Tracing GC

Slide 28

Slide 28 text

Garbage collector(GC) reference cycle

Slide 29

Slide 29 text

Garbage collector(GC) reference cycle >>> def ref_cycle(): ... list = [1, 2, 3, 4] ... list.append(list) ... return list

Slide 30

Slide 30 text

Garbage collector(GC) reference cycle import gc for i in range(8): ref_cycle() n = gc.collect() print("Number of unreachable objects collected by GC:", n) print("Uncollectable garbage:", gc.garbage) print("Number of unreachable objects collected by GC:", gc.collect())

Slide 31

Slide 31 text

Garbage collector(GC) reference cycle

Slide 32

Slide 32 text

Python Object Graphs https://mg.pov.lt/objgraph/ >>> x = [] >>> list = [x, [x], dict(x=x)] >>> import objgraph >>> objgraph.show_refs([y], filename='sample-graph.png')

Slide 33

Slide 33 text

Best practices for memory managment ● Using gc.collect() carefully print("Collecting...") n = gc.collect() print("Number of unreachable objects collected:", n) print("Uncollectable garbage:", gc.garbage)

Slide 34

Slide 34 text

Best practices for memory managment ● Using with context manager for working with files with open('data.txt', 'r') as file: data = ','.join(line.strip() for line in file)

Slide 35

Slide 35 text

Best practices for memory managment ● Avoid List Slicing with [:] list= [1,2,3,4] list[1:3] list[slice(1,3)]

Slide 36

Slide 36 text

Best practices for memory managment ● String Concatenation string= “hello” string+= “world” wordList = ("hello", "world") string = " ".join(wordList)

Slide 37

Slide 37 text

Best practices for memory managment ● Use Iterators and Generators

Slide 38

Slide 38 text

References ● https://stackabuse.com/basics-of-memory-management-in-p ython/ ● https://realpython.com/python-memory-management ● https://rushter.com/blog/python-garbage-collector ● https://pythonchb.github.io/PythonTopics/weak_references. html

Slide 39

Slide 39 text

https://www.youtube.com/c/JoseManuelOrtegadev

Slide 40

Slide 40 text

https://speakerdeck.com/jmortega