Python Memory Management 101.Deeping in Garbage collector

In this talk I will try explain the memory internals of Python and discover how it handles memory management and object creation.
The idea is explain how objects are created and deleted in Python and how garbage collector(gc) functions to automatically release memory when the object taking the space is no longer in use.
I will review the main mechanims for memory allocation and how the garbage collector works in conjunction with the memory manager for reference counting of the python objects.
Finally, I will comment the best practices for memory managment such as writing efficient code in python scripts.

Transcript

1. José Manuel Ortega
   @jmortegac
   Python Memory
   Management 101
   Deeping in Garbage collector
   

   View Slide

2. About me
   ● @jmortegac
   ● http://jmortega.github.io
   ● https://www.linkedin.com/in/jmortega1/
   

   View Slide

3. Agenda
   ● Introduction to memory management
   ● Garbage collector and reference
   counting with python
   ● Review the gc module for configuring
   the python garbage collector
   ● Best practices for memory managment
   

   View Slide

4. Introduction to memory management
   ● Memory management is the process of
   efficiently allocating, de-allocating, and
   coordinating memory so that all the
   different processes run smoothly and can
   optimally access different system
   resources.
   

   View Slide

5. Python Objects in Memory
   

   View Slide

6. Python Memory Manager
   

   View Slide

7. Heap allocation
   

   View Slide

8. def main():
   x=300
   print(id(x))
   w=fun(x)
   print(id(w))
   def sqr(x):
   print (id(x))
   z=x*x
   print(id(z))
   return z
   if __name__ == "__main__":
   main()
   Heap allocation
   

   View Slide

9. Python Objects in Memory
   ● Each variable in Python acts as an object
   ● Python is a dynamically typed language
   which means that we do not need to
   declare types for variables.
   

   View Slide

10. Python Objects in Memory
    

    View Slide

11. Python Objects in Memory
    

    View Slide

12. Python Objects in Memory
    

    View Slide

13. Python Objects in Memory
    

    View Slide

14. id() method
    

    View Slide

15. id() method
    

    View Slide

16. id() method
    

    View Slide

17. is Operator
    

    View Slide

18. Reference counting
    ● Python manages objects by using reference
    counting
    ● Reference counting works by counting the
    number of times an object is referenced by
    other objects in the application.
    ● When references to an object are removed,
    the reference count for an object is
    decremented.
    

    View Slide

19. ● A reference is a container object pointing at
    another object.
    ● Reference counting is a simple technique
    in which objects are allocated when there is
    reference to them in a program
    Reference counting
    

    View Slide

20. ● when reference count increases?
    ○ x=1
    ○ def(x):
    ○ list.append(x)
    Reference counting
    

    View Slide

21. Reference counting
    

    View Slide

22. Reference counting
    

    View Slide

23. Reference counting
    

    View Slide

24. ● Easy to implement
    ● Objects are immediately deleted when
    reference counter is 0
    ✗ Not thread-safe
    ✗ Doesn’t detect cyclic references
    ✗ space overhead - reference count is
    stored for every object
    Reference counting
    

    View Slide

25. Garbage collector(GC) module
    

    View Slide

26. ● Reference Counting + Generational GC
    ● RefCount reaches zero, immediate
    deletion
    ● Deleted objects with cyclic references
    are deleted with Tracing GC
    Python Garbage collector
    

    View Slide

27. Garbage collector(GC) reference cycle
    

    View Slide

28. >>> def ref_cycle():
    ... list = [1, 2, 3, 4]
    ... list.append(list)
    ... return list
    Garbage collector(GC) reference cycle
    

    View Slide

29. Garbage collector(GC) reference cycle
    import gc
    for i in range(8):
    ref_cycle()
    n = gc.collect()
    print("Number of unreachable objects collected by GC:", n)
    print("Uncollectable garbage:", gc.garbage)
    print("Number of unreachable objects collected by GC:",
    gc.collect())
    

    View Slide

30. Garbage collector(GC) reference cycle
    

    View Slide

31. Python Object Graphs
    https://mg.pov.lt/objgraph/
    import objgraph
    x = "hello"
    y = [x, [x], list(x), dict(x=x)]
    objgraph.show_refs([y],
    filename='sample-graph.png')
    

    View Slide

32. Best practices for memory management
    ● Using gc.collect() carefully
    print("Collecting...")
    n = gc.collect()
    print("Number of unreachable objects collected:", n)
    print("Uncollectable garbage:", gc.garbage)
    

    View Slide

33. Garbage collector(GC) methods
    

    View Slide

34. ● Using with context manager for working
    with files
    with open('data.txt', 'r') as file:
    data = ','.join(line.strip() for line in file)
    Best practices for memory management
    

    View Slide

35. ● Avoid List Slicing with [:]
    list= [1,2,3,4]
    list[1:3]
    list[slice(1,3)]
    Best practices for memory management
    

    View Slide

36. Best practices for memory managment
    ● String Concatenation
    string= “hello”
    string+= “world”
    wordList = ("hello", "world")
    string = " ".join(wordList)
    

    View Slide

37. ● Use Iterators and Generators
    Best practices for memory management
    

    View Slide

38. ● https://stackabuse.com/basics-of-memory-management-in-p
    ython/
    ● https://realpython.com/python-memory-management
    ● https://rushter.com/blog/python-garbage-collector
    ● https://pythonchb.github.io/PythonTopics/weak_references.
    html
    References
    

    View Slide

39. https://www.youtube.com/c/JoseManuelOrtegadev
    

    View Slide