Python Memory Management 101.Deeping in Garbage collector

Python Memory Management 101.Deeping in Garbage collector

In this talk I will try explain the memory internals of Python and discover how it handles memory management and object creation.
The idea is explain how objects are created and deleted in Python and how garbage collector(gc) functions to automatically release memory when the object taking the space is no longer in use.
I will review the main mechanims for memory allocation and how the garbage collector works in conjunction with the memory manager for reference counting of the python objects.
Finally, I will comment the best practices for memory managment such as writing efficient code in python scripts.

7c4b1ae16723b56facc7a8a8f95aa6ce?s=128

jmortegac

July 24, 2020
Tweet

Transcript

  1. José Manuel Ortega @jmortegac Python Memory Management 101 Deeping in

    Garbage collector
  2. About me • @jmortegac • http://jmortega.github.io • https://www.linkedin.com/in/jmortega1/

  3. Agenda • Introduction to memory management • Garbage collector and

    reference counting with python • Review the gc module for configuring the python garbage collector • Best practices for memory managment
  4. Introduction to memory management • Memory management is the process

    of efficiently allocating, de-allocating, and coordinating memory so that all the different processes run smoothly and can optimally access different system resources.
  5. Python Objects in Memory

  6. Python Memory Manager

  7. Heap allocation

  8. def main(): x=300 print(id(x)) w=fun(x) print(id(w)) def sqr(x): print (id(x))

    z=x*x print(id(z)) return z if __name__ == "__main__": main() Heap allocation
  9. Python Objects in Memory • Each variable in Python acts

    as an object • Python is a dynamically typed language which means that we do not need to declare types for variables.
  10. Python Objects in Memory

  11. Python Objects in Memory

  12. Python Objects in Memory

  13. Python Objects in Memory

  14. id() method

  15. id() method

  16. id() method

  17. is Operator

  18. Reference counting • Python manages objects by using reference counting

    • Reference counting works by counting the number of times an object is referenced by other objects in the application. • When references to an object are removed, the reference count for an object is decremented.
  19. • A reference is a container object pointing at another

    object. • Reference counting is a simple technique in which objects are allocated when there is reference to them in a program Reference counting
  20. • when reference count increases? ◦ x=1 ◦ def(x): ◦

    list.append(x) Reference counting
  21. Reference counting

  22. Reference counting

  23. Reference counting

  24. • Easy to implement • Objects are immediately deleted when

    reference counter is 0 ✗ Not thread-safe ✗ Doesn’t detect cyclic references ✗ space overhead - reference count is stored for every object Reference counting
  25. Garbage collector(GC) module

  26. • Reference Counting + Generational GC • RefCount reaches zero,

    immediate deletion • Deleted objects with cyclic references are deleted with Tracing GC Python Garbage collector
  27. Garbage collector(GC) reference cycle

  28. >>> def ref_cycle(): ... list = [1, 2, 3, 4]

    ... list.append(list) ... return list Garbage collector(GC) reference cycle
  29. Garbage collector(GC) reference cycle import gc for i in range(8):

    ref_cycle() n = gc.collect() print("Number of unreachable objects collected by GC:", n) print("Uncollectable garbage:", gc.garbage) print("Number of unreachable objects collected by GC:", gc.collect())
  30. Garbage collector(GC) reference cycle

  31. Python Object Graphs https://mg.pov.lt/objgraph/ import objgraph x = "hello" y

    = [x, [x], list(x), dict(x=x)] objgraph.show_refs([y], filename='sample-graph.png')
  32. Best practices for memory management • Using gc.collect() carefully print("Collecting...")

    n = gc.collect() print("Number of unreachable objects collected:", n) print("Uncollectable garbage:", gc.garbage)
  33. Garbage collector(GC) methods

  34. • Using with context manager for working with files with

    open('data.txt', 'r') as file: data = ','.join(line.strip() for line in file) Best practices for memory management
  35. • Avoid List Slicing with [:] list= [1,2,3,4] list[1:3] list[slice(1,3)]

    Best practices for memory management
  36. Best practices for memory managment • String Concatenation string= “hello”

    string+= “world” wordList = ("hello", "world") string = " ".join(wordList)
  37. • Use Iterators and Generators Best practices for memory management

  38. • https://stackabuse.com/basics-of-memory-management-in-p ython/ • https://realpython.com/python-memory-management • https://rushter.com/blog/python-garbage-collector • https://pythonchb.github.io/PythonTopics/weak_references. html

    References
  39. https://www.youtube.com/c/JoseManuelOrtegadev