Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python Memory Management 101.Deeping in Garbage collector

Python Memory Management 101.Deeping in Garbage collector

In this talk I will try explain the memory internals of Python and discover how it handles memory management and object creation.
The idea is explain how objects are created and deleted in Python and how garbage collector(gc) functions to automatically release memory when the object taking the space is no longer in use.
I will review the main mechanisms for memory allocation and how the garbage collector works in conjunction with the memory manager for reference counting of the python objects.
Finally, I will comment the best practices for memory management such as writing efficient code in python scripts.

jmortegac

May 09, 2020
Tweet

More Decks by jmortegac

Other Decks in Programming

Transcript

  1. José Manuel Ortega
    @jmortegac
    Python Memory
    Management 101
    Deeping in Garbage collector

    View full-size slide

  2. About me
    ● @jmortegac
    ● http://jmortega.github.io
    ● https://www.linkedin.com/in/jmortega1/

    View full-size slide

  3. Agenda
    ● Introduction to memory management
    ● Garbage collector and reference counting
    with python
    ● Review the gc module for configuring the
    python garbage collector
    ● Best practices for memory managment

    View full-size slide

  4. Introduction to memory management
    ● Memory management is the process of
    efficiently allocating, de-allocating, and
    coordinating memory so that all the
    different processes run smoothly and can
    optimally access different system
    resources.

    View full-size slide

  5. Python Objects in Memory

    View full-size slide

  6. Python Memory Manager

    View full-size slide

  7. Heap allocation

    View full-size slide

  8. Heap allocation
    def main():
    x=300
    print(id(x))
    w=fun(x)
    print(id(w))
    def sqr(x):
    print (id(x))
    z=x*x
    print(id(z))
    return z
    if __name__ == "__main__":
    main()

    View full-size slide

  9. Python Objects in Memory
    ● Each variable in Python acts as an object
    ● Python is a dynamically typed language
    which means that we do not need to
    declare types for variables.

    View full-size slide

  10. Python Objects in Memory

    View full-size slide

  11. Python Objects in Memory

    View full-size slide

  12. Python Objects in Memory

    View full-size slide

  13. Python Objects in Memory

    View full-size slide

  14. id() function

    View full-size slide

  15. id() function

    View full-size slide

  16. Reference counting
    ● Python manages objects by using reference
    counting
    ● Reference counting works by counting the
    number of times an object is referenced by
    other objects in the application.
    ● When references to an object are removed,
    the reference count for an object is
    decremented.

    View full-size slide

  17. Reference counting
    ● A reference is a container object pointing at
    another object.
    ● Reference counting is a simple technique
    in which objects are allocated when there is
    reference to them in a program

    View full-size slide

  18. Reference counting
    ● when reference count increases?
    ○ x=1
    ○ def(x):
    ○ list.append(x)

    View full-size slide

  19. Reference counting

    View full-size slide

  20. Reference counting

    View full-size slide

  21. Reference counting

    View full-size slide

  22. Reference counting
    ● Easy to implement
    ● Objects are immediately deleted when
    reference counter is 0
    ✗ Not thread-safe
    ✗ Doesn’t detect cyclic references
    ✗ space overhead - reference count is
    stored for every object

    View full-size slide

  23. Python Garbage collector
    ● The Python Garbage Collector (GC) runs
    during the program execution and is
    triggered if the reference count reduces to
    zero.

    View full-size slide

  24. Garbage collector(GC) module

    View full-size slide

  25. Python Garbage collector
    ● Reference Counting + Generational GC
    ● RefCount reaches zero, immediate
    deletion
    ● Deleted objects with cyclic references
    are deleted with Tracing GC

    View full-size slide

  26. Garbage collector(GC) reference cycle

    View full-size slide

  27. Garbage collector(GC) reference cycle
    >>> def ref_cycle():
    ... list = [1, 2, 3, 4]
    ... list.append(list)
    ... return list

    View full-size slide

  28. Garbage collector(GC) reference cycle
    import gc
    for i in range(8):
    ref_cycle()
    n = gc.collect()
    print("Number of unreachable objects collected by GC:", n)
    print("Uncollectable garbage:", gc.garbage)
    print("Number of unreachable objects collected by GC:",
    gc.collect())

    View full-size slide

  29. Garbage collector(GC) reference cycle

    View full-size slide

  30. Python Object Graphs
    https://mg.pov.lt/objgraph/
    >>> x = []
    >>> list = [x, [x], dict(x=x)]
    >>> import objgraph
    >>> objgraph.show_refs([y],
    filename='sample-graph.png')

    View full-size slide

  31. Best practices for memory managment
    ● Using gc.collect() carefully
    print("Collecting...")
    n = gc.collect()
    print("Number of unreachable objects collected:", n)
    print("Uncollectable garbage:", gc.garbage)

    View full-size slide

  32. Best practices for memory managment
    ● Using with context manager for working
    with files
    with open('data.txt', 'r') as file:
    data = ','.join(line.strip() for line in file)

    View full-size slide

  33. Best practices for memory managment
    ● Avoid List Slicing with [:]
    list= [1,2,3,4]
    list[1:3]
    list[slice(1,3)]

    View full-size slide

  34. Best practices for memory managment
    ● String Concatenation
    string= “hello”
    string+= “world”
    wordList = ("hello", "world")
    string = " ".join(wordList)

    View full-size slide

  35. Best practices for memory managment
    ● Use Iterators and Generators

    View full-size slide

  36. References
    ● https://stackabuse.com/basics-of-memory-management-in-p
    ython/
    ● https://realpython.com/python-memory-management
    ● https://rushter.com/blog/python-garbage-collector
    ● https://pythonchb.github.io/PythonTopics/weak_references.
    html

    View full-size slide

  37. https://www.youtube.com/c/JoseManuelOrtegadev

    View full-size slide

  38. https://speakerdeck.com/jmortega

    View full-size slide