Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Hidden Power of the Python Runtime

The Hidden Power of the Python Runtime

Many people like Python for its simplicity and beauty. But every statement in Python, even the simple one, produces a lot of events during the program execution. These events are usually hidden from a user, so it helps developers to skip low-level implementation details and focus on bigger things.

At the same time many parts of this hidden information are very useful and interesting to examine. The good news is that the Python Runtime allows to retrieve it really simply, so there is no need to configure additional libraries or pass additional parameters to interpreter. Everybody can do it right inside their Python code.

During this talk we will learn how Python allows to inspect current program state during the execution. We will learn about Python memory model, variables, frame objects and about useful information they store. After that we will discuss several powerful tools which are based on the runtime information and which can be very helpful for any Python programmer in their everyday life.

Elizaveta Shashkova

April 17, 2020
Tweet

More Decks by Elizaveta Shashkova

Other Decks in Programming

Transcript

  1. @lisa_shashkova AssertionError w VOJUUFTU w !5 Traceback (most recent call

    last): File “/file.py”, line 8, in test_value assert a == 2, "Wrong value!" AssertionError: Wrong value!
  2. @lisa_shashkova AssertionError w VOJUUFTU w QZUFTU !6 Traceback (most recent

    call last): File “/file.py”, line 8, in test_value assert a == 2, "Wrong value!" AssertionError: Wrong value! E AssertionError: Wrong value! E assert 1 == 2
  3. @lisa_shashkova Python Objects w $PSFUZQFT w OVNCFST TUSJOHT DPMMFDUJPOT w

    1SPHSBNVOJUPCKFDUT w GVODUJPOT DMBTTFT NPEVMFT !10
  4. @lisa_shashkova Created Explicitly a = 27 def get_answer(): return 42

    class Person: def __init__(self, age): self.age = age !11
  5. @lisa_shashkova Stack Frame !15 def foo(n): n = n -

    1 return n def bar(k): res = foo(k) return res * 2 print(bar(1)) module bar() 1 2 3 4 5 6 7 8 9 10 11
  6. @lisa_shashkova Stack Frame !16 module bar() foo() def foo(n): n

    = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1)) 1 2 3 4 5 6 7 8 9 10 11 def foo(n): n = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1))
  7. @lisa_shashkova Stack Frame !17 module bar() foo() def foo(n): n

    = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1)) 1 2 3 4 5 6 7 8 9 10 11 def foo(n): n = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1))
  8. @lisa_shashkova Stack Frame !18 module bar() foo() def foo(n): n

    = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1)) 1 2 3 4 5 6 7 8 9 10 11 def foo(n): n = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1))
  9. @lisa_shashkova Stack Frame !19 module bar() def foo(n): n =

    n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1)) 1 2 3 4 5 6 7 8 9 10 11 def foo(n): n = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1))
  10. @lisa_shashkova Frame Object: Variables !27 w -PDBMWBSJBCMFT • frame.f_locals w

    (MPCBMWBSJBCMFT • frame.f_globals • locals() & globals()
  11. @lisa_shashkova Code Object !30 w 3FQSFTFOUTBDIVOLPGFYFDVUBCMFDPEF >>> c = compile('a

    + b', 'a.py', 'eval') <code object <module> at 0x104f8fc90, file "a.py", line 1>
  12. @lisa_shashkova Code Object !31 w 3FQSFTFOUTBDIVOLPGFYFDVUBCMFDPEF >>> c = compile('a

    + b', 'a.py', 'eval') <code object <module> at 0x104f8fc90, file "a.py", line 1> >>> eval(c, {'a': 1, 'b': 2}) 3
  13. @lisa_shashkova Previous Frame !36 Traceback (most recent call last): File

    "file.py", line 12, in <module> print(bar(1)) File "file.py", line 8, in bar res = foo(k) File "file.py", line 2, in foo raise ValueError("Wrong value!") ValueError: Wrong value! module bar() foo()
  14. @lisa_shashkova Frame Object !38 w inspectNPEVMF w )BOEMFGSBNFWBSJBCMFDBSFGVMMZ def handle_stackframe_without_leak():

    frame = inspect.currentframe() try: # do something with the frame finally: del frame
  15. @lisa_shashkova AssertionError Variables !43 def vars_in_assert(e): tb = e.__traceback__ frame

    = tb.tb_frame code = frame.f_code line = tb.tb_lineno - code.co_firstlineno + 1 source = inspect.getsource(code)
  16. @lisa_shashkova AssertionError Variables !44 def get_vars_names(source, line): # get variables

    names with `ast` module def vars_in_assert(e): tb = e.__traceback__ frame = tb.tb_frame code = frame.f_code line = tb.tb_lineno - code.co_firstlineno + 1 source = inspect.getsource(code) for name in get_vars_names(source, line): pass
  17. @lisa_shashkova AssertionError Variables !45 def get_vars_names(source, line): # get variables

    names with `ast` module def vars_in_assert(e): tb = e.__traceback__ frame = tb.tb_frame code = frame.f_code line = tb.tb_lineno - code.co_firstlineno + 1 source = inspect.getsource(code) for name in get_vars_names(source, line): if name in frame.f_locals: var = frame.f_locals[name] print(f"{name} = {var}")
  18. @lisa_shashkova Usage !46 >>> try: assert a + b <

    1 except AssertionError as e: vars_in_assert(e)
  19. @lisa_shashkova Usage !47 >>> try: assert a + b <

    1 except AssertionError as e: vars_in_assert(e) File “/file.py", line 10, in foo assert a + b < 1 Variables Values: a = 1 b = 2
  20. @lisa_shashkova Tracing Function w tracefunc(frame, event, arg) w sys.settrace(tracefunc) -

    TFUUPDVSSFOUGSBNF w 4UPSFEJOBGSBNFframe.f_trace w %FCVHHFSBOBMZTFTFWFOUT !52
  21. @lisa_shashkova Collecting Types !73 def arg_names(co): nargs = co.co_argcount names

    = co.co_varnames return list(names[:nargs]) names = arg_names(frame.f_code)
  22. @lisa_shashkova Collecting Types !74 def arg_names(co): nargs = co.co_argcount names

    = co.co_varnames return list(names[:nargs]) names = arg_names(frame.f_code) locs = frame.f_locals objects = [locs[n] for n in names]
  23. @lisa_shashkova Deadlock !81 def run1(): with lock1: with lock2: #

    do sth def run2(): with lock2: with lock1: # do sth else Thread(target=run1).start() Thread(target=run2).start()
  24. @lisa_shashkova Async Locks !87 alock = asyncio.Lock() alock.acquire() # only

    one task here alock.release() async with alock: # equivalent
  25. @lisa_shashkova Async Fault Handler w *OBTFQBSBUFUISFBE !90 def dump_traceback(loop): for

    task in asyncio.all_tasks(loop): task.print_stack() def dump_traceback_later(timeout, loop): while True: sleep(timeout) dump_traceback(loop, timeout)
  26. @lisa_shashkova The Hidden Power of Runtime w 1ZUIPO3VOUJNFJTWFSZQPXFSGVM w &BTZBDDFTTUPTUBDLGSBNFBOEDPEFPCKFDUT

    w %FWFMPQNFOU5PPMT w QZUFTU %FCVHHFS $PEF$PWFSBHF 
 5ZQJOH*OGPSNBUJPO 'BVMU
 )BOEMFS !92