Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Hidden Power of the Python Runtime

The Hidden Power of the Python Runtime

Many people like Python for its simplicity and beauty. But every statement in Python, even the simple one, produces a lot of events during the program execution. These events are usually hidden from a user, so it helps developers to skip low-level implementation details and focus on bigger things.

At the same time many parts of this hidden information are very useful and interesting to examine. The good news is that the Python Runtime allows to retrieve it really simply, so there is no need to configure additional libraries or pass additional parameters to interpreter. Everybody can do it right inside their Python code.

During this talk we will learn how Python allows to inspect current program state during the execution. We will learn about Python memory model, variables, frame objects and about useful information they store. After that we will discuss several powerful tools which are based on the runtime information and which can be very helpful for any Python programmer in their everyday life.

658f084c2c09c8161aa6e38165b05a02?s=128

Elizaveta Shashkova

April 17, 2020
Tweet

Transcript

  1. The Hidden Power of the Python Runtime Elizaveta Shashkova PyCon

    US 2020 Online Edition
  2. @lisa_shashkova About Me w 4PGUXBSF%FWFMPQFSBU+FU#SBJOT 1Z$IBSN*%& w 4U1FUFSTCVSH 3VTTJB w

    !MJTB@TIBTILPWB !2
  3. @lisa_shashkova !3 :PVBMSFBEZVTFJUFWFSZEBZ

  4. @lisa_shashkova Test Frameworks w VOJUUFTU w QZUFTU !4

  5. @lisa_shashkova AssertionError w VOJUUFTU w !5 Traceback (most recent call

    last): File “/file.py”, line 8, in test_value assert a == 2, "Wrong value!" AssertionError: Wrong value!
  6. @lisa_shashkova AssertionError w VOJUUFTU w QZUFTU !6 Traceback (most recent

    call last): File “/file.py”, line 8, in test_value assert a == 2, "Wrong value!" AssertionError: Wrong value! E AssertionError: Wrong value! E assert 1 == 2
  7. @lisa_shashkova Contents w 1ZUIPO3VOUJNF w (FUUJOH3VOUJNF*OGPSNBUJPO w %FWFMPQNFOU5PPMT !7

  8. @lisa_shashkova Contents w 1ZUIPO3VOUJNF w (FUUJOH3VOUJNF*OGPSNBUJPO w %FWFMPQNFOU5PPMT !8

  9. @lisa_shashkova !9 l&WFSZUIJOHJTBOPCKFDUJO1ZUIPOz

  10. @lisa_shashkova Python Objects w $PSFUZQFT w OVNCFST TUSJOHT DPMMFDUJPOT w

    1SPHSBNVOJUPCKFDUT w GVODUJPOT DMBTTFT NPEVMFT !10
  11. @lisa_shashkova Created Explicitly a = 27 def get_answer(): return 42

    class Person: def __init__(self, age): self.age = age !11
  12. @lisa_shashkova Python Objects w $SFBUFEFYQMJDJUMZ w WBSJBCMFT GVODUJPOT DMBTTFT NPEVMFT

    !12
  13. @lisa_shashkova Python Objects w $SFBUFEFYQMJDJUMZ w WBSJBCMFT GVODUJPOT DMBTTFT NPEVMFT

    w $SFBUFEJNQMJDJUMZ w stack frame DPEFPCKFDU !13
  14. @lisa_shashkova Stack Frame w 3FQSFTFOUTBQSPHSBNTDPQF w *OGPSNBUJPOBCPVUFYFDVUJPOTUBUF w $PSSFTQPOEJOHDPEFPCKFDU w

    -PDBMBOEHMPCBMWBSJBCMFT w 0UIFSEBUB !14
  15. @lisa_shashkova Stack Frame !15 def foo(n): n = n -

    1 return n def bar(k): res = foo(k) return res * 2 print(bar(1)) module bar() 1 2 3 4 5 6 7 8 9 10 11
  16. @lisa_shashkova Stack Frame !16 module bar() foo() def foo(n): n

    = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1)) 1 2 3 4 5 6 7 8 9 10 11 def foo(n): n = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1))
  17. @lisa_shashkova Stack Frame !17 module bar() foo() def foo(n): n

    = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1)) 1 2 3 4 5 6 7 8 9 10 11 def foo(n): n = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1))
  18. @lisa_shashkova Stack Frame !18 module bar() foo() def foo(n): n

    = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1)) 1 2 3 4 5 6 7 8 9 10 11 def foo(n): n = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1))
  19. @lisa_shashkova Stack Frame !19 module bar() def foo(n): n =

    n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1)) 1 2 3 4 5 6 7 8 9 10 11 def foo(n): n = n - 1 return n def bar(k): res = foo(k) return res * 2 print(bar(1))
  20. @lisa_shashkova Python Power w "WBJMBCMFPVUPGUIFCPY !20 module bar()

  21. @lisa_shashkova Python Power w "WBJMBCMFPVUPGUIFCPY w 4UBDLGSBNFPCKFDUTBDDFTT !21 module bar()

  22. @lisa_shashkova Contents w 1ZUIPO3VOUJNF w (FUUJOH3VOUJNF*OGPSNBUJPO w %FWFMPQNFOU5PPMT !22

  23. @lisa_shashkova Python Stack Frame •sys._getframe([depth]) •depth OVNCFSPGDBMMTCFMPXUIFUPQ •DVSSFOUGSBNF !23

  24. @lisa_shashkova Frame Object !24

  25. @lisa_shashkova Frame Object: Variables !25 w -PDBMWBSJBCMFT • frame.f_locals

  26. @lisa_shashkova Frame Object: Variables !26 w -PDBMWBSJBCMFT • frame.f_locals w

    (MPCBMWBSJBCMFT • frame.f_globals
  27. @lisa_shashkova Frame Object: Variables !27 w -PDBMWBSJBCMFT • frame.f_locals w

    (MPCBMWBSJBCMFT • frame.f_globals • locals() & globals()
  28. @lisa_shashkova Code Object !28 •frame.f_code -GSBNF`TBUUSJCVUF

  29. @lisa_shashkova Code Object !29 w 3FQSFTFOUTBDIVOLPGFYFDVUBCMFDPEF

  30. @lisa_shashkova Code Object !30 w 3FQSFTFOUTBDIVOLPGFYFDVUBCMFDPEF >>> c = compile('a

    + b', 'a.py', 'eval') <code object <module> at 0x104f8fc90, file "a.py", line 1>
  31. @lisa_shashkova Code Object !31 w 3FQSFTFOUTBDIVOLPGFYFDVUBCMFDPEF >>> c = compile('a

    + b', 'a.py', 'eval') <code object <module> at 0x104f8fc90, file "a.py", line 1> >>> eval(c, {'a': 1, 'b': 2}) 3
  32. @lisa_shashkova Code Object !32 •code.co_filename pMFOBNFXIFSFJUXBTDSFBUFE •code.co_nameOBNFPGGVODUJPOPSNPEVMF •code.co_varnames OBNFTPGWBSJBCMFT

  33. @lisa_shashkova Code Object !33 •code.co_filename pMFOBNFXIFSFJUXBTDSFBUFE •code.co_nameOBNFPGGVODUJPOPSNPEVMF •code.co_varnames OBNFTPGWBSJBCMFT •code.co_code

    DPNQJMFECZUFDPEF EJTBTTFNCMFXJUIdis.dis()
  34. @lisa_shashkova Frame Object !34 •frame.f_lineno -DVSSFOUMJOFOVNCFS •frame.f_trace -USBDJOHGVODUJPO •frame.f_back -

    QSFWJPVTGSBNF
  35. @lisa_shashkova Previous Frame !35 module bar() foo()

  36. @lisa_shashkova Previous Frame !36 Traceback (most recent call last): File

    "file.py", line 12, in <module> print(bar(1)) File "file.py", line 8, in bar res = foo(k) File "file.py", line 2, in foo raise ValueError("Wrong value!") ValueError: Wrong value! module bar() foo()
  37. @lisa_shashkova Frame Object !37 w inspectNPEVMF

  38. @lisa_shashkova Frame Object !38 w inspectNPEVMF w )BOEMFGSBNFWBSJBCMFDBSFGVMMZ def handle_stackframe_without_leak():

    frame = inspect.currentframe() try: # do something with the frame finally: del frame
  39. @lisa_shashkova Contents w 1ZUIPO3VOUJNF w (FUUJOH3VOUJNF*OGPSNBUJPO w %FWFMPQNFOU5PPMT !39

  40. @lisa_shashkova Contents w 1ZUIPO3VOUJNF w (FUUJOH3VOUJNF*OGPSNBUJPO w %FWFMPQNFOU5PPMT !40

  41. @lisa_shashkova Development Tools w "TTFSUJPO&SSPSJOpytest !41

  42. @lisa_shashkova Exception Object w tb = e.__traceback__USBDFCBDLPCKFDU w tb.tb_frameGSBNFPCKFDU !42

  43. @lisa_shashkova AssertionError Variables !43 def vars_in_assert(e): tb = e.__traceback__ frame

    = tb.tb_frame code = frame.f_code line = tb.tb_lineno - code.co_firstlineno + 1 source = inspect.getsource(code)
  44. @lisa_shashkova AssertionError Variables !44 def get_vars_names(source, line): # get variables

    names with `ast` module def vars_in_assert(e): tb = e.__traceback__ frame = tb.tb_frame code = frame.f_code line = tb.tb_lineno - code.co_firstlineno + 1 source = inspect.getsource(code) for name in get_vars_names(source, line): pass
  45. @lisa_shashkova AssertionError Variables !45 def get_vars_names(source, line): # get variables

    names with `ast` module def vars_in_assert(e): tb = e.__traceback__ frame = tb.tb_frame code = frame.f_code line = tb.tb_lineno - code.co_firstlineno + 1 source = inspect.getsource(code) for name in get_vars_names(source, line): if name in frame.f_locals: var = frame.f_locals[name] print(f"{name} = {var}")
  46. @lisa_shashkova Usage !46 >>> try: assert a + b <

    1 except AssertionError as e: vars_in_assert(e)
  47. @lisa_shashkova Usage !47 >>> try: assert a + b <

    1 except AssertionError as e: vars_in_assert(e) File “/file.py", line 10, in foo assert a + b < 1 Variables Values: a = 1 b = 2
  48. @lisa_shashkova Development Tools w "TTFSUJPO&SSPSJOpytest !48

  49. @lisa_shashkova Development Tools w "TTFSUJPO&SSPSJOpytest w %FCVHHFS !49

  50. @lisa_shashkova Debugger !50

  51. @lisa_shashkova Python Debugger w 5SBDJOHGVODUJPO w 'SBNFFWBMVBUJPOGVODUJPO !51

  52. @lisa_shashkova Tracing Function w tracefunc(frame, event, arg) w sys.settrace(tracefunc) -

    TFUUPDVSSFOUGSBNF w 4UPSFEJOBGSBNFframe.f_trace w %FCVHHFSBOBMZTFTFWFOUT !52
  53. @lisa_shashkova Frame Evaluation w frame_eval(frame, exc) w %FCVHHFSJOTFSUTCSFBLQPJOU`TDPEFJOUPDPEFPCKFDU !53

  54. @lisa_shashkova More About Debuggers w 1Z$PO64 w l%FCVHHJOHXJUI1ZUIPO#FUUFS 'BTUFS 4USPOHFSz

    !54
  55. @lisa_shashkova Access to Frame w tracefunc(frame, event, arg) w frame_eval(frame,

    exc) !55
  56. @lisa_shashkova Debugger: Location !56 w frame.f_code.co_filename BOE frame.f_lineno

  57. @lisa_shashkova Debugger: Variables !57 w frame.f_locals

  58. @lisa_shashkova Debugger: Frames !58 w frame.f_back

  59. @lisa_shashkova Development Tools w "TTFSUJPO&SSPSJOpytest w %FCVHHFS !59

  60. @lisa_shashkova Development Tools w "TTFSUJPO&SSPSJOpytest w %FCVHHFS w $PEFDPWFSBHF !60

  61. @lisa_shashkova Code Coverage w 4IPXTXIJDIMJOFTXFSFFYFDVUFE !61

  62. @lisa_shashkova coverage.py !62 w 5IFNPTUQPQVMBSDPEFDPWFSBHFMJCSBSZ

  63. @lisa_shashkova coverage.py !63 • tracefunc(frame, event, arg) • frame.f_code.co_filename BOE

    frame.f_lineno
  64. @lisa_shashkova Development Tools w "TTFSUJPO&SSPSJOpytest w %FCVHHFS w $PEFDPWFSBHF !64

  65. @lisa_shashkova Development Tools w "TTFSUJPO&SSPSJOpytest w %FCVHHFS w $PEFDPWFSBHF w

    3VOUJNFUZQJOHUPPMT !65
  66. @lisa_shashkova Typing Tools w 1Z"OOPUBUFCZ%SPQCPY w .POLFZ5ZQFCZ*OTUBHSBN w l$PMMFDU3VOUJNFJOGPSNBUJPOzJO
 1Z$IBSN

    !66
  67. @lisa_shashkova Typing Tools w 1Z"OOPUBUFCZ%SPQCPY w .POLFZ5ZQFCZ*OTUBHSBN w l$PMMFDU3VOUJNFJOGPSNBUJPOzJO
 1Z$IBSN

    !67
  68. @lisa_shashkova Typing Tools w 1Z"OOPUBUFCZ%SPQCPY w .POLFZ5ZQFCZ*OTUBHSBN w l$PMMFDU3VOUJNFJOGPSNBUJPOzJO
 1Z$IBSN

    !68
  69. @lisa_shashkova Typing Tools w 1Z"OOPUBUFCZ%SPQCPY w .POLFZ5ZQFCZ*OTUBHSBN w l$PMMFDU3VOUJNFJOGPSNBUJPOzJO
 1Z$IBSN

    !69
  70. @lisa_shashkova Typing Tools w 1Z"OOPUBUFCZ%SPQCPY w .POLFZ5ZQFCZ*OTUBHSBN w l$PMMFDU3VOUJNFJOGPSNBUJPOzJO
 1Z$IBSN

    !70
  71. @lisa_shashkova Acces To Frame w 1Z"OOPUBUF .POLFZ5ZQF • sys.setprofile(profilefunc) •

    profilefunc(frame, event, arg) !71
  72. @lisa_shashkova Acces To Frame w l$PMMFDU3VOUJNFJOGPSNBUJPOzJO1Z$IBSN w *OUFHSBUFEXJUI%FCVHHFS w "DDFTTUPBGSBNFPCKFDU

    !72
  73. @lisa_shashkova Collecting Types !73 def arg_names(co): nargs = co.co_argcount names

    = co.co_varnames return list(names[:nargs]) names = arg_names(frame.f_code)
  74. @lisa_shashkova Collecting Types !74 def arg_names(co): nargs = co.co_argcount names

    = co.co_varnames return list(names[:nargs]) names = arg_names(frame.f_code) locs = frame.f_locals objects = [locs[n] for n in names]
  75. @lisa_shashkova Typing Tools w 1Z"OOPUBUFCZ%SPQCPY w .POLFZ5ZQFCZ*OTUBHSBN w l$PMMFDU3VOUJNFJOGPSNBUJPOzJO
 1Z$IBSN

    !75
  76. @lisa_shashkova Development Tools w "TTFSUJPO&SSPSJOpytest w %FCVHHFS w $PEFDPWFSBHF w

    3VOUJNFUZQJOHUPPMT w !76
  77. @lisa_shashkova Concurrent Execution w 5ISFBET w "TZOD5BTLT !77

  78. @lisa_shashkova Python Threads !78 import threading def fun(): print("Hello!") t

    = threading.Thread(target=fun) t.start() t.join()
  79. @lisa_shashkova Synchronisation w -PDLGVOEBNFOUBMTZODISPOJTBUJPOPCKFDU !79 lock = threading.Lock() lock.acquire() #

    only one thread here lock.release() with lock: # equivalent
  80. @lisa_shashkova Deadlock w 8BJUJOHGPSSFTPVSDFTXIJDIDBO`UCFSFMFBTFE !80

  81. @lisa_shashkova Deadlock !81 def run1(): with lock1: with lock2: #

    do sth def run2(): with lock2: with lock1: # do sth else Thread(target=run1).start() Thread(target=run2).start()
  82. @lisa_shashkova Deadlock w 8BJUJOHGPSSFTPVSDFTXIJDIDBO`UCFSFMFBTFE w )BSEUPEFUFDUJOCJHQSPKFDUT !82

  83. @lisa_shashkova Thread States •sys._getframe() - GSBNFPCKFDUGPSDVSSFOUUISFBE •sys._current_frames() - UPQNPTUTUBDLGSBNFGPSFBDIUISFBE !83

  84. @lisa_shashkova Thread Handler w 1SJOUUSBDFCBDLTGPSUISFBETXJUIJOUFSWBM w )FMQUPpOEEFBEMPDLMPDBUJPO !84

  85. @lisa_shashkova Fault Handler •faulthandler.dump_traceback(file) •%VNQTUIFUSBDFCBDLTPGBMMUISFBETJOGPpMF •*NQMFNFOUFEOBUJWFMZ !85

  86. @lisa_shashkova Concurrent Execution w 5ISFBET w "TZOD5BTLT !86

  87. @lisa_shashkova Async Locks !87 alock = asyncio.Lock() alock.acquire() # only

    one task here alock.release() async with alock: # equivalent
  88. @lisa_shashkova Async Fault Handler •asyncio.all_tasks(loop) - BMMUIFSVOOJOHUBTLT •Task.get_stack() - MJTUPGTUBDLGSBNFTGPSUIJT5BTL

    !88
  89. @lisa_shashkova Async Fault Handler w *OBTFQBSBUFUISFBE !89 def dump_traceback_later(timeout, loop):

    while True: sleep(timeout) dump_traceback(loop, timeout)
  90. @lisa_shashkova Async Fault Handler w *OBTFQBSBUFUISFBE !90 def dump_traceback(loop): for

    task in asyncio.all_tasks(loop): task.print_stack() def dump_traceback_later(timeout, loop): while True: sleep(timeout) dump_traceback(loop, timeout)
  91. @lisa_shashkova Development Tools w "TTFSUJPO&SSPSJOpytest w %FCVHHFS w $PEFDPWFSBHF w

    3VOUJNFUZQJOHUPPMT w "TZOD 'BVMU)BOEMFS !91
  92. @lisa_shashkova The Hidden Power of Runtime w 1ZUIPO3VOUJNFJTWFSZQPXFSGVM w &BTZBDDFTTUPTUBDLGSBNFBOEDPEFPCKFDUT

    w %FWFMPQNFOU5PPMT w QZUFTU %FCVHHFS $PEF$PWFSBHF 
 5ZQJOH*OGPSNBUJPO 'BVMU
 )BOEMFS !92
  93. @lisa_shashkova Inspiration w 6TFFYJTUJOH3VOUJNF%FWFMPQNFOU5PPMT NPSFPGUFO  w $SFBUFTPNFUIJOHOFX !93

  94. @lisa_shashkova Links w IUUQTHJUIVCDPN&MJ[BWFUB1Z3VOUJNF5SJDLT w IUUQTEPDTQZUIPOPSH w FMJ[BWFUBTIBTILPWB!KFUCSBJOTDPN w !MJTB@TIBTILPWB

    !94