I will never restart! Automatic hot reloading in CPython

Transcript

1. I will never restart! Automatic hot reloading in CPython Raphael Gaschignard MakeLeaps
2. Hi Everyone!
3. About Me • 5 years Python experience w/ MakeLeaps • Heavy Django usage • Top Emoji 
 
 ✔ 
 (According To Slack)
4. Python is great! • Easy to get started • Can quickly prototype software • Principle of least surprise. No weird magic
5. My Project • I’m making a universe • It has humans in it
6. # universe.py class Human: def __init__(self, name): self.name = name def greet(self): print(f"Hello, I am {self.naem}") def yell(self, msg: str): print(msg.capitalize() + "!")
7. # main.py import universe jane = universe.Human("Jane") while True: msg = input("?") jane.yell(msg)
8. # universe.py class Human: def __init__(self, name): self.name = name def greet(self): print(f"Hello, I am {self.naem}") def yell(self, msg: str): print(msg.capitalize() + "!") # main.py import universe jane = universe.Human("Jane") while True: msg = input("?") jane.yell(msg)
9. >>> python main.py ?hello Hello! ?all caps All caps!
10. Debugging • I write tests • They are slow • They fail • They fail a lot
12. Let me change the code! • Lots of mistakes when coding are debugged quickly • They are also often easily recoverable • Python has stuff like `eval` so… there should be a way for me to hot swap code nicely
13. Let’s Web Search
14. Quite Popular https://stackoverflow.com/a/3862885/122757
15. importlib.reload(module) Reload a previously imported module. The argument must be a module object, so it must have been successfully imported before. This is useful if you have edited the module source file using an external editor and want to try out the new version without leaving the Python interpreter. The return value is the module object https://docs.python.org/3/library/importlib.html#importlib.reload
16. Takeaways • Search the internet for your programming problems, • you will find solutions • Use importlib.reload, seems pretty cool
18. Hmm…
19. Code Example! import importlib import universe jane = universe.Human("Jane") while True: msg = input("?") jane.yell(msg) importlib.reload(universe)
20. Sanity Check # In universe.py # verify the file is being re-loaded print("Loaded universe.py")
21. >>> python _main.py ?Loaded universe.py ! Loaded universe.py ?hello Hello! Loaded universe.py ?all caps All caps! Loaded universe.py ?
22. # universe.py class Human: def __init__(self, name): self.name = name def greet(self): print(f"Hello, I am {self.naem}") def yell(self, msg: str): print(msg.capitalize() + "!")
23. >>> s = "this is a test" >>> s.capitalize() 'This is a test' >>> s.upper() 'THIS IS A TEST' def yell(self, msg: str): - print(msg.capitalize() + "!") + print(msg.upper() + "!")
24. Loaded universe.py ?all caps All caps! Loaded universe.py ? ! Loaded universe.py ? ! Loaded universe.py ?all caps All caps!
25. Hmm….. jane = universe.Human("Jane") jane.yell("tiny yell") # Tiny yell! # Fix file, and reload importlib.reload(universe) janet = universe.Human("Janet") janet.yell("big yell") # BIG YELL!
26. jane = universe.Human("Jane") importlib.reload(universe) print(isinstance(jane, universe.Human)) # False print(jane.__class__, universe.Human) # <class 'universe.Human'> <class 'universe.Human'> print(id(jane.__class__), id(universe.Human)) # 140661652388944 140661651209344
27. importlib.reload First show module state + instance state Then show what importlib.reload ends up doing
29. “Reload a Module” • In Python, most stuff is “just data” • Modules are basically fancy dictionaries • Reloading code just means building new objects from source • Reloading a module just updates a dictionary • No real notion of “updating class methods”
30. And Now For Something Completely Different
31. jane.name • Objects are basically a dictionary • arbitrary attribute assignment • Attributes of an instance are stored on __dict__ >>> from pprint import pprint >>> pprint(jane.__dict__) {'name': 'Jane'}
32. Classes are objects • Classes also store attributes in a dictionary-like object >>> pprint(universe.Human.__dict__) mappingproxy( {'__dict__': <attribute '__dict__' of 'Human' objects>, '__doc__': None, ‘__init__': <function Human.__init__ at 0x108a10320>, '__module__': 'universe', ‘__weakref__’: <attribute '__weakref__' of 'Human' objects>, ‘greet’: <function Human.greet at 0x108a10dd0>, 'yell': <function Human.yell at 0x108a10d40>})
33. Attribute Resolution • When looking up an attribute on an instance (roughly) • Check class for data descriptors (__get__ and __set__) • Check instance for attribute • Check class for non-data descriptor or attribute https://medium.com/stepping-through-the-cpython-interpreter/how-does-attribute- access-work-d19371898fee
34. jane.name
35. jane.yell
37. “Reloading the class” • We basically want to update class methods (class __dict__) • We want to leave instance attributes alone (instance dict) • If we can update the class __dict__ we win!
39. Resolution Path (reloaded) Show the state after you have the importlib.reload
40. class OldClass: def f(self): return 1 class NewClass: def f(self): return 2 obj = OldClass() assert obj.f() == 1 OldClass.__dict__ = NewClass.__dict__ assert obj.f() == 2
41. Traceback (most recent call last): File "_test_replacement.py", line 11, in <module> OldClass.__dict__ = NewClass.__dict__ AttributeError: attribute '__dict__' of 'type' objects is not writable
43. Plan B • Python runs on my computer • Code is data, data is in memory • C allows for arbitrary memory access • Force CPython to accept my trick through a C Extension
44. static PyObject* replace_type(PyObject* self, PyObject* args){ PyTypeObject* fst; PyTypeObject* snd; if(!PyArg_UnpackTuple(args, "ref", 2,2, &fst, &snd)) { return PyUnicode_FromString("failed unpack"); } PyTypeObject* old_fst = PyObject_GC_New(PyTypeObject, &PyType_Type); // MEMORY LEAK HERE, this is a const char* copy_old(tp_name); copy_old(tp_basicsize); copy_old(tp_itemsize); […]
45. copy_old(tp_dealloc); copy_old(tp_print); copy_old(tp_getattr); copy_old(tp_setattr); copy_old(tp_as_async); copy_old(tp_repr); copy_old(tp_as_number); copy_old(tp_as_sequence); copy_old(tp_as_mapping); copy_old(tp_hash); copy_old(tp_call); copy_old(tp_str); copy_old(tp_getattro); copy_old(tp_setattro); copy_old(tp_as_buffer); copy_old(tp_flags); copy_old(tp_doc); copy_old(tp_traverse); copy_old(tp_clear); copy_old(tp_richcompare); ; copy_old(tp_del);
46. copy_old(tp_weaklistoffset); copy_old(tp_iter); copy_old(tp_iternext); copy_old(tp_methods); copy_old(tp_members); copy_old(tp_getset); copy_old(tp_base); copy_old(tp_dict); copy_old(tp_descr_get); copy_old(tp_descr_set); copy_old(tp_dictoffset); copy_old(tp_init); copy_old(tp_alloc); copy_old(tp_new);
47. Py_XINCREF(fst); Py_XINCREF(snd); Py_XINCREF(old_fst);
49. Loaded universe.py ?not working Not working! Loaded universe.py REPLACING ELT Human ?working? Working?! Loaded universe.py REPLACING ELT Human ?Working? WORKING?! Loaded universe.py REPLACING ELT Human ?
51. Ergonomics • Through module reloading, you can edit class methods at runtime and have it be immediately reflected • Don't want to write something for each module • Ideal flow involves watching all files and reloading automatically
52. from mighty_patcher.watch import AutoReloader from universe import Human reloader = AutoReloader( path="/Users/rtpg/proj/pycon-test/" ) jane = Human("Jane") while True: msg = input("?") jane.yell(msg)
53. Demo
54. Library •mighty_patcher • Provides a one-line file watcher to hot reload modules • Supports most object types, experimental function support
 
 from mighty_patcher.watch import AutoReloader reloader = AutoReloader( path="/path/to/my/project/src/" )
55. Pytest Plugin • Library has built-in pytest plugin support • pytest test_directory --reload-loop • Activates debugger on test failures • Sets up code reloader • Retry test after code reloading • No instrumentation required
56. Future Work • Make function reloading more reliable/less crash-y • CPython extension debugging • Add some tricks to work around common framework issues • (Django registry pattern) • (Holy Grail) Be able to edit the call stack to recover at _exactly_ the error call site
57. Future Reading • Python Descriptor HowTo • https://docs.python.org/3/howto/descriptor.html • Object Models • https://eev.ee/blog/2017/11/28/object-models/ • "The language also goes to almost superhuman lengths to expose all of its moving parts. Even the prototypical behavior is an implementation of __getattribute__ somewhere, which you are free to completely replace in your own types. ”
58. Takeaways • Python exposes a lot of the inner workings • You should take advantage of it • even if it doesn’t always work • Remember that your computer is a machine
59. Appendix
60. Method Descriptor class Function(object): . . . def __get__(self, obj, objtype=None): if obj is None: return self return types.MethodType(self, obj) https://docs.python.org/3/howto/descriptor.html
61. Principle Of Least Surprise def f(): x = 2 g() print(f"The value of x is {x}!") f() # The value of x is 3!
62. import sys import ctypes def g(): upper_frame = sys._getframe().f_back upper_frame.f_locals['x'] = 3 ctypes.pythonapi.PyFrame_LocalsToFast( ctypes.py_object(upper_frame), ctypes.c_int(0) ) def f(): x = 2 g() print(f"The value of x is {x}!") f() # The value of x is 3!