I will never restart! Automatic hot reloading in CPython

Slide 1

Slide 1 text

I will never restart! Automatic hot reloading in CPython Raphael Gaschignard MakeLeaps

Slide 2

Slide 2 text

Hi Everyone!

Slide 3

Slide 3 text

About Me • 5 years Python experience w/ MakeLeaps • Heavy Django usage • Top Emoji     ✔   (According To Slack)

Slide 4

Slide 4 text

Python is great! • Easy to get started • Can quickly prototype software • Principle of least surprise. No weird magic

Slide 5

Slide 5 text

My Project • I’m making a universe • It has humans in it

Slide 6

Slide 6 text

# universe.py class Human: def __init__(self, name): self.name = name def greet(self): print(f"Hello, I am {self.naem}") def yell(self, msg: str): print(msg.capitalize() + "!")

Slide 7

Slide 7 text

# main.py import universe jane = universe.Human("Jane") while True: msg = input("?") jane.yell(msg)

Slide 8

Slide 8 text

# universe.py class Human: def __init__(self, name): self.name = name def greet(self): print(f"Hello, I am {self.naem}") def yell(self, msg: str): print(msg.capitalize() + "!") # main.py import universe jane = universe.Human("Jane") while True: msg = input("?") jane.yell(msg)

Slide 9

Slide 9 text

>>> python main.py ?hello Hello! ?all caps All caps!

Slide 10

Slide 10 text

Debugging • I write tests • They are slow • They fail • They fail a lot

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Let me change the code! • Lots of mistakes when coding are debugged quickly • They are also often easily recoverable • Python has stuﬀ like `eval` so… there should be a way for me to hot swap code nicely

Slide 13

Slide 13 text

Let’s Web Search

Slide 14

Slide 14 text

Quite Popular https://stackoverﬂow.com/a/3862885/122757

Slide 15

Slide 15 text

importlib.reload(module) Reload a previously imported module. The argument must be a module object, so it must have been successfully imported before. This is useful if you have edited the module source ﬁle using an external editor and want to try out the new version without leaving the Python interpreter. The return value is the module object https://docs.python.org/3/library/importlib.html#importlib.reload

Slide 16

Slide 16 text

Takeaways • Search the internet for your programming problems, • you will ﬁnd solutions • Use importlib.reload, seems pretty cool

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Hmm…

Slide 19

Slide 19 text

Code Example! import importlib import universe jane = universe.Human("Jane") while True: msg = input("?") jane.yell(msg) importlib.reload(universe)

Slide 20

Slide 20 text

Sanity Check # In universe.py # verify the file is being re-loaded print("Loaded universe.py")

Slide 21

Slide 21 text

>>> python _main.py ?Loaded universe.py ! Loaded universe.py ?hello Hello! Loaded universe.py ?all caps All caps! Loaded universe.py ?

Slide 22

Slide 22 text

# universe.py class Human: def __init__(self, name): self.name = name def greet(self): print(f"Hello, I am {self.naem}") def yell(self, msg: str): print(msg.capitalize() + "!")

Slide 23

Slide 23 text

>>> s = "this is a test" >>> s.capitalize() 'This is a test' >>> s.upper() 'THIS IS A TEST' def yell(self, msg: str): - print(msg.capitalize() + "!") + print(msg.upper() + "!")

Slide 24

Slide 24 text

Loaded universe.py ?all caps All caps! Loaded universe.py ? ! Loaded universe.py ? ! Loaded universe.py ?all caps All caps!

Slide 25

Slide 25 text

Hmm….. jane = universe.Human("Jane") jane.yell("tiny yell") # Tiny yell! # Fix file, and reload importlib.reload(universe) janet = universe.Human("Janet") janet.yell("big yell") # BIG YELL!

Slide 26

Slide 26 text

jane = universe.Human("Jane") importlib.reload(universe) print(isinstance(jane, universe.Human)) # False print(jane.__class__, universe.Human) # print(id(jane.__class__), id(universe.Human)) # 140661652388944 140661651209344

Slide 27

Slide 27 text

importlib.reload First show module state + instance state Then show what importlib.reload ends up doing

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

“Reload a Module” • In Python, most stuﬀ is “just data” • Modules are basically fancy dictionaries • Reloading code just means building new objects from source • Reloading a module just updates a dictionary • No real notion of “updating class methods”

Slide 30

Slide 30 text

And Now For Something Completely Different

Slide 31

Slide 31 text

jane.name • Objects are basically a dictionary • arbitrary attribute assignment • Attributes of an instance are stored on __dict__ >>> from pprint import pprint >>> pprint(jane.__dict__) {'name': 'Jane'}

Slide 32

Slide 32 text

Classes are objects • Classes also store attributes in a dictionary-like object >>> pprint(universe.Human.__dict__) mappingproxy( {'__dict__': , '__doc__': None, ‘__init__': , '__module__': 'universe', ‘__weakref__’: , ‘greet’: , 'yell': })

Slide 33

Slide 33 text

Attribute Resolution • When looking up an attribute on an instance (roughly) • Check class for data descriptors (__get__ and __set__) • Check instance for attribute • Check class for non-data descriptor or attribute https://medium.com/stepping-through-the-cpython-interpreter/how-does-attribute- access-work-d19371898fee

Slide 34

Slide 34 text

jane.name

Slide 35

Slide 35 text

jane.yell

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

“Reloading the class” • We basically want to update class methods (class __dict__) • We want to leave instance attributes alone (instance dict) • If we can update the class __dict__ we win!

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

Resolution Path (reloaded) Show the state after you have the importlib.reload

Slide 40

Slide 40 text

class OldClass: def f(self): return 1 class NewClass: def f(self): return 2 obj = OldClass() assert obj.f() == 1 OldClass.__dict__ = NewClass.__dict__ assert obj.f() == 2

Slide 41

Slide 41 text

Traceback (most recent call last): File "_test_replacement.py", line 11, in OldClass.__dict__ = NewClass.__dict__ AttributeError: attribute '__dict__' of 'type' objects is not writable

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

Plan B • Python runs on my computer • Code is data, data is in memory • C allows for arbitrary memory access • Force CPython to accept my trick through a C Extension

Slide 44

Slide 44 text

static PyObject* replace_type(PyObject* self, PyObject* args){ PyTypeObject* fst; PyTypeObject* snd; if(!PyArg_UnpackTuple(args, "ref", 2,2, &fst, &snd)) { return PyUnicode_FromString("failed unpack"); } PyTypeObject* old_fst = PyObject_GC_New(PyTypeObject, &PyType_Type); // MEMORY LEAK HERE, this is a const char* copy_old(tp_name); copy_old(tp_basicsize); copy_old(tp_itemsize); […]

Slide 45

Slide 45 text

copy_old(tp_dealloc); copy_old(tp_print); copy_old(tp_getattr); copy_old(tp_setattr); copy_old(tp_as_async); copy_old(tp_repr); copy_old(tp_as_number); copy_old(tp_as_sequence); copy_old(tp_as_mapping); copy_old(tp_hash); copy_old(tp_call); copy_old(tp_str); copy_old(tp_getattro); copy_old(tp_setattro); copy_old(tp_as_buffer); copy_old(tp_flags); copy_old(tp_doc); copy_old(tp_traverse); copy_old(tp_clear); copy_old(tp_richcompare); ; copy_old(tp_del);

Slide 46

Slide 46 text

copy_old(tp_weaklistoffset); copy_old(tp_iter); copy_old(tp_iternext); copy_old(tp_methods); copy_old(tp_members); copy_old(tp_getset); copy_old(tp_base); copy_old(tp_dict); copy_old(tp_descr_get); copy_old(tp_descr_set); copy_old(tp_dictoffset); copy_old(tp_init); copy_old(tp_alloc); copy_old(tp_new);

Slide 47

Slide 47 text

Py_XINCREF(fst); Py_XINCREF(snd); Py_XINCREF(old_fst);

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

Loaded universe.py ?not working Not working! Loaded universe.py REPLACING ELT Human ?working? Working?! Loaded universe.py REPLACING ELT Human ?Working? WORKING?! Loaded universe.py REPLACING ELT Human ?

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

Ergonomics • Through module reloading, you can edit class methods at runtime and have it be immediately reflected • Don't want to write something for each module • Ideal flow involves watching all files and reloading automatically

Slide 52

Slide 52 text

from mighty_patcher.watch import AutoReloader from universe import Human reloader = AutoReloader( path="/Users/rtpg/proj/pycon-test/" ) jane = Human("Jane") while True: msg = input("?") jane.yell(msg)

Slide 53

Slide 53 text

Demo

Slide 54

Slide 54 text

Library •mighty_patcher • Provides a one-line ﬁle watcher to hot reload modules • Supports most object types, experimental function support    from mighty_patcher.watch import AutoReloader reloader = AutoReloader( path="/path/to/my/project/src/" )

Slide 55

Slide 55 text

Pytest Plugin • Library has built-in pytest plugin support • pytest test_directory --reload-loop • Activates debugger on test failures • Sets up code reloader • Retry test after code reloading • No instrumentation required

Slide 56

Slide 56 text

Future Work • Make function reloading more reliable/less crash-y • CPython extension debugging • Add some tricks to work around common framework issues • (Django registry pattern) • (Holy Grail) Be able to edit the call stack to recover at _exactly_ the error call site

Slide 57

Slide 57 text

Future Reading • Python Descriptor HowTo • https://docs.python.org/3/howto/descriptor.html • Object Models • https://eev.ee/blog/2017/11/28/object-models/ • "The language also goes to almost superhuman lengths to expose all of its moving parts. Even the prototypical behavior is an implementation of __getattribute__ somewhere, which you are free to completely replace in your own types. ”

Slide 58

Slide 58 text

Takeaways • Python exposes a lot of the inner workings • You should take advantage of it • even if it doesn’t always work • Remember that your computer is a machine

Slide 59

Slide 59 text

Appendix

Slide 60

Slide 60 text

Method Descriptor class Function(object): . . . def __get__(self, obj, objtype=None): if obj is None: return self return types.MethodType(self, obj) https://docs.python.org/3/howto/descriptor.html

Slide 61

Slide 61 text

Principle Of Least Surprise def f(): x = 2 g() print(f"The value of x is {x}!") f() # The value of x is 3!

Slide 62

Slide 62 text

import sys import ctypes def g(): upper_frame = sys._getframe().f_back upper_frame.f_locals['x'] = 3 ctypes.pythonapi.PyFrame_LocalsToFast( ctypes.py_object(upper_frame), ctypes.c_int(0) ) def f(): x = 2 g() print(f"The value of x is {x}!") f() # The value of x is 3!