Exploring Python Code Objects

Python is an interpreted language, right? Wrong! In this talk, dive deep into Python bytecode, and learn what actually happens in everyone's favorite Python program, 'print "Hello world"'. Learn to use the compile() builtin and its best friend the exec statement, understand what your Python code is doing with the dis and compiler modules, and discover new ways to explore and enjoy Python at a low level.

PyGotham II, June 8, 2012.

"The Known Universe" video from American Museum of Natural History and the Hayden Planetarium. See http://www.youtube.com/watch?v=17jymDn0W6U.

Transcript

1. ABOUT DATE AUTHOR 8 JUNE 2012 DAN CROSTA (@LAZLOFRUVOUS) CODE OBJECTS EXPLORING PYTHON
2. "This popular meme is incorrect, or, rather, constructed upon a misunderstanding of (natural) language levels: a similar mistake would be to say 'the Bible is a hardcover book'." "I've been given to understand that Python is an interpreted language..." Alex Martelli, http://stackoverflow.com/questions/2998215
3. CPYTHON • Compiles Python source to "bytecode" • On demand, when modules are loaded • Virtual machine for this bytecode
4. BYTECODE? • About 150 primitive instructions • Associates data with those operations • CPython implements a virtual processor • Stored in .pyc files
5. MAKE YOUR OWN • compile() (we will use this) • Many other ways: • compiler • parser • compileall • py_compile
6. MAKE YOUR OWN >>> code_str = """ ... print "Hello, world" ... """
7. MAKE YOUR OWN >>> code_str = """ ... print "Hello, world" ... """ >>> code_obj = compile( ... code_str, '<string>', 'exec')
8. MAKE YOUR OWN >>> code_str = """ ... print "Hello, world" ... """ >>> code_obj = compile( ... code_str, '<string>', 'exec') >>> code_obj <code object <module> at 0x1054c74b0, file "<string>", line 2>
9. LOOK INSIDE >>> code_obj.co_filename '<string>' >>> code_obj.co_name '<module>' >>> dir(code_obj) ['co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames']
10. MAKE IT GO >>> code_obj() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'code' object is not callable
11. MAKE IT GO >>> code_obj() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'code' object is not callable >>> exec code_obj Hello, world
12. MAKE IT GO >>> code_obj() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'code' object is not callable >>> exec code_obj Hello, world
13. MAKE IT GO >>> code_str = """ ... x = 1 ... y = x + 1 ... z = x + y ... """ >>> code_obj = compile( ... code_str, '<string>', 'exec')
14. MAKE IT GO >>> code_str = """ ... x = 1 ... y = x + 1 ... z = x + y ... """ >>> code_obj = compile( ... code_str, '<string>', 'exec') >>> myglobals = {} >>> mylocals = {} >>> exec code_obj in myglobals, mylocals >>> mylocals['z'] 3
15. This is probably a bad idea. WARNING:
16. UNDER THE HOOD >>> code_str = """ ... x = 1 ... y = x + 1 ... z = x + y ... """ >>> code_obj = compile( ... code_str, '<string>', 'exec') >>> code_obj.co_consts (1, None) >>> code_obj.co_names ('x', 'y', 'z') >>> code_obj.co_stacksize 2
17. UNDER THE HOOD >>> dis.dis(code_obj) 2 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 3 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 4 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE
18. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => undefined 'y' => undefined 'z' => undefined
19. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => undefined 'y' => undefined 'z' => undefined
20. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => undefined 'y' => undefined 'z' => undefined 1
21. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => undefined 'y' => undefined 'z' => undefined 1
22. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => undefined 'z' => undefined
23. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => undefined 'z' => undefined
24. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => undefined 'z' => undefined 1
25. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => undefined 'z' => undefined 1
26. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => undefined 'z' => undefined 1 1
27. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => undefined 'z' => undefined 1 1
28. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => undefined 'z' => undefined 2
29. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => undefined 'z' => undefined 2
30. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => undefined
31. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => undefined
32. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => undefined 1
33. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => undefined 1
34. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => undefined 1 2
35. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => undefined 1 2
36. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => undefined 3
37. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => undefined 3
38. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => 3
39. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => 3
40. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => 3 None
41. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => 3 None
42. UNDER THE HOOD 0 LOAD_CONST 0 (1) 3 STORE_NAME 0 (x) 6 LOAD_NAME 0 (x) 9 LOAD_CONST 0 (1) 12 BINARY_ADD 13 STORE_NAME 1 (y) 16 LOAD_NAME 0 (x) 19 LOAD_NAME 1 (y) 22 BINARY_ADD 23 STORE_NAME 2 (z) 26 LOAD_CONST 1 (None) 29 RETURN_VALUE locals: 'x' => 1 'y' => 2 'z' => 3
43. C Ahead! WARNING:
44. CEVAL.C PyObject * PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) { PyObject *retval = NULL; PyCodeObject *co; co = f->f_code; for (;;) { opcode = NEXTOP(); switch (opcode) { /* eventually retval is set */ } } return retval; }
45. CEVAL.C case BINARY_ADD: w = POP(); v = TOP(); if (PyInt_CheckExact(v) && PyInt_CheckExact(w)) { long a, b, i; a = PyInt_AS_LONG(v); b = PyInt_AS_LONG(w); /* cast to avoid undefined behaviour on overflow */ i = (long)((unsigned long)a + b); x = PyInt_FromLong(i); } Py_DECREF(v); Py_DECREF(w); SET_TOP(x);
46. Code objects in code objects!
47. CLASSES ARE CODE, TOO >>> def factory(): ... class MyClass(object): ... def method(self, arg): ... return arg ... return MyClass ... >>> dis.dis(factory) 2 0 LOAD_CONST 1 ('MyClass') 3 LOAD_GLOBAL 0 (object) 6 BUILD_TUPLE 1 9 LOAD_CONST 2 (<code object MyClass>) 12 MAKE_FUNCTION 0 15 CALL_FUNCTION 0 18 BUILD_CLASS 19 STORE_FAST 0 (MyClass) 5 22 LOAD_FAST 0 (MyClass) 25 RETURN_VALUE
48. CLASSES ARE CODE, TOO >>> dis.dis(factory.func_code.co_consts[2]) 2 0 LOAD_NAME 0 (__name__) 3 STORE_NAME 1 (__module__) 3 6 LOAD_CONST 0 (<code object method>) 9 MAKE_FUNCTION 0 12 STORE_NAME 2 (method) 15 LOAD_LOCALS 16 RETURN_VALUE >>> dis.dis(factory.func_code.co_consts[2].co_consts[0]) 4 0 LOAD_FAST 1 (arg) 3 RETURN_VALUE
49. And closures, how the f*%!$ do they work?
50. CLOSURES >>> def outer(x): ... def inner(y): ... return x + y ... return inner ... >>> dis.dis(outer) 2 0 LOAD_CLOSURE 0 (x) 3 BUILD_TUPLE 1 6 LOAD_CONST 1 (<code object inner>) 9 MAKE_CLOSURE 0 12 STORE_FAST 1 (inner) 4 15 LOAD_FAST 1 (inner) 18 RETURN_VALUE
51. CLOSURES >>> inner = outer(1) >>> dis.dis(inner) 3 0 LOAD_DEREF 0 (x) 3 LOAD_FAST 0 (y) 6 BINARY_ADD 7 RETURN_VALUE
52. CLOSURES >>> inner = outer(1) >>> dis.dis(inner) 3 0 LOAD_DEREF 0 (x) 3 LOAD_FAST 0 (y) 6 BINARY_ADD 7 RETURN_VALUE >>> inner.func_closure (<cell at 0x1050ef210: int object at 0x104f131b8>,) >>> inner.func_closure[0].cell_contents 1
53. ZOOMING OUT • I hope this was interesting • This is all only for CPython (2.x)! • Go exploring, share what you find • Questions?