Exploring Python Code Objects (PyOhio)

Python is an interpreted language, right? Wrong! In this talk, dive deep into Python bytecode, and learn what actually happens in everyone's favorite Python program, 'print "Hello world"'. Learn to use the compile() and exec statement, understand what your Python code is doing with the dis and compiler modules, and discover new ways to explore and enjoy Python at a low level.

PyOhio, July 29, 2012.

"The Known Universe" video from American Museum of Natural History and the Hayden Planetarium. See http://www.youtube.com/watch?v=17jymDn0W6U.

Transcript

1. ABOUT DATE AUTHOR 29 JULY 2012 DAN CROSTA (@LAZLOFRUVOUS) CODE OBJECTS EXPLORING PYTHON
2. "This popular meme is incorrect, or, rather, constructed upon a misunderstanding of (natural) language levels: a similar mistake would be to say 'the Bible is a hardcover book'." "I've been given to understand that Python is an interpreted language..." Alex Martelli, http://stackoverflow.com/questions/2998215
3. CPYTHON • Compiles Python source to "bytecode" • On demand, when modules are loaded • Virtual machine for this bytecode
4. BYTECODE? • About 150 primitive instructions • Associates data with those operations • CPython implements a virtual processor • Stored in .pyc files
5. MAKE YOUR OWN • compile() (we will use this) • Many other ways: • compiler • parser • compileall • py_compile
6. MAKE YOUR OWN >>> code_str = """ ... print "Hello, world" ... """
7. MAKE YOUR OWN >>> code_str = """ ... print "Hello, world" ... """ >>> code_obj = compile( ... code_str, '<string>', 'exec')
8. MAKE YOUR OWN >>> code_str = """ ... print "Hello, world" ... """ >>> code_obj = compile( ... code_str, '<string>', 'exec') >>> code_obj <code object <module> at 0x1054c74b0, file "<string>", line 2>
9. LOOK INSIDE >>> code_obj.co_filename '<string>' >>> code_obj.co_name '<module>' >>> dir(code_obj) ['co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames']
10. MAKE IT GO >>> code_obj() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'code' object is not callable
11. MAKE IT GO >>> code_obj() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'code' object is not callable >>> exec code_obj Hello, world
12. MAKE IT GO >>> code_str = """ ... tmp = max(a, b) ... out = tmp + 1 ... """ >>> code_obj = compile( ... code_str, '<string>', 'exec')
13. MAKE IT GO >>> code_str = """ ... tmp = max(a, b) ... out = tmp + 1 ... """ >>> code_obj = compile( ... code_str, '<string>', 'exec') >>> myglobals = {} >>> mylocals = {'a': 5, 'b': 10} >>> exec code_obj in myglobals, mylocals >>> mylocals['out'] 11
14. This is probably a bad idea. WARNING:
15. MAKE IT GO >>> code_str = """ ... tmp = max(a, b) ... out = tmp + 1 ... """ >>> code_obj = compile( ... code_str, '<string>', 'exec') >>> code_obj.co_consts (1, None) >>> code_obj.co_names ('max', 'a', 'b', 'tmp', 'out') >>> code_obj.co_stacksize 3
16. UNDER THE HOOD >>> dis.dis(code_obj) 2 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 3 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE
17. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' =>
18. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' =>
19. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => <built-in function max>
20. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => <built-in function max>
21. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => <built-in function max> 5
22. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => <built-in function max> 5
23. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => <built-in function max> 5 10
24. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => <built-in function max> 5 10
25. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => <built-in function max> 5
26. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => <built-in function max>
27. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' =>
28. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => 10
29. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => 10
30. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' =>
31. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' =>
32. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 10
33. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 10
34. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 10 1
35. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 10 1
36. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 10
37. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' =>
38. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11
39. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11
40. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11
41. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11
42. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11 None
43. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11 None
44. UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11
45. C Ahead! WARNING:
46. CEVAL.C PyObject * PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) { PyObject *retval = NULL; PyCodeObject *co = f->f_code; for (;;) { opcode = NEXTOP(); switch (opcode) { /* eventually retval is set */ } if (retval) break; } return retval; }
47. CEVAL.C case BINARY_ADD: w = POP(); v = TOP(); if (PyInt_CheckExact(v) && PyInt_CheckExact(w)) { long a, b, i; a = PyInt_AS_LONG(v); b = PyInt_AS_LONG(w); /* cast to avoid undefined behaviour on overflow */ i = (long)((unsigned long)a + b); x = PyInt_FromLong(i); } Py_DECREF(v); Py_DECREF(w); SET_TOP(x);
48. Code objects in code objects!
49. CLASSES ARE CODE, TOO >>> def factory(): ... class MyClass(object): ... def method(self, arg): ... return arg ... return MyClass ... >>> dis.dis(factory.func_code) 2 0 LOAD_CONST 1 ('MyClass') 3 LOAD_GLOBAL 0 (object) 6 BUILD_TUPLE 1 9 LOAD_CONST 2 (<code object MyClass>) 12 MAKE_FUNCTION 0 15 CALL_FUNCTION 0 18 BUILD_CLASS 19 STORE_FAST 0 (MyClass) 5 22 LOAD_FAST 0 (MyClass) 25 RETURN_VALUE
50. CLASSES ARE CODE, TOO >>> dis.dis(factory.func_code.co_consts[2]) 2 0 LOAD_NAME 0 (__name__) 3 STORE_NAME 1 (__module__) 3 6 LOAD_CONST 0 (<code object method>) 9 MAKE_FUNCTION 0 12 STORE_NAME 2 (method) 15 LOAD_LOCALS 16 RETURN_VALUE >>> dis.dis(factory.func_code.co_consts[2].co_consts[0]) 4 0 LOAD_FAST 1 (arg) 3 RETURN_VALUE
51. And closures, how the f*%!$ do they work?
52. CLOSURES >>> def outer(x): ... def inner(y): ... return x + y ... return inner ... >>> dis.dis(outer) 2 0 LOAD_CLOSURE 0 (x) 3 BUILD_TUPLE 1 6 LOAD_CONST 1 (<code object inner>) 9 MAKE_CLOSURE 0 12 STORE_FAST 1 (inner) 4 15 LOAD_FAST 1 (inner) 18 RETURN_VALUE
53. CLOSURES >>> inner = outer(1) >>> dis.dis(inner) 3 0 LOAD_DEREF 0 (x) 3 LOAD_FAST 0 (y) 6 BINARY_ADD 7 RETURN_VALUE
54. CLOSURES >>> inner = outer(1) >>> dis.dis(inner) 3 0 LOAD_DEREF 0 (x) 3 LOAD_FAST 0 (y) 6 BINARY_ADD 7 RETURN_VALUE >>> inner.func_closure (<cell at 0x1050ef210: int object at 0x104f131b8>,) >>> inner.func_closure[0].cell_contents 1
55. ZOOMING OUT • I hope this was interesting • This is all only for CPython (2.x)! • Go exploring, share what you find • Questions?