Tweet
Tweet

Slide 1

Slide 1 text

ABOUT DATE AUTHOR 29 JULY 2012 DAN CROSTA (@LAZLOFRUVOUS) CODE OBJECTS EXPLORING PYTHON

Slide 2

Slide 2 text

"This popular meme is incorrect, or, rather, constructed upon a misunderstanding of (natural) language levels: a similar mistake would be to say 'the Bible is a hardcover book'." "I've been given to understand that Python is an interpreted language..." Alex Martelli, http://stackoverflow.com/questions/2998215

Slide 3

Slide 3 text

CPYTHON • Compiles Python source to "bytecode" • On demand, when modules are loaded • Virtual machine for this bytecode

Slide 4

Slide 4 text

BYTECODE? • About 150 primitive instructions • Associates data with those operations • CPython implements a virtual processor • Stored in .pyc files

Slide 5

Slide 5 text

MAKE YOUR OWN • compile() (we will use this) • Many other ways: • compiler • parser • compileall • py_compile

Slide 6

Slide 6 text

MAKE YOUR OWN >>> code_str = """ ... print "Hello, world" ... """

Slide 7

Slide 7 text

MAKE YOUR OWN >>> code_str = """ ... print "Hello, world" ... """ >>> code_obj = compile( ... code_str, '', 'exec')

Slide 8

Slide 8 text

MAKE YOUR OWN >>> code_str = """ ... print "Hello, world" ... """ >>> code_obj = compile( ... code_str, '', 'exec') >>> code_obj at 0x1054c74b0, file "", line 2>

Slide 9

Slide 9 text

LOOK INSIDE >>> code_obj.co_filename '' >>> code_obj.co_name '' >>> dir(code_obj) ['co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames']

Slide 10

Slide 10 text

MAKE IT GO >>> code_obj() Traceback (most recent call last): File "", line 1, in TypeError: 'code' object is not callable

Slide 11

Slide 11 text

MAKE IT GO >>> code_obj() Traceback (most recent call last): File "", line 1, in TypeError: 'code' object is not callable >>> exec code_obj Hello, world

Slide 12

Slide 12 text

MAKE IT GO >>> code_str = """ ... tmp = max(a, b) ... out = tmp + 1 ... """ >>> code_obj = compile( ... code_str, '', 'exec')

Slide 13

Slide 13 text

MAKE IT GO >>> code_str = """ ... tmp = max(a, b) ... out = tmp + 1 ... """ >>> code_obj = compile( ... code_str, '', 'exec') >>> myglobals = {} >>> mylocals = {'a': 5, 'b': 10} >>> exec code_obj in myglobals, mylocals >>> mylocals['out'] 11

Slide 14

Slide 14 text

This is probably a bad idea. WARNING:

Slide 15

Slide 15 text

MAKE IT GO >>> code_str = """ ... tmp = max(a, b) ... out = tmp + 1 ... """ >>> code_obj = compile( ... code_str, '', 'exec') >>> code_obj.co_consts (1, None) >>> code_obj.co_names ('max', 'a', 'b', 'tmp', 'out') >>> code_obj.co_stacksize 3

Slide 16

Slide 16 text

UNDER THE HOOD >>> dis.dis(code_obj) 2 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 3 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE

Slide 17

Slide 17 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' =>

Slide 18

Slide 18 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' =>

Slide 19

Slide 19 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' =>

Slide 20

Slide 20 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' =>

Slide 21

Slide 21 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => 5

Slide 22

Slide 22 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => 5

Slide 23

Slide 23 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => 5 10

Slide 24

Slide 24 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => 5 10

Slide 25

Slide 25 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => 5

Slide 26

Slide 26 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' =>

Slide 27

Slide 27 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' =>

Slide 28

Slide 28 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => 10

Slide 29

Slide 29 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 'out' => 10

Slide 30

Slide 30 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' =>

Slide 31

Slide 31 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' =>

Slide 32

Slide 32 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 10

Slide 33

Slide 33 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 10

Slide 34

Slide 34 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 10 1

Slide 35

Slide 35 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 10 1

Slide 36

Slide 36 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 10

Slide 37

Slide 37 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' =>

Slide 38

Slide 38 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11

Slide 39

Slide 39 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11

Slide 40

Slide 40 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11

Slide 41

Slide 41 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11

Slide 42

Slide 42 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11 None

Slide 43

Slide 43 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11 None

Slide 44

Slide 44 text

UNDER THE HOOD 0 LOAD_NAME 0 (max) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 CALL_FUNCTION 2 12 STORE_NAME 3 (tmp) 15 LOAD_NAME 3 (tmp) 18 LOAD_CONST 0 (1) 21 BINARY_ADD 22 STORE_NAME 4 (out) 25 LOAD_CONST 1 (None) 28 RETURN_VALUE locals: 'a' => 5 'b' => 10 'tmp' => 10 'out' => 11

Slide 45

Slide 45 text

C Ahead! WARNING:

Slide 46

Slide 46 text

CEVAL.C PyObject * PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) { PyObject *retval = NULL; PyCodeObject *co = f->f_code; for (;;) { opcode = NEXTOP(); switch (opcode) { /* eventually retval is set */ } if (retval) break; } return retval; }

Slide 47

Slide 47 text

CEVAL.C case BINARY_ADD: w = POP(); v = TOP(); if (PyInt_CheckExact(v) && PyInt_CheckExact(w)) { long a, b, i; a = PyInt_AS_LONG(v); b = PyInt_AS_LONG(w); /* cast to avoid undefined behaviour on overflow */ i = (long)((unsigned long)a + b); x = PyInt_FromLong(i); } Py_DECREF(v); Py_DECREF(w); SET_TOP(x);

Slide 48

Slide 48 text

Code objects in code objects!

Slide 49

Slide 49 text

CLASSES ARE CODE, TOO >>> def factory(): ... class MyClass(object): ... def method(self, arg): ... return arg ... return MyClass ... >>> dis.dis(factory.func_code) 2 0 LOAD_CONST 1 ('MyClass') 3 LOAD_GLOBAL 0 (object) 6 BUILD_TUPLE 1 9 LOAD_CONST 2 () 12 MAKE_FUNCTION 0 15 CALL_FUNCTION 0 18 BUILD_CLASS 19 STORE_FAST 0 (MyClass) 5 22 LOAD_FAST 0 (MyClass) 25 RETURN_VALUE

Slide 50

Slide 50 text

CLASSES ARE CODE, TOO >>> dis.dis(factory.func_code.co_consts[2]) 2 0 LOAD_NAME 0 (__name__) 3 STORE_NAME 1 (__module__) 3 6 LOAD_CONST 0 () 9 MAKE_FUNCTION 0 12 STORE_NAME 2 (method) 15 LOAD_LOCALS 16 RETURN_VALUE >>> dis.dis(factory.func_code.co_consts[2].co_consts[0]) 4 0 LOAD_FAST 1 (arg) 3 RETURN_VALUE

Slide 51

Slide 51 text

And closures, how the f*%!$ do they work?

Slide 52

Slide 52 text

CLOSURES >>> def outer(x): ... def inner(y): ... return x + y ... return inner ... >>> dis.dis(outer) 2 0 LOAD_CLOSURE 0 (x) 3 BUILD_TUPLE 1 6 LOAD_CONST 1 () 9 MAKE_CLOSURE 0 12 STORE_FAST 1 (inner) 4 15 LOAD_FAST 1 (inner) 18 RETURN_VALUE

Slide 53

Slide 53 text

CLOSURES >>> inner = outer(1) >>> dis.dis(inner) 3 0 LOAD_DEREF 0 (x) 3 LOAD_FAST 0 (y) 6 BINARY_ADD 7 RETURN_VALUE

Slide 54

Slide 54 text

CLOSURES >>> inner = outer(1) >>> dis.dis(inner) 3 0 LOAD_DEREF 0 (x) 3 LOAD_FAST 0 (y) 6 BINARY_ADD 7 RETURN_VALUE >>> inner.func_closure (,) >>> inner.func_closure[0].cell_contents 1

Slide 55

Slide 55 text

ZOOMING OUT • I hope this was interesting • This is all only for CPython (2.x)! • Go exploring, share what you find • Questions?