Python White Magic - Speaker Deck

Slide 1

Slide 1 text

Python White Magic Antonio Cuni EuroPython 2012 July 3 2012 antocuni (EuroPython 2012) Python White Magic July 3 2012 1 / 47

Slide 2

Slide 2 text

About me PyPy core dev PyPy py3k tech leader pdb++, fancycompleter, ... Consultant, trainer http://antocuni.eu antocuni (EuroPython 2012) Python White Magic July 3 2012 2 / 47

Slide 3

Slide 3 text

About this talk (1) a collection of hacks :-) HOWEVER real-life problems real-life solutions they improve the rest of the code (you might not like them) antocuni (EuroPython 2012) Python White Magic July 3 2012 3 / 47

Slide 4

Slide 4 text

Slide 5

Slide 5 text

About this talk (2) Learn about internals and/or advanced Python function and code objects bytecode metaclasses decorators import logic antocuni (EuroPython 2012) Python White Magic July 3 2012 4 / 47

Slide 6

Slide 6 text

Python Powerful language Surface of simplicity “just works” (TM) as expected simplicity is an emergent property lots of internal rules and layers (ab)use them “black magic” antocuni (EuroPython 2012) Python White Magic July 3 2012 5 / 47

Slide 7

Slide 7 text

Slide 8

Slide 8 text

“Black” magic harder to read harder to maintain might break unexpectedly might rely on implementation details ... antocuni (EuroPython 2012) Python White Magic July 3 2012 6 / 47

Slide 9

Slide 9 text

What is magic? Any sufﬁciently advanced technology is indistinguishable from magic (Arthur C. Clarke) antocuni (EuroPython 2012) Python White Magic July 3 2012 7 / 47

Slide 10

Slide 10 text

What is magic? Any sufﬁciently advanced technology is indistinguishable from magic (Arthur C. Clarke) antocuni (EuroPython 2012) Python White Magic July 3 2012 8 / 47

Slide 11

Slide 11 text

What is magic? Any sufﬁciently advanced technology is indistinguishable from magic (Arthur C. Clarke) antocuni (EuroPython 2012) Python White Magic July 3 2012 9 / 47

Slide 12

Slide 12 text

White magic Still magic can help to have better code more readable more maintainable however: lots of cons some pros solid understanding of concepts is needed use with care antocuni (EuroPython 2012) Python White Magic July 3 2012 10 / 47

Slide 13

Slide 13 text

Problem #1: extending pdb.py (1) pdb++: drop-in replacement for pdb.py keep the same API how extend a module? subclassing is not enouogh module-level functions? antocuni (EuroPython 2012) Python White Magic July 3 2012 11 / 47

Slide 14

Slide 14 text

Problem #1: extending pdb.py (2) pdb.py ... class Pdb(bdb.Bdb, cmd.Cmd): ... def set_trace(): Pdb().set_trace(...) def main(): ... pdb = Pdb() pdb._runscript(...) ... pdbpp.py import pdb class Pdb(pdb.Pdb): ... def set_trace(): pdb.set_trace() # ??? def main(): pdb.main() # ??? ... antocuni (EuroPython 2012) Python White Magic July 3 2012 12 / 47

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Problem #1: extending pdb.py (3) Logic inside set_trace and main (and others) Pdb() refers to pdb.Pdb() our new class is never instantiated copy&paste is not a solution :-) antocuni (EuroPython 2012) Python White Magic July 3 2012 13 / 47

Slide 17

Slide 17 text

Spell #1: rebind_globals (1) python >>> A = ’hello’ >>> def foo(): ... print A ... >>> foo() hello >>> from magic import rebind_globals >>> foo2 = rebind_globals(foo, {’A’: ’ciao’}) >>> A ’hello’ >>> foo() hello >>> foo2() ciao antocuni (EuroPython 2012) Python White Magic July 3 2012 14 / 47

Slide 18

Slide 18 text

Slide 19

Slide 19 text

LOAD_GLOBAL explained func_globals func_code .... antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

Slide 20

Slide 20 text

LOAD_GLOBAL explained func_globals func_code .... { 'foo': , 'A': 'hello', } antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

Slide 21

Slide 21 text

LOAD_GLOBAL explained func_globals func_code .... { 'foo': , 'A': 'hello', }


co_code
....
antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

Slide 22

Slide 22 text

LOAD_GLOBAL explained func_globals func_code .... { 'foo': , 'A': 'hello', }


co_code
....
0 LOAD_GLOBAL 0 (A)
3 PRINT_ITEM
4 PRINT_NEWLINE
5 LOAD_CONST 0 (None)
8 RETURN_VALUE
antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

Slide 23

Slide 23 text

LOAD_GLOBAL explained func_globals func_code .... { 'foo': , 'A': 'hello', }


co_code
....
0 LOAD_GLOBAL 0 (A)
3 PRINT_ITEM
4 PRINT_NEWLINE
5 LOAD_CONST 0 (None)
8 RETURN_VALUE
antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

Slide 24

Slide 24 text

LOAD_GLOBAL explained func_globals func_code .... { 'foo': , 'A': 'hello', }


co_code
....
0 LOAD_GLOBAL 0 (A)
3 PRINT_ITEM
4 PRINT_NEWLINE
5 LOAD_CONST 0 (None)
8 RETURN_VALUE

func_globals
func_code
....
{
'A': 'ciao',
}
antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

Slide 25

Slide 25 text

Spell #1: rebind_globals (2) magic.py def rebind_globals(func, newglobals=None): if newglobals is None: newglobals = globals() newfunc = types.FunctionType(func.func_code, newglobals, func.func_name, func.func_defaults) return newfunc antocuni (EuroPython 2012) Python White Magic July 3 2012 16 / 47

Slide 26

Slide 26 text

Problem #1 solved pdbpp.py import pdb class Pdb(pdb.Pdb): ... set_trace = rebind_globals( pdb.set_trace) main = rebind_globals(pdb.main) ... Pros code reuse cross-version compatibility Cons complexity fragility unexpected bugs (e.g. func_defaults) antocuni (EuroPython 2012) Python White Magic July 3 2012 17 / 47

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Problem #2: Open Classes Huge hierarchy of classes Many (loosely related) features per class e.g. visitor pattern two-axis split: by feature by class Split the class among more ﬁles? no way in Python in Ruby: “open classes” antocuni (EuroPython 2012) Python White Magic July 3 2012 18 / 47

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Problem #2: the PyPy AST compiler AST stmt FunctionDef Print Assign ... expr UnaryOp BinaryOp IfExpr ... ... antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

Slide 31

Slide 31 text

Problem #2: the PyPy AST compiler AST stmt FunctionDef Print Assign ... expr UnaryOp BinaryOp IfExpr ... ... as_node_list() set_context() build_container() get_generators() ... as_constant() accept_jump_if() ... antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

Slide 32

Slide 32 text

Problem #2: the PyPy AST compiler AST stmt FunctionDef Print Assign ... expr UnaryOp BinaryOp IfExpr ... ... as_node_list() set_context() build_container() get_generators() ... as_constant() accept_jump_if() ... asthelpers.py antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

Slide 33

Slide 33 text

Problem #2: the PyPy AST compiler AST stmt FunctionDef Print Assign ... expr UnaryOp BinaryOp IfExpr ... ... as_node_list() set_context() build_container() get_generators() ... as_constant() accept_jump_if() ... asthelpers.py codegen.py antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

Slide 34

Slide 34 text

Problem #2: the PyPy AST compiler AST stmt FunctionDef Print Assign ... expr UnaryOp BinaryOp IfExpr ... ... as_node_list() set_context() build_container() get_generators() ... as_constant() accept_jump_if() ... asthelpers.py codegen.py optimize.py antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

Slide 35

Slide 35 text

Spell #2: __extend__ (1) python >>> from magic import extendabletype >>> class A(object): ... __metaclass__ = extendabletype ... def foo(self): ... print ’foo’ ... >>> obj = A() >>> obj.foo() foo >>> class __extend__(A): ... def bar(self): ... print ’bar’ ... >>> obj.foo() foo >>> obj.bar() bar antocuni (EuroPython 2012) Python White Magic July 3 2012 20 / 47

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Metaclasses for dummies (1) Everything is an object Every object has a type (a class) A class is an object The class of a class: metaclass class statement --> metaclass instantiation antocuni (EuroPython 2012) Python White Magic July 3 2012 21 / 47

Slide 38

Slide 38 text

class statement class Foo(Bar): myvar = 42 def foo(self): .... antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

Slide 39

Slide 39 text

class statement class Foo(Bar): myvar = 42 def foo(self): .... { 'myvar': 42, 'foo': } antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

Slide 40

Slide 40 text

class statement class Foo(Bar): myvar = 42 def foo(self): .... { 'myvar': 42, 'foo': } Foo = metaclass('Foo', (Bar,), dic) antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

Slide 41

Slide 41 text

class statement class Foo(Bar): myvar = 42 def foo(self): .... { 'myvar': 42, 'foo': } Foo = metaclass('Foo', (Bar,), dic) antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

Slide 42

Slide 42 text

class statement class Foo(Bar): myvar = 42 def foo(self): .... Foo = metaclass('Foo', (Bar,), dic) ????? antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

Slide 43

Slide 43 text

Metaclasses for dummies (2) Find the metaclass (simpliﬁed) __metaclass__ in the class body type(bases[0]) (if any) global __metaclass__ Call it! subclass of type (usually) override __new__ tweak the parameters? antocuni (EuroPython 2012) Python White Magic July 3 2012 23 / 47

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Spell #2: __extend__ (2) example class A(object): __metaclass__ = extendabletype class __extend__(A): def bar(self): print ’bar’ magic.py class extendabletype(type): def __new__(cls, name, bases, dict): if name == ’__extend__’: for cls in bases: for key, value in dict.items(): if key == ’__module__’: continue setattr(cls, key, value) return None else: return type.__new__(cls, name, bases, dict) antocuni (EuroPython 2012) Python White Magic July 3 2012 24 / 47

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Problem #2 solved astcompiler/ # ast.py class AST(...): __metaclass__ = extendabletype ... class expr(AST): ... class stmt(AST): ... # asthelpers.py class __extend__(AST): def as_node_list(...): ... def set_context(...): ... class __extend__(expr): ... class __extend__(stmt): ... # codegen.py ... # optimize.py ... Pros better organization well-contained easy to understand Cons naming convention import-time side effects unobvious at ﬁrst antocuni (EuroPython 2012) Python White Magic July 3 2012 25 / 47

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Simple is better than complex Use as much much magic as actually needed Example: Camelot admin classes Entity subclass for the data model Foo.Admin for UI stuff mixing business and presentation logic :-( model.py class User(Entity): name = Field(Unicode) age = Field(Integer) class Admin(EntityAdmin): verbose_name = ’User’ field_list = [’name’, ’age’] antocuni (EuroPython 2012) Python White Magic July 3 2012 26 / 47

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Problem #3: wrong solution model.py class User(Entity): __metaclass__ = \ extendabletype name = Field(Unicode) age = Field(Integer) admin.py from model import User class __extend__(User): class Admin(EntityAdmin): verbose_name = ’User detail’ field_list = [’name’, ’age’] Entity has its own metaclass metaclass conﬂict there is a simpler solution anyway antocuni (EuroPython 2012) Python White Magic July 3 2012 27 / 47

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Spell #3: attach_to (not really magical) python >>> from magic import attach_to >>> >>> class Foo(object): ... pass ... >>> @attach_to(Foo) ... class Admin(object): ... verbose_name = ’My Foo’ ... >>> print Foo.Admin.verbose_name My Foo antocuni (EuroPython 2012) Python White Magic July 3 2012 28 / 47

Slide 56

Slide 56 text

Decorators Syntactic sugar class decorators only for Python >= 2.6 decor1.py @foo def myfunc(): pass @bar(42) class MyClass(object): pass decor2.py def myfunc(): pass myfunc = foo(myfunc) class MyClass(object): pass MyClass = bar(42)(MyClass) antocuni (EuroPython 2012) Python White Magic July 3 2012 29 / 47

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Spell #3: attach_to (not really magical) magic.py def attach_to(obj): def attach(cls): name = cls.__name__ setattr(obj, name, cls) return attach antocuni (EuroPython 2012) Python White Magic July 3 2012 30 / 47

Slide 59

Slide 59 text

Problem #3 solved model.py class User(Entity): name = Field(Unicode) age = Field(Integer) admin.py @attach_to(User): class Admin(EntityAdmin): verbose_name = ’User detail’ field_list = [’name’, ’age’] Pros Simple is better than complex No metaclasses Much less magic Cons None :-) antocuni (EuroPython 2012) Python White Magic July 3 2012 31 / 47

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Problem #4: quicker turnaround time (1) Developing new gdb commands gdbdemo.py import gdb class MyCommand(gdb.Command): def __init__(self): gdb.Command.__init__(self, "mycmd", gdb.COMMAND_NONE) def invoke(self, arg, from_tty): print ’Hello from Python’ MyCommand() # register the command gdb $ PYTHONPATH="" gdb GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08 ... (gdb) python import gdbdemo (gdb) mycmd Hello from Python (gdb) antocuni (EuroPython 2012) Python White Magic July 3 2012 32 / 47

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Problem #4: quicker turnaround time (2) Annoying to develop Several steps to do every time: start gdb start the program run until a certain point try your command antocuni (EuroPython 2012) Python White Magic July 3 2012 33 / 47

Slide 65

Slide 65 text

Spell #4: reload & rebind __class__ gdbdemo.py import gdb class MyCommand(gdb.Command): def __init__(self): gdb.Command.__init__(self, "mycmd", gdb.COMMAND_NONE) def invoke(self, arg, from_tty): import gdbdemo2 reload(gdbdemo2) self.__class__ = gdbdemo2.MyCommand self.do_invoke(arg, from_tty) def do_invoke(self, arg, from_tty): print ’Hello from Python’ MyCommand() # register the command antocuni (EuroPython 2012) Python White Magic July 3 2012 34 / 47

Slide 66

Slide 66 text

Method lookup x y __class__ ... __init__() invoke() do_invoke() ... MyCommand().invoke(...) antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

Slide 67

Slide 67 text

Method lookup x y __class__ ... __init__() invoke() do_invoke() ... MyCommand().invoke(...) x y __class__ ... antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

Slide 68

Slide 68 text

Method lookup x y __class__ ... __init__() invoke() do_invoke() ... MyCommand().invoke(...) antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

Slide 69

Slide 69 text

Method lookup x y __class__ ... __init__() invoke() do_invoke() ... MyCommand().invoke(...) invoke() antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

Slide 70

Slide 70 text

Method lookup x y __class__ ... __init__() invoke() do_invoke() ... MyCommand().invoke(...) reload(gdbdemo) __init__() invoke() do_invoke() ... antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

Slide 71

Slide 71 text

Method lookup x y __class__ ... __init__() invoke() do_invoke() ... MyCommand().invoke(...) reload(gdbdemo) __init__() invoke() do_invoke() ... self.__class__ = gdbdemo.MyCommand antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

Slide 72

Slide 72 text

Method lookup x y __class__ ... __init__() invoke() do_invoke() ... MyCommand().invoke(...) reload(gdbdemo) __init__() invoke() do_invoke() ... self.__class__ = gdbdemo.MyCommand self.do_invoke(...) do_invoke() antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

Slide 73

Slide 73 text

Problem #4 solved Only during development Pros Quicker turnaround Useful in various situations Cons Fragile Confusion++ (two different classes with the same name) Does not work in more complex cases antocuni (EuroPython 2012) Python White Magic July 3 2012 36 / 47

Slide 74

Slide 74 text

Bonus spell How much time is left? bonus.py import time time.go_back(minutes=10) How much time is left? antocuni (EuroPython 2012) Python White Magic July 3 2012 37 / 47

Slide 75

Slide 75 text

Bonus spell How much time is left? bonus.py import time time.go_back(minutes=10) How much time is left? antocuni (EuroPython 2012) Python White Magic July 3 2012 37 / 47

Slide 76

Slide 76 text

Bonus spell How much time is left? bonus.py import time time.go_back(minutes=10) How much time is left? antocuni (EuroPython 2012) Python White Magic July 3 2012 37 / 47

Slide 77

Slide 77 text

Problem #5: implicit global state elixir declarative layer on top of SQLAlchemy (precursor of sqlalchemy.declarative) model.py from elixir import (Entity, Field, Unicode, Integer, metadata, setup_all, create_all) metadata.bind = "sqlite:///db.sqlite" class User(Entity): name = Field(Unicode) age = Field(Integer) setup_all() create_all() # ... antocuni (EuroPython 2012) Python White Magic July 3 2012 38 / 47

Slide 78

Slide 78 text

Slide 79

Slide 79 text

Global state Global state is evil that’s it implicit is even worse Hard to test no isolation for tests persistent side effects (e.g. DB) Goal multiple, isolated, independent DBs one DB per test (or group of tests) antocuni (EuroPython 2012) Python White Magic July 3 2012 39 / 47

Slide 80

Slide 80 text

Slide 81

Slide 81 text

Slide 82

Slide 82 text

A step forward (1) model.py import sqlalchemy from elixir import (Entity, Field, Unicode, Integer, setup_entities, GlobalEntityCollection) __session__ = sqlalchemy.orm.scoped_session( sqlalchemy.orm.sessionmaker()) __metadata__ = sqlalchemy.MetaData(bind="sqlite:///db.sqlite") __collection__ = GlobalEntityCollection() class User(Entity): name = Field(Unicode) age = Field(Integer) if __name__ == ’__main__’: setup_entities(__collection__) __metadata__.create_all() antocuni (EuroPython 2012) Python White Magic July 3 2012 40 / 47

Slide 83

Slide 83 text

A step forward (2) Still global state but explicit Goal: turn global state into local N indepentent __metadata__ & co. model.User Multiple copies of the same module Modules are singleton Cannot rely on import antocuni (EuroPython 2012) Python White Magic July 3 2012 41 / 47

Slide 84

Slide 84 text

Slide 85

Slide 85 text

import logic (simpliﬁed) import foo ﬁnd foo.py this is a whole mess of its own :-) import foo mod = types.ModuleType(’foo’) # 1. create the module object mod.__file__ = ’/path/to/foo.py’ sys.modules[’foo’] = mod # 2. update sys.modules src = open(’/path/to/foo.py’).read() # 3. compile&exec the code exec src in mod.__dict__ antocuni (EuroPython 2012) Python White Magic July 3 2012 42 / 47

Slide 86

Slide 86 text

Slide 87

Slide 87 text

Spell #5: import_model (1) db.py class Database(object): def __init__(self, url, modelfile): self.url = url self.session = sqlalchemy.orm.scoped_session( sqlalchemy.orm.sessionmaker()) self.metadata = sqlalchemy.MetaData(bind=url) self.collection = elixir.GlobalEntityCollection() self.import_model(modelfile) elixir.setup_entities(self.collection) def import_model(self, filename): ... def create_all(self, *args, **kwds): self.metadata.create_all(*args, **kwds) db = Database(’sqlite:///db.sqlite’) db.create_all() myuser = db.model.User(’antocuni’, 30) antocuni (EuroPython 2012) Python White Magic July 3 2012 43 / 47

Slide 88

Slide 88 text

Spell #5: import_model (2) db.py def import_model(self, filename): # 1. create the module object self.model = types.ModuleType(’model’) self.model.__file__ = filename # 2. update sys.modules assert ’model’ not in sys.modules sys.modules[’model’] = self.model try: # inject the "local" state! self.model.__session__ = self.session self.model.__metadata__ = self.metadata self.model.__collection__ = self.collection # 3. compile&exec the code src = open(filename).read() code = compile(src, filename, ’exec’) exec src in self.model.__dict__ finally: # (Python doesn’t do it usually :-)) del sys.modules[’model’] antocuni (EuroPython 2012) Python White Magic July 3 2012 44 / 47

Slide 89

Slide 89 text

Problem #5 solved test_model.py def test_user(): url = "sqlite:///tmp/db.tmp" db = Database(url, ’/path/to/model.py’) db.create_all() myuser = db.model.User( ’antocuni’, 30) ... Pros testable multiple DB model.py kept simple magic well contained Cons complex db1.model.User != db2.model.User antocuni (EuroPython 2012) Python White Magic July 3 2012 45 / 47

Slide 90

Slide 90 text

Slide 91

Slide 91 text

Conclusion Magic might be helpful code reuse readability testability Use with care only if you know what you are doing lots of comments, please :-) Debugging is twice as hard as writing the code in the ﬁrst place. Therefore, if you write the code as cleverly as possible, you are, by deﬁnition, not smart enough to debug it. (Brian Kernighan) antocuni (EuroPython 2012) Python White Magic July 3 2012 46 / 47

Slide 92

Slide 92 text

Slide 93

Slide 93 text

Contacts, Q&A http://bitbucket.org/antocuni/whitemagic twitter: @antocuni Available for consultancy & training: http://antocuni.eu [email protected] Any question? antocuni (EuroPython 2012) Python White Magic July 3 2012 47 / 47