Python White Magic

Transcript

1. Python White Magic Antonio Cuni EuroPython 2012 July 3 2012

   antocuni (EuroPython 2012) Python White Magic July 3 2012 1 / 47

2. About me PyPy core dev PyPy py3k tech leader pdb++,

   fancycompleter, ... Consultant, trainer http://antocuni.eu antocuni (EuroPython 2012) Python White Magic July 3 2012 2 / 47

3. About this talk (1) a collection of hacks :-) HOWEVER

   real-life problems real-life solutions they improve the rest of the code (you might not like them) antocuni (EuroPython 2012) Python White Magic July 3 2012 3 / 47

4. About this talk (1) a collection of hacks :-) HOWEVER

   real-life problems real-life solutions they improve the rest of the code (you might not like them) antocuni (EuroPython 2012) Python White Magic July 3 2012 3 / 47

5. About this talk (2) Learn about internals and/or advanced Python

   function and code objects bytecode metaclasses decorators import logic antocuni (EuroPython 2012) Python White Magic July 3 2012 4 / 47

6. Python Powerful language Surface of simplicity “just works” (TM) as

   expected simplicity is an emergent property lots of internal rules and layers (ab)use them “black magic” antocuni (EuroPython 2012) Python White Magic July 3 2012 5 / 47

7. Python Powerful language Surface of simplicity “just works” (TM) as

   expected simplicity is an emergent property lots of internal rules and layers (ab)use them “black magic” antocuni (EuroPython 2012) Python White Magic July 3 2012 5 / 47

8. “Black” magic harder to read harder to maintain might break

   unexpectedly might rely on implementation details ... antocuni (EuroPython 2012) Python White Magic July 3 2012 6 / 47

9. What is magic? Any sufficiently advanced technology is indistinguishable from

   magic (Arthur C. Clarke) antocuni (EuroPython 2012) Python White Magic July 3 2012 7 / 47

10. What is magic? Any sufficiently advanced technology is indistinguishable from

    magic (Arthur C. Clarke) antocuni (EuroPython 2012) Python White Magic July 3 2012 8 / 47

11. What is magic? Any sufficiently advanced technology is indistinguishable from

    magic (Arthur C. Clarke) antocuni (EuroPython 2012) Python White Magic July 3 2012 9 / 47

12. White magic Still magic can help to have better code

    more readable more maintainable however: lots of cons some pros solid understanding of concepts is needed use with care antocuni (EuroPython 2012) Python White Magic July 3 2012 10 / 47

13. Problem #1: extending pdb.py (1) pdb++: drop-in replacement for pdb.py

    keep the same API how extend a module? subclassing is not enouogh module-level functions? antocuni (EuroPython 2012) Python White Magic July 3 2012 11 / 47

14. Problem #1: extending pdb.py (2) pdb.py ... class Pdb(bdb.Bdb, cmd.Cmd):

    ... def set_trace(): Pdb().set_trace(...) def main(): ... pdb = Pdb() pdb._runscript(...) ... pdbpp.py import pdb class Pdb(pdb.Pdb): ... def set_trace(): pdb.set_trace() # ??? def main(): pdb.main() # ??? ... antocuni (EuroPython 2012) Python White Magic July 3 2012 12 / 47

15. Problem #1: extending pdb.py (2) pdb.py ... class Pdb(bdb.Bdb, cmd.Cmd):

    ... def set_trace(): Pdb().set_trace(...) def main(): ... pdb = Pdb() pdb._runscript(...) ... pdbpp.py import pdb class Pdb(pdb.Pdb): ... def set_trace(): pdb.set_trace() # ??? def main(): pdb.main() # ??? ... antocuni (EuroPython 2012) Python White Magic July 3 2012 12 / 47

16. Problem #1: extending pdb.py (3) Logic inside set_trace and main

    (and others) Pdb() refers to pdb.Pdb() our new class is never instantiated copy&paste is not a solution :-) antocuni (EuroPython 2012) Python White Magic July 3 2012 13 / 47

17. Spell #1: rebind_globals (1) python >>> A = ’hello’ >>>

    def foo(): ... print A ... >>> foo() hello >>> from magic import rebind_globals >>> foo2 = rebind_globals(foo, {’A’: ’ciao’}) >>> A ’hello’ >>> foo() hello >>> foo2() ciao antocuni (EuroPython 2012) Python White Magic July 3 2012 14 / 47

18. Spell #1: rebind_globals (1) python >>> A = ’hello’ >>>

    def foo(): ... print A ... >>> foo() hello >>> from magic import rebind_globals >>> foo2 = rebind_globals(foo, {’A’: ’ciao’}) >>> A ’hello’ >>> foo() hello >>> foo2() ciao antocuni (EuroPython 2012) Python White Magic July 3 2012 14 / 47

19. LOAD_GLOBAL explained <function 'foo'> func_globals func_code .... antocuni (EuroPython 2012)

    Python White Magic July 3 2012 15 / 47

20. LOAD_GLOBAL explained <function 'foo'> func_globals func_code .... { 'foo': <function'foo'>,

    'A': 'hello', } antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

21. LOAD_GLOBAL explained <function 'foo'> func_globals func_code .... { 'foo': <function'foo'>,

    'A': 'hello', } <code object 'foo'> co_code .... antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

22. LOAD_GLOBAL explained <function 'foo'> func_globals func_code .... { 'foo': <function'foo'>,

    'A': 'hello', } <code object 'foo'> co_code .... 0 LOAD_GLOBAL 0 (A) 3 PRINT_ITEM 4 PRINT_NEWLINE 5 LOAD_CONST 0 (None) 8 RETURN_VALUE antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

23. LOAD_GLOBAL explained <function 'foo'> func_globals func_code .... { 'foo': <function'foo'>,

    'A': 'hello', } <code object 'foo'> co_code .... 0 LOAD_GLOBAL 0 (A) 3 PRINT_ITEM 4 PRINT_NEWLINE 5 LOAD_CONST 0 (None) 8 RETURN_VALUE antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

24. LOAD_GLOBAL explained <function 'foo'> func_globals func_code .... { 'foo': <function'foo'>,

    'A': 'hello', } <code object 'foo'> co_code .... 0 LOAD_GLOBAL 0 (A) 3 PRINT_ITEM 4 PRINT_NEWLINE 5 LOAD_CONST 0 (None) 8 RETURN_VALUE <function 'foo2'> func_globals func_code .... { 'A': 'ciao', } antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

25. Spell #1: rebind_globals (2) magic.py def rebind_globals(func, newglobals=None): if newglobals

    is None: newglobals = globals() newfunc = types.FunctionType(func.func_code, newglobals, func.func_name, func.func_defaults) return newfunc antocuni (EuroPython 2012) Python White Magic July 3 2012 16 / 47

26. Problem #1 solved pdbpp.py import pdb class Pdb(pdb.Pdb): ... set_trace

    = rebind_globals( pdb.set_trace) main = rebind_globals(pdb.main) ... Pros code reuse cross-version compatibility Cons complexity fragility unexpected bugs (e.g. func_defaults) antocuni (EuroPython 2012) Python White Magic July 3 2012 17 / 47

27. Problem #1 solved pdbpp.py import pdb class Pdb(pdb.Pdb): ... set_trace

    = rebind_globals( pdb.set_trace) main = rebind_globals(pdb.main) ... Pros code reuse cross-version compatibility Cons complexity fragility unexpected bugs (e.g. func_defaults) antocuni (EuroPython 2012) Python White Magic July 3 2012 17 / 47

28. Problem #2: Open Classes Huge hierarchy of classes Many (loosely

    related) features per class e.g. visitor pattern two-axis split: by feature by class Split the class among more files? no way in Python in Ruby: “open classes” antocuni (EuroPython 2012) Python White Magic July 3 2012 18 / 47

29. Problem #2: Open Classes Huge hierarchy of classes Many (loosely

    related) features per class e.g. visitor pattern two-axis split: by feature by class Split the class among more files? no way in Python in Ruby: “open classes” antocuni (EuroPython 2012) Python White Magic July 3 2012 18 / 47

30. Problem #2: the PyPy AST compiler AST stmt FunctionDef Print

    Assign ... expr UnaryOp BinaryOp IfExpr ... ... antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

31. Problem #2: the PyPy AST compiler AST stmt FunctionDef Print

    Assign ... expr UnaryOp BinaryOp IfExpr ... ... as_node_list() set_context() build_container() get_generators() ... as_constant() accept_jump_if() ... antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

32. Problem #2: the PyPy AST compiler AST stmt FunctionDef Print

    Assign ... expr UnaryOp BinaryOp IfExpr ... ... as_node_list() set_context() build_container() get_generators() ... as_constant() accept_jump_if() ... asthelpers.py antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

33. Problem #2: the PyPy AST compiler AST stmt FunctionDef Print

    Assign ... expr UnaryOp BinaryOp IfExpr ... ... as_node_list() set_context() build_container() get_generators() ... as_constant() accept_jump_if() ... asthelpers.py codegen.py antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

34. Problem #2: the PyPy AST compiler AST stmt FunctionDef Print

    Assign ... expr UnaryOp BinaryOp IfExpr ... ... as_node_list() set_context() build_container() get_generators() ... as_constant() accept_jump_if() ... asthelpers.py codegen.py optimize.py antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

35. Spell #2: __extend__ (1) python >>> from magic import extendabletype

    >>> class A(object): ... __metaclass__ = extendabletype ... def foo(self): ... print ’foo’ ... >>> obj = A() >>> obj.foo() foo >>> class __extend__(A): ... def bar(self): ... print ’bar’ ... >>> obj.foo() foo >>> obj.bar() bar antocuni (EuroPython 2012) Python White Magic July 3 2012 20 / 47

36. Spell #2: __extend__ (1) python >>> from magic import extendabletype

    >>> class A(object): ... __metaclass__ = extendabletype ... def foo(self): ... print ’foo’ ... >>> obj = A() >>> obj.foo() foo >>> class __extend__(A): ... def bar(self): ... print ’bar’ ... >>> obj.foo() foo >>> obj.bar() bar antocuni (EuroPython 2012) Python White Magic July 3 2012 20 / 47

37. Metaclasses for dummies (1) Everything is an object Every object

    has a type (a class) A class is an object The class of a class: metaclass class statement --> metaclass instantiation antocuni (EuroPython 2012) Python White Magic July 3 2012 21 / 47

38. class statement class Foo(Bar): myvar = 42 def foo(self): ....

    antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

39. class statement class Foo(Bar): myvar = 42 def foo(self): ....

    { 'myvar': 42, 'foo': <function 'foo'> } antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

40. class statement class Foo(Bar): myvar = 42 def foo(self): ....

    { 'myvar': 42, 'foo': <function 'foo'> } Foo = metaclass('Foo', (Bar,), dic) antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

41. class statement class Foo(Bar): myvar = 42 def foo(self): ....

    { 'myvar': 42, 'foo': <function 'foo'> } Foo = metaclass('Foo', (Bar,), dic) antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

42. class statement class Foo(Bar): myvar = 42 def foo(self): ....

    Foo = metaclass('Foo', (Bar,), dic) ????? antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

43. Metaclasses for dummies (2) Find the metaclass (simplified) __metaclass__ in

    the class body type(bases[0]) (if any) global __metaclass__ Call it! subclass of type (usually) override __new__ tweak the parameters? antocuni (EuroPython 2012) Python White Magic July 3 2012 23 / 47

44. Metaclasses for dummies (2) Find the metaclass (simplified) __metaclass__ in

    the class body type(bases[0]) (if any) global __metaclass__ Call it! subclass of type (usually) override __new__ tweak the parameters? antocuni (EuroPython 2012) Python White Magic July 3 2012 23 / 47

45. Spell #2: __extend__ (2) example class A(object): __metaclass__ = extendabletype

    class __extend__(A): def bar(self): print ’bar’ magic.py class extendabletype(type): def __new__(cls, name, bases, dict): if name == ’__extend__’: for cls in bases: for key, value in dict.items(): if key == ’__module__’: continue setattr(cls, key, value) return None else: return type.__new__(cls, name, bases, dict) antocuni (EuroPython 2012) Python White Magic July 3 2012 24 / 47

46. Spell #2: __extend__ (2) example class A(object): __metaclass__ = extendabletype

    class __extend__(A): def bar(self): print ’bar’ magic.py class extendabletype(type): def __new__(cls, name, bases, dict): if name == ’__extend__’: for cls in bases: for key, value in dict.items(): if key == ’__module__’: continue setattr(cls, key, value) return None else: return type.__new__(cls, name, bases, dict) antocuni (EuroPython 2012) Python White Magic July 3 2012 24 / 47

47. Problem #2 solved astcompiler/ # ast.py class AST(...): __metaclass__ =

    extendabletype ... class expr(AST): ... class stmt(AST): ... # asthelpers.py class __extend__(AST): def as_node_list(...): ... def set_context(...): ... class __extend__(expr): ... class __extend__(stmt): ... # codegen.py ... # optimize.py ... Pros better organization well-contained easy to understand Cons naming convention import-time side effects unobvious at first antocuni (EuroPython 2012) Python White Magic July 3 2012 25 / 47

48. Problem #2 solved astcompiler/ # ast.py class AST(...): __metaclass__ =

    extendabletype ... class expr(AST): ... class stmt(AST): ... # asthelpers.py class __extend__(AST): def as_node_list(...): ... def set_context(...): ... class __extend__(expr): ... class __extend__(stmt): ... # codegen.py ... # optimize.py ... Pros better organization well-contained easy to understand Cons naming convention import-time side effects unobvious at first antocuni (EuroPython 2012) Python White Magic July 3 2012 25 / 47

49. Problem #2 solved astcompiler/ # ast.py class AST(...): __metaclass__ =

    extendabletype ... class expr(AST): ... class stmt(AST): ... # asthelpers.py class __extend__(AST): def as_node_list(...): ... def set_context(...): ... class __extend__(expr): ... class __extend__(stmt): ... # codegen.py ... # optimize.py ... Pros better organization well-contained easy to understand Cons naming convention import-time side effects unobvious at first antocuni (EuroPython 2012) Python White Magic July 3 2012 25 / 47

50. Simple is better than complex Use as much much magic

    as actually needed Example: Camelot admin classes Entity subclass for the data model Foo.Admin for UI stuff mixing business and presentation logic :-( model.py class User(Entity): name = Field(Unicode) age = Field(Integer) class Admin(EntityAdmin): verbose_name = ’User’ field_list = [’name’, ’age’] antocuni (EuroPython 2012) Python White Magic July 3 2012 26 / 47

51. Simple is better than complex Use as much much magic

    as actually needed Example: Camelot admin classes Entity subclass for the data model Foo.Admin for UI stuff mixing business and presentation logic :-( model.py class User(Entity): name = Field(Unicode) age = Field(Integer) class Admin(EntityAdmin): verbose_name = ’User’ field_list = [’name’, ’age’] antocuni (EuroPython 2012) Python White Magic July 3 2012 26 / 47

52. Problem #3: wrong solution model.py class User(Entity): __metaclass__ = \

    extendabletype name = Field(Unicode) age = Field(Integer) admin.py from model import User class __extend__(User): class Admin(EntityAdmin): verbose_name = ’User detail’ field_list = [’name’, ’age’] Entity has its own metaclass metaclass conflict there is a simpler solution anyway antocuni (EuroPython 2012) Python White Magic July 3 2012 27 / 47

53. Problem #3: wrong solution model.py class User(Entity): __metaclass__ = \

    extendabletype name = Field(Unicode) age = Field(Integer) admin.py from model import User class __extend__(User): class Admin(EntityAdmin): verbose_name = ’User detail’ field_list = [’name’, ’age’] Entity has its own metaclass metaclass conflict there is a simpler solution anyway antocuni (EuroPython 2012) Python White Magic July 3 2012 27 / 47

54. Problem #3: wrong solution model.py class User(Entity): __metaclass__ = \

    extendabletype name = Field(Unicode) age = Field(Integer) admin.py from model import User class __extend__(User): class Admin(EntityAdmin): verbose_name = ’User detail’ field_list = [’name’, ’age’] Entity has its own metaclass metaclass conflict there is a simpler solution anyway antocuni (EuroPython 2012) Python White Magic July 3 2012 27 / 47

55. Spell #3: attach_to (not really magical) python >>> from magic

    import attach_to >>> >>> class Foo(object): ... pass ... >>> @attach_to(Foo) ... class Admin(object): ... verbose_name = ’My Foo’ ... >>> print Foo.Admin.verbose_name My Foo antocuni (EuroPython 2012) Python White Magic July 3 2012 28 / 47

56. Decorators Syntactic sugar class decorators only for Python >= 2.6

    decor1.py @foo def myfunc(): pass @bar(42) class MyClass(object): pass decor2.py def myfunc(): pass myfunc = foo(myfunc) class MyClass(object): pass MyClass = bar(42)(MyClass) antocuni (EuroPython 2012) Python White Magic July 3 2012 29 / 47

57. Decorators Syntactic sugar class decorators only for Python >= 2.6

    decor1.py @foo def myfunc(): pass @bar(42) class MyClass(object): pass decor2.py def myfunc(): pass myfunc = foo(myfunc) class MyClass(object): pass MyClass = bar(42)(MyClass) antocuni (EuroPython 2012) Python White Magic July 3 2012 29 / 47

58. Spell #3: attach_to (not really magical) magic.py def attach_to(obj): def

    attach(cls): name = cls.__name__ setattr(obj, name, cls) return attach antocuni (EuroPython 2012) Python White Magic July 3 2012 30 / 47

59. Problem #3 solved model.py class User(Entity): name = Field(Unicode) age

    = Field(Integer) admin.py @attach_to(User): class Admin(EntityAdmin): verbose_name = ’User detail’ field_list = [’name’, ’age’] Pros Simple is better than complex No metaclasses Much less magic Cons None :-) antocuni (EuroPython 2012) Python White Magic July 3 2012 31 / 47

60. Problem #3 solved model.py class User(Entity): name = Field(Unicode) age

    = Field(Integer) admin.py @attach_to(User): class Admin(EntityAdmin): verbose_name = ’User detail’ field_list = [’name’, ’age’] Pros Simple is better than complex No metaclasses Much less magic Cons None :-) antocuni (EuroPython 2012) Python White Magic July 3 2012 31 / 47

61. Problem #3 solved model.py class User(Entity): name = Field(Unicode) age

    = Field(Integer) admin.py @attach_to(User): class Admin(EntityAdmin): verbose_name = ’User detail’ field_list = [’name’, ’age’] Pros Simple is better than complex No metaclasses Much less magic Cons None :-) antocuni (EuroPython 2012) Python White Magic July 3 2012 31 / 47

62. Problem #4: quicker turnaround time (1) Developing new gdb commands

    gdbdemo.py import gdb class MyCommand(gdb.Command): def __init__(self): gdb.Command.__init__(self, "mycmd", gdb.COMMAND_NONE) def invoke(self, arg, from_tty): print ’Hello from Python’ MyCommand() # register the command gdb $ PYTHONPATH="" gdb GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08 ... (gdb) python import gdbdemo (gdb) mycmd Hello from Python (gdb) antocuni (EuroPython 2012) Python White Magic July 3 2012 32 / 47

63. Problem #4: quicker turnaround time (1) Developing new gdb commands

    gdbdemo.py import gdb class MyCommand(gdb.Command): def __init__(self): gdb.Command.__init__(self, "mycmd", gdb.COMMAND_NONE) def invoke(self, arg, from_tty): print ’Hello from Python’ MyCommand() # register the command gdb $ PYTHONPATH="" gdb GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08 ... (gdb) python import gdbdemo (gdb) mycmd Hello from Python (gdb) antocuni (EuroPython 2012) Python White Magic July 3 2012 32 / 47

64. Problem #4: quicker turnaround time (2) Annoying to develop Several

    steps to do every time: start gdb start the program run until a certain point try your command antocuni (EuroPython 2012) Python White Magic July 3 2012 33 / 47

65. Spell #4: reload & rebind __class__ gdbdemo.py import gdb class

    MyCommand(gdb.Command): def __init__(self): gdb.Command.__init__(self, "mycmd", gdb.COMMAND_NONE) def invoke(self, arg, from_tty): import gdbdemo2 reload(gdbdemo2) self.__class__ = gdbdemo2.MyCommand self.do_invoke(arg, from_tty) def do_invoke(self, arg, from_tty): print ’Hello from Python’ MyCommand() # register the command antocuni (EuroPython 2012) Python White Magic July 3 2012 34 / 47

66. Method lookup <MyCommand object> x y __class__ ... <class 'MyCommand'>

    __init__() invoke() do_invoke() ... MyCommand().invoke(...) antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

67. Method lookup <MyCommand object> x y __class__ ... <class 'MyCommand'>

    __init__() invoke() do_invoke() ... MyCommand().invoke(...) <MyCommand object> x y __class__ ... antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

68. Method lookup <MyCommand object> x y __class__ ... <class 'MyCommand'>

    __init__() invoke() do_invoke() ... MyCommand().invoke(...) antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

69. Method lookup <MyCommand object> x y __class__ ... <class 'MyCommand'>

    __init__() invoke() do_invoke() ... MyCommand().invoke(...) invoke() antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

70. Method lookup <MyCommand object> x y __class__ ... <class 'MyCommand'>

    __init__() invoke() do_invoke() ... MyCommand().invoke(...) reload(gdbdemo) <class 'MyCommand'> __init__() invoke() do_invoke() ... antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

71. Method lookup <MyCommand object> x y __class__ ... <class 'MyCommand'>

    __init__() invoke() do_invoke() ... MyCommand().invoke(...) reload(gdbdemo) <class 'MyCommand'> __init__() invoke() do_invoke() ... self.__class__ = gdbdemo.MyCommand antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

72. Method lookup <MyCommand object> x y __class__ ... <class 'MyCommand'>

    __init__() invoke() do_invoke() ... MyCommand().invoke(...) reload(gdbdemo) <class 'MyCommand'> __init__() invoke() do_invoke() ... self.__class__ = gdbdemo.MyCommand self.do_invoke(...) do_invoke() antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

73. Problem #4 solved Only during development Pros Quicker turnaround Useful

    in various situations Cons Fragile Confusion++ (two different classes with the same name) Does not work in more complex cases antocuni (EuroPython 2012) Python White Magic July 3 2012 36 / 47

74. Bonus spell How much time is left? bonus.py import time

    time.go_back(minutes=10) How much time is left? antocuni (EuroPython 2012) Python White Magic July 3 2012 37 / 47

75. Bonus spell How much time is left? bonus.py import time

    time.go_back(minutes=10) How much time is left? antocuni (EuroPython 2012) Python White Magic July 3 2012 37 / 47

76. Bonus spell How much time is left? bonus.py import time

    time.go_back(minutes=10) How much time is left? antocuni (EuroPython 2012) Python White Magic July 3 2012 37 / 47

77. Problem #5: implicit global state elixir declarative layer on top

    of SQLAlchemy (precursor of sqlalchemy.declarative) model.py from elixir import (Entity, Field, Unicode, Integer, metadata, setup_all, create_all) metadata.bind = "sqlite:///db.sqlite" class User(Entity): name = Field(Unicode) age = Field(Integer) setup_all() create_all() # ... antocuni (EuroPython 2012) Python White Magic July 3 2012 38 / 47

78. Problem #5: implicit global state elixir declarative layer on top

    of SQLAlchemy (precursor of sqlalchemy.declarative) model.py from elixir import (Entity, Field, Unicode, Integer, metadata, setup_all, create_all) metadata.bind = "sqlite:///db.sqlite" class User(Entity): name = Field(Unicode) age = Field(Integer) setup_all() create_all() # ... antocuni (EuroPython 2012) Python White Magic July 3 2012 38 / 47

79. Global state Global state is evil that’s it implicit is

    even worse Hard to test no isolation for tests persistent side effects (e.g. DB) Goal multiple, isolated, independent DBs one DB per test (or group of tests) antocuni (EuroPython 2012) Python White Magic July 3 2012 39 / 47

80. Global state Global state is evil that’s it implicit is

    even worse Hard to test no isolation for tests persistent side effects (e.g. DB) Goal multiple, isolated, independent DBs one DB per test (or group of tests) antocuni (EuroPython 2012) Python White Magic July 3 2012 39 / 47

81. Global state Global state is evil that’s it implicit is

    even worse Hard to test no isolation for tests persistent side effects (e.g. DB) Goal multiple, isolated, independent DBs one DB per test (or group of tests) antocuni (EuroPython 2012) Python White Magic July 3 2012 39 / 47

82. A step forward (1) model.py import sqlalchemy from elixir import

    (Entity, Field, Unicode, Integer, setup_entities, GlobalEntityCollection) __session__ = sqlalchemy.orm.scoped_session( sqlalchemy.orm.sessionmaker()) __metadata__ = sqlalchemy.MetaData(bind="sqlite:///db.sqlite") __collection__ = GlobalEntityCollection() class User(Entity): name = Field(Unicode) age = Field(Integer) if __name__ == ’__main__’: setup_entities(__collection__) __metadata__.create_all() antocuni (EuroPython 2012) Python White Magic July 3 2012 40 / 47

83. A step forward (2) Still global state but explicit Goal:

    turn global state into local N indepentent __metadata__ & co. model.User Multiple copies of the same module Modules are singleton Cannot rely on import antocuni (EuroPython 2012) Python White Magic July 3 2012 41 / 47

84. A step forward (2) Still global state but explicit Goal:

    turn global state into local N indepentent __metadata__ & co. model.User Multiple copies of the same module Modules are singleton Cannot rely on import antocuni (EuroPython 2012) Python White Magic July 3 2012 41 / 47

85. import logic (simplified) import foo find foo.py this is a

    whole mess of its own :-) import foo mod = types.ModuleType(’foo’) # 1. create the module object mod.__file__ = ’/path/to/foo.py’ sys.modules[’foo’] = mod # 2. update sys.modules src = open(’/path/to/foo.py’).read() # 3. compile&exec the code exec src in mod.__dict__ antocuni (EuroPython 2012) Python White Magic July 3 2012 42 / 47

86. import logic (simplified) import foo find foo.py this is a

    whole mess of its own :-) import foo mod = types.ModuleType(’foo’) # 1. create the module object mod.__file__ = ’/path/to/foo.py’ sys.modules[’foo’] = mod # 2. update sys.modules src = open(’/path/to/foo.py’).read() # 3. compile&exec the code exec src in mod.__dict__ antocuni (EuroPython 2012) Python White Magic July 3 2012 42 / 47

87. Spell #5: import_model (1) db.py class Database(object): def __init__(self, url,

    modelfile): self.url = url self.session = sqlalchemy.orm.scoped_session( sqlalchemy.orm.sessionmaker()) self.metadata = sqlalchemy.MetaData(bind=url) self.collection = elixir.GlobalEntityCollection() self.import_model(modelfile) elixir.setup_entities(self.collection) def import_model(self, filename): ... def create_all(self, *args, **kwds): self.metadata.create_all(*args, **kwds) db = Database(’sqlite:///db.sqlite’) db.create_all() myuser = db.model.User(’antocuni’, 30) antocuni (EuroPython 2012) Python White Magic July 3 2012 43 / 47

88. Spell #5: import_model (2) db.py def import_model(self, filename): # 1.

    create the module object self.model = types.ModuleType(’model’) self.model.__file__ = filename # 2. update sys.modules assert ’model’ not in sys.modules sys.modules[’model’] = self.model try: # inject the "local" state! self.model.__session__ = self.session self.model.__metadata__ = self.metadata self.model.__collection__ = self.collection # 3. compile&exec the code src = open(filename).read() code = compile(src, filename, ’exec’) exec src in self.model.__dict__ finally: # (Python doesn’t do it usually :-)) del sys.modules[’model’] antocuni (EuroPython 2012) Python White Magic July 3 2012 44 / 47

89. Problem #5 solved test_model.py def test_user(): url = "sqlite:///tmp/db.tmp" db

    = Database(url, ’/path/to/model.py’) db.create_all() myuser = db.model.User( ’antocuni’, 30) ... Pros testable multiple DB model.py kept simple magic well contained Cons complex db1.model.User != db2.model.User antocuni (EuroPython 2012) Python White Magic July 3 2012 45 / 47

90. Problem #5 solved test_model.py def test_user(): url = "sqlite:///tmp/db.tmp" db

    = Database(url, ’/path/to/model.py’) db.create_all() myuser = db.model.User( ’antocuni’, 30) ... Pros testable multiple DB model.py kept simple magic well contained Cons complex db1.model.User != db2.model.User antocuni (EuroPython 2012) Python White Magic July 3 2012 45 / 47

91. Conclusion Magic might be helpful code reuse readability testability Use

    with care only if you know what you are doing lots of comments, please :-) Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. (Brian Kernighan) antocuni (EuroPython 2012) Python White Magic July 3 2012 46 / 47

92. Conclusion Magic might be helpful code reuse readability testability Use

    with care only if you know what you are doing lots of comments, please :-) Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. (Brian Kernighan) antocuni (EuroPython 2012) Python White Magic July 3 2012 46 / 47

93. Contacts, Q&A http://bitbucket.org/antocuni/whitemagic twitter: @antocuni Available for consultancy & training:

    http://antocuni.eu [email protected] Any question? antocuni (EuroPython 2012) Python White Magic July 3 2012 47 / 47