Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python White Magic

Python White Magic

EuroPython 2012, Firenze, Italy

Antonio Cuni

July 03, 2012
Tweet

More Decks by Antonio Cuni

Other Decks in Programming

Transcript

  1. Python White Magic
    Antonio Cuni
    EuroPython 2012
    July 3 2012
    antocuni (EuroPython 2012) Python White Magic July 3 2012 1 / 47

    View full-size slide

  2. About me
    PyPy core dev
    PyPy py3k tech leader
    pdb++, fancycompleter, ...
    Consultant, trainer
    http://antocuni.eu
    antocuni (EuroPython 2012) Python White Magic July 3 2012 2 / 47

    View full-size slide

  3. About this talk (1)
    a collection of hacks :-)
    HOWEVER
    real-life problems
    real-life solutions
    they improve the rest of the code
    (you might not like them)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 3 / 47

    View full-size slide

  4. About this talk (1)
    a collection of hacks :-)
    HOWEVER
    real-life problems
    real-life solutions
    they improve the rest of the code
    (you might not like them)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 3 / 47

    View full-size slide

  5. About this talk (2)
    Learn about internals and/or advanced Python
    function and code objects
    bytecode
    metaclasses
    decorators
    import logic
    antocuni (EuroPython 2012) Python White Magic July 3 2012 4 / 47

    View full-size slide

  6. Python
    Powerful language
    Surface of simplicity
    “just works” (TM) as expected
    simplicity is an emergent property
    lots of internal rules and layers
    (ab)use them
    “black magic”
    antocuni (EuroPython 2012) Python White Magic July 3 2012 5 / 47

    View full-size slide

  7. Python
    Powerful language
    Surface of simplicity
    “just works” (TM) as expected
    simplicity is an emergent property
    lots of internal rules and layers
    (ab)use them
    “black magic”
    antocuni (EuroPython 2012) Python White Magic July 3 2012 5 / 47

    View full-size slide

  8. “Black” magic
    harder to read
    harder to maintain
    might break unexpectedly
    might rely on implementation details
    ...
    antocuni (EuroPython 2012) Python White Magic July 3 2012 6 / 47

    View full-size slide

  9. What is magic?
    Any sufficiently advanced technology is
    indistinguishable from magic
    (Arthur C. Clarke)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 7 / 47

    View full-size slide

  10. What is magic?
    Any sufficiently advanced technology is
    indistinguishable from magic
    (Arthur C. Clarke)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 8 / 47

    View full-size slide

  11. What is magic?
    Any sufficiently advanced technology is
    indistinguishable from magic
    (Arthur C. Clarke)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 9 / 47

    View full-size slide

  12. White magic
    Still magic
    can help to have better code
    more readable
    more maintainable
    however:
    lots of cons
    some pros
    solid understanding of concepts is needed
    use with care
    antocuni (EuroPython 2012) Python White Magic July 3 2012 10 / 47

    View full-size slide

  13. Problem #1: extending pdb.py (1)
    pdb++: drop-in replacement for pdb.py
    keep the same API
    how extend a module?
    subclassing is not enouogh
    module-level functions?
    antocuni (EuroPython 2012) Python White Magic July 3 2012 11 / 47

    View full-size slide

  14. Problem #1: extending pdb.py (2)
    pdb.py
    ...
    class Pdb(bdb.Bdb, cmd.Cmd):
    ...
    def set_trace():
    Pdb().set_trace(...)
    def main():
    ...
    pdb = Pdb()
    pdb._runscript(...)
    ...
    pdbpp.py
    import pdb
    class Pdb(pdb.Pdb):
    ...
    def set_trace():
    pdb.set_trace() # ???
    def main():
    pdb.main() # ???
    ...
    antocuni (EuroPython 2012) Python White Magic July 3 2012 12 / 47

    View full-size slide

  15. Problem #1: extending pdb.py (2)
    pdb.py
    ...
    class Pdb(bdb.Bdb, cmd.Cmd):
    ...
    def set_trace():
    Pdb().set_trace(...)
    def main():
    ...
    pdb = Pdb()
    pdb._runscript(...)
    ...
    pdbpp.py
    import pdb
    class Pdb(pdb.Pdb):
    ...
    def set_trace():
    pdb.set_trace() # ???
    def main():
    pdb.main() # ???
    ...
    antocuni (EuroPython 2012) Python White Magic July 3 2012 12 / 47

    View full-size slide

  16. Problem #1: extending pdb.py (3)
    Logic inside set_trace and main (and others)
    Pdb() refers to pdb.Pdb()
    our new class is never instantiated
    copy&paste is not a solution :-)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 13 / 47

    View full-size slide

  17. Spell #1: rebind_globals (1)
    python
    >>> A = ’hello’
    >>> def foo():
    ... print A
    ...
    >>> foo()
    hello
    >>> from magic import rebind_globals
    >>> foo2 = rebind_globals(foo, {’A’: ’ciao’})
    >>> A
    ’hello’
    >>> foo()
    hello
    >>> foo2()
    ciao
    antocuni (EuroPython 2012) Python White Magic July 3 2012 14 / 47

    View full-size slide

  18. Spell #1: rebind_globals (1)
    python
    >>> A = ’hello’
    >>> def foo():
    ... print A
    ...
    >>> foo()
    hello
    >>> from magic import rebind_globals
    >>> foo2 = rebind_globals(foo, {’A’: ’ciao’})
    >>> A
    ’hello’
    >>> foo()
    hello
    >>> foo2()
    ciao
    antocuni (EuroPython 2012) Python White Magic July 3 2012 14 / 47

    View full-size slide

  19. LOAD_GLOBAL explained

    func_globals
    func_code
    ....
    antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

    View full-size slide

  20. LOAD_GLOBAL explained

    func_globals
    func_code
    ....
    {
    'foo': ,
    'A': 'hello',
    }
    antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

    View full-size slide

  21. LOAD_GLOBAL explained

    func_globals
    func_code
    ....
    {
    'foo': ,
    'A': 'hello',
    }

    co_code
    ....
    antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

    View full-size slide

  22. LOAD_GLOBAL explained

    func_globals
    func_code
    ....
    {
    'foo': ,
    'A': 'hello',
    }

    co_code
    ....
    0 LOAD_GLOBAL 0 (A)
    3 PRINT_ITEM
    4 PRINT_NEWLINE
    5 LOAD_CONST 0 (None)
    8 RETURN_VALUE
    antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

    View full-size slide

  23. LOAD_GLOBAL explained

    func_globals
    func_code
    ....
    {
    'foo': ,
    'A': 'hello',
    }

    co_code
    ....
    0 LOAD_GLOBAL 0 (A)
    3 PRINT_ITEM
    4 PRINT_NEWLINE
    5 LOAD_CONST 0 (None)
    8 RETURN_VALUE
    antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

    View full-size slide

  24. LOAD_GLOBAL explained

    func_globals
    func_code
    ....
    {
    'foo': ,
    'A': 'hello',
    }

    co_code
    ....
    0 LOAD_GLOBAL 0 (A)
    3 PRINT_ITEM
    4 PRINT_NEWLINE
    5 LOAD_CONST 0 (None)
    8 RETURN_VALUE

    func_globals
    func_code
    ....
    {
    'A': 'ciao',
    }
    antocuni (EuroPython 2012) Python White Magic July 3 2012 15 / 47

    View full-size slide

  25. Spell #1: rebind_globals (2)
    magic.py
    def rebind_globals(func, newglobals=None):
    if newglobals is None:
    newglobals = globals()
    newfunc = types.FunctionType(func.func_code,
    newglobals,
    func.func_name,
    func.func_defaults)
    return newfunc
    antocuni (EuroPython 2012) Python White Magic July 3 2012 16 / 47

    View full-size slide

  26. Problem #1 solved
    pdbpp.py
    import pdb
    class Pdb(pdb.Pdb):
    ...
    set_trace = rebind_globals(
    pdb.set_trace)
    main = rebind_globals(pdb.main)
    ...
    Pros
    code reuse
    cross-version
    compatibility
    Cons
    complexity
    fragility
    unexpected bugs
    (e.g.
    func_defaults)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 17 / 47

    View full-size slide

  27. Problem #1 solved
    pdbpp.py
    import pdb
    class Pdb(pdb.Pdb):
    ...
    set_trace = rebind_globals(
    pdb.set_trace)
    main = rebind_globals(pdb.main)
    ...
    Pros
    code reuse
    cross-version
    compatibility
    Cons
    complexity
    fragility
    unexpected bugs
    (e.g.
    func_defaults)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 17 / 47

    View full-size slide

  28. Problem #2: Open Classes
    Huge hierarchy of classes
    Many (loosely related) features per class
    e.g. visitor pattern
    two-axis split:
    by feature
    by class
    Split the class among more files?
    no way in Python
    in Ruby: “open classes”
    antocuni (EuroPython 2012) Python White Magic July 3 2012 18 / 47

    View full-size slide

  29. Problem #2: Open Classes
    Huge hierarchy of classes
    Many (loosely related) features per class
    e.g. visitor pattern
    two-axis split:
    by feature
    by class
    Split the class among more files?
    no way in Python
    in Ruby: “open classes”
    antocuni (EuroPython 2012) Python White Magic July 3 2012 18 / 47

    View full-size slide

  30. Problem #2: the PyPy AST compiler
    AST
    stmt
    FunctionDef Print Assign ...
    expr
    UnaryOp BinaryOp IfExpr ...
    ...
    antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

    View full-size slide

  31. Problem #2: the PyPy AST compiler
    AST
    stmt
    FunctionDef Print Assign ...
    expr
    UnaryOp BinaryOp IfExpr ...
    ...
    as_node_list()
    set_context()
    build_container()
    get_generators()
    ...
    as_constant()
    accept_jump_if()
    ...
    antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

    View full-size slide

  32. Problem #2: the PyPy AST compiler
    AST
    stmt
    FunctionDef Print Assign ...
    expr
    UnaryOp BinaryOp IfExpr ...
    ...
    as_node_list()
    set_context()
    build_container()
    get_generators()
    ...
    as_constant()
    accept_jump_if()
    ...
    asthelpers.py
    antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

    View full-size slide

  33. Problem #2: the PyPy AST compiler
    AST
    stmt
    FunctionDef Print Assign ...
    expr
    UnaryOp BinaryOp IfExpr ...
    ...
    as_node_list()
    set_context()
    build_container()
    get_generators()
    ...
    as_constant()
    accept_jump_if()
    ...
    asthelpers.py
    codegen.py
    antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

    View full-size slide

  34. Problem #2: the PyPy AST compiler
    AST
    stmt
    FunctionDef Print Assign ...
    expr
    UnaryOp BinaryOp IfExpr ...
    ...
    as_node_list()
    set_context()
    build_container()
    get_generators()
    ...
    as_constant()
    accept_jump_if()
    ...
    asthelpers.py
    codegen.py
    optimize.py
    antocuni (EuroPython 2012) Python White Magic July 3 2012 19 / 47

    View full-size slide

  35. Spell #2: __extend__ (1)
    python
    >>> from magic import extendabletype
    >>> class A(object):
    ... __metaclass__ = extendabletype
    ... def foo(self):
    ... print ’foo’
    ...
    >>> obj = A()
    >>> obj.foo()
    foo
    >>> class __extend__(A):
    ... def bar(self):
    ... print ’bar’
    ...
    >>> obj.foo()
    foo
    >>> obj.bar()
    bar
    antocuni (EuroPython 2012) Python White Magic July 3 2012 20 / 47

    View full-size slide

  36. Spell #2: __extend__ (1)
    python
    >>> from magic import extendabletype
    >>> class A(object):
    ... __metaclass__ = extendabletype
    ... def foo(self):
    ... print ’foo’
    ...
    >>> obj = A()
    >>> obj.foo()
    foo
    >>> class __extend__(A):
    ... def bar(self):
    ... print ’bar’
    ...
    >>> obj.foo()
    foo
    >>> obj.bar()
    bar
    antocuni (EuroPython 2012) Python White Magic July 3 2012 20 / 47

    View full-size slide

  37. Metaclasses for dummies (1)
    Everything is an object
    Every object has a type (a class)
    A class is an object
    The class of a class: metaclass
    class statement --> metaclass instantiation
    antocuni (EuroPython 2012) Python White Magic July 3 2012 21 / 47

    View full-size slide

  38. class statement
    class Foo(Bar):
    myvar = 42
    def foo(self):
    ....
    antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

    View full-size slide

  39. class statement
    class Foo(Bar):
    myvar = 42
    def foo(self):
    ....
    {
    'myvar': 42,
    'foo':
    }
    antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

    View full-size slide

  40. class statement
    class Foo(Bar):
    myvar = 42
    def foo(self):
    ....
    {
    'myvar': 42,
    'foo':
    }
    Foo = metaclass('Foo', (Bar,), dic)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

    View full-size slide

  41. class statement
    class Foo(Bar):
    myvar = 42
    def foo(self):
    ....
    {
    'myvar': 42,
    'foo':
    }
    Foo = metaclass('Foo', (Bar,), dic)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

    View full-size slide

  42. class statement
    class Foo(Bar):
    myvar = 42
    def foo(self):
    ....
    Foo = metaclass('Foo', (Bar,), dic)
    ?????
    antocuni (EuroPython 2012) Python White Magic July 3 2012 22 / 47

    View full-size slide

  43. Metaclasses for dummies (2)
    Find the metaclass (simplified)
    __metaclass__ in the class body
    type(bases[0]) (if any)
    global __metaclass__
    Call it!
    subclass of type (usually)
    override __new__
    tweak the parameters?
    antocuni (EuroPython 2012) Python White Magic July 3 2012 23 / 47

    View full-size slide

  44. Metaclasses for dummies (2)
    Find the metaclass (simplified)
    __metaclass__ in the class body
    type(bases[0]) (if any)
    global __metaclass__
    Call it!
    subclass of type (usually)
    override __new__
    tweak the parameters?
    antocuni (EuroPython 2012) Python White Magic July 3 2012 23 / 47

    View full-size slide

  45. Spell #2: __extend__ (2)
    example
    class A(object):
    __metaclass__ = extendabletype
    class __extend__(A):
    def bar(self): print ’bar’
    magic.py
    class extendabletype(type):
    def __new__(cls, name, bases, dict):
    if name == ’__extend__’:
    for cls in bases:
    for key, value in dict.items():
    if key == ’__module__’:
    continue
    setattr(cls, key, value)
    return None
    else:
    return type.__new__(cls, name, bases, dict)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 24 / 47

    View full-size slide

  46. Spell #2: __extend__ (2)
    example
    class A(object):
    __metaclass__ = extendabletype
    class __extend__(A):
    def bar(self): print ’bar’
    magic.py
    class extendabletype(type):
    def __new__(cls, name, bases, dict):
    if name == ’__extend__’:
    for cls in bases:
    for key, value in dict.items():
    if key == ’__module__’:
    continue
    setattr(cls, key, value)
    return None
    else:
    return type.__new__(cls, name, bases, dict)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 24 / 47

    View full-size slide

  47. Problem #2 solved
    astcompiler/
    # ast.py
    class AST(...):
    __metaclass__ = extendabletype
    ...
    class expr(AST): ...
    class stmt(AST): ...
    # asthelpers.py
    class __extend__(AST):
    def as_node_list(...): ...
    def set_context(...): ...
    class __extend__(expr): ...
    class __extend__(stmt): ...
    # codegen.py ...
    # optimize.py ...
    Pros
    better organization
    well-contained
    easy to understand
    Cons
    naming convention
    import-time side
    effects
    unobvious at first
    antocuni (EuroPython 2012) Python White Magic July 3 2012 25 / 47

    View full-size slide

  48. Problem #2 solved
    astcompiler/
    # ast.py
    class AST(...):
    __metaclass__ = extendabletype
    ...
    class expr(AST): ...
    class stmt(AST): ...
    # asthelpers.py
    class __extend__(AST):
    def as_node_list(...): ...
    def set_context(...): ...
    class __extend__(expr): ...
    class __extend__(stmt): ...
    # codegen.py ...
    # optimize.py ...
    Pros
    better organization
    well-contained
    easy to understand
    Cons
    naming convention
    import-time side
    effects
    unobvious at first
    antocuni (EuroPython 2012) Python White Magic July 3 2012 25 / 47

    View full-size slide

  49. Problem #2 solved
    astcompiler/
    # ast.py
    class AST(...):
    __metaclass__ = extendabletype
    ...
    class expr(AST): ...
    class stmt(AST): ...
    # asthelpers.py
    class __extend__(AST):
    def as_node_list(...): ...
    def set_context(...): ...
    class __extend__(expr): ...
    class __extend__(stmt): ...
    # codegen.py ...
    # optimize.py ...
    Pros
    better organization
    well-contained
    easy to understand
    Cons
    naming convention
    import-time side
    effects
    unobvious at first
    antocuni (EuroPython 2012) Python White Magic July 3 2012 25 / 47

    View full-size slide

  50. Simple is better than complex
    Use as much much magic as actually needed
    Example: Camelot admin classes
    Entity subclass for the data model
    Foo.Admin for UI stuff
    mixing business and presentation logic :-(
    model.py
    class User(Entity):
    name = Field(Unicode)
    age = Field(Integer)
    class Admin(EntityAdmin):
    verbose_name = ’User’
    field_list = [’name’, ’age’]
    antocuni (EuroPython 2012) Python White Magic July 3 2012 26 / 47

    View full-size slide

  51. Simple is better than complex
    Use as much much magic as actually needed
    Example: Camelot admin classes
    Entity subclass for the data model
    Foo.Admin for UI stuff
    mixing business and presentation logic :-(
    model.py
    class User(Entity):
    name = Field(Unicode)
    age = Field(Integer)
    class Admin(EntityAdmin):
    verbose_name = ’User’
    field_list = [’name’, ’age’]
    antocuni (EuroPython 2012) Python White Magic July 3 2012 26 / 47

    View full-size slide

  52. Problem #3: wrong solution
    model.py
    class User(Entity):
    __metaclass__ = \
    extendabletype
    name = Field(Unicode)
    age = Field(Integer)
    admin.py
    from model import User
    class __extend__(User):
    class Admin(EntityAdmin):
    verbose_name = ’User detail’
    field_list = [’name’, ’age’]
    Entity has its own metaclass
    metaclass conflict
    there is a simpler solution anyway
    antocuni (EuroPython 2012) Python White Magic July 3 2012 27 / 47

    View full-size slide

  53. Problem #3: wrong solution
    model.py
    class User(Entity):
    __metaclass__ = \
    extendabletype
    name = Field(Unicode)
    age = Field(Integer)
    admin.py
    from model import User
    class __extend__(User):
    class Admin(EntityAdmin):
    verbose_name = ’User detail’
    field_list = [’name’, ’age’]
    Entity has its own metaclass
    metaclass conflict
    there is a simpler solution anyway
    antocuni (EuroPython 2012) Python White Magic July 3 2012 27 / 47

    View full-size slide

  54. Problem #3: wrong solution
    model.py
    class User(Entity):
    __metaclass__ = \
    extendabletype
    name = Field(Unicode)
    age = Field(Integer)
    admin.py
    from model import User
    class __extend__(User):
    class Admin(EntityAdmin):
    verbose_name = ’User detail’
    field_list = [’name’, ’age’]
    Entity has its own metaclass
    metaclass conflict
    there is a simpler solution anyway
    antocuni (EuroPython 2012) Python White Magic July 3 2012 27 / 47

    View full-size slide

  55. Spell #3: attach_to (not really magical)
    python
    >>> from magic import attach_to
    >>>
    >>> class Foo(object):
    ... pass
    ...
    >>> @attach_to(Foo)
    ... class Admin(object):
    ... verbose_name = ’My Foo’
    ...
    >>> print Foo.Admin.verbose_name
    My Foo
    antocuni (EuroPython 2012) Python White Magic July 3 2012 28 / 47

    View full-size slide

  56. Decorators
    Syntactic sugar
    class decorators only for Python >= 2.6
    decor1.py
    @foo
    def myfunc():
    pass
    @bar(42)
    class MyClass(object):
    pass
    decor2.py
    def myfunc():
    pass
    myfunc = foo(myfunc)
    class MyClass(object):
    pass
    MyClass = bar(42)(MyClass)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 29 / 47

    View full-size slide

  57. Decorators
    Syntactic sugar
    class decorators only for Python >= 2.6
    decor1.py
    @foo
    def myfunc():
    pass
    @bar(42)
    class MyClass(object):
    pass
    decor2.py
    def myfunc():
    pass
    myfunc = foo(myfunc)
    class MyClass(object):
    pass
    MyClass = bar(42)(MyClass)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 29 / 47

    View full-size slide

  58. Spell #3: attach_to (not really magical)
    magic.py
    def attach_to(obj):
    def attach(cls):
    name = cls.__name__
    setattr(obj, name, cls)
    return attach
    antocuni (EuroPython 2012) Python White Magic July 3 2012 30 / 47

    View full-size slide

  59. Problem #3 solved
    model.py
    class User(Entity):
    name = Field(Unicode)
    age = Field(Integer)
    admin.py
    @attach_to(User):
    class Admin(EntityAdmin):
    verbose_name = ’User detail’
    field_list = [’name’, ’age’]
    Pros
    Simple is better than complex
    No metaclasses
    Much less magic
    Cons
    None :-)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 31 / 47

    View full-size slide

  60. Problem #3 solved
    model.py
    class User(Entity):
    name = Field(Unicode)
    age = Field(Integer)
    admin.py
    @attach_to(User):
    class Admin(EntityAdmin):
    verbose_name = ’User detail’
    field_list = [’name’, ’age’]
    Pros
    Simple is better than complex
    No metaclasses
    Much less magic
    Cons
    None :-)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 31 / 47

    View full-size slide

  61. Problem #3 solved
    model.py
    class User(Entity):
    name = Field(Unicode)
    age = Field(Integer)
    admin.py
    @attach_to(User):
    class Admin(EntityAdmin):
    verbose_name = ’User detail’
    field_list = [’name’, ’age’]
    Pros
    Simple is better than complex
    No metaclasses
    Much less magic
    Cons
    None :-)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 31 / 47

    View full-size slide

  62. Problem #4: quicker turnaround time (1)
    Developing new gdb commands
    gdbdemo.py
    import gdb
    class MyCommand(gdb.Command):
    def __init__(self):
    gdb.Command.__init__(self, "mycmd", gdb.COMMAND_NONE)
    def invoke(self, arg, from_tty):
    print ’Hello from Python’
    MyCommand() # register the command
    gdb
    $ PYTHONPATH="" gdb
    GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08
    ...
    (gdb) python import gdbdemo
    (gdb) mycmd
    Hello from Python
    (gdb)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 32 / 47

    View full-size slide

  63. Problem #4: quicker turnaround time (1)
    Developing new gdb commands
    gdbdemo.py
    import gdb
    class MyCommand(gdb.Command):
    def __init__(self):
    gdb.Command.__init__(self, "mycmd", gdb.COMMAND_NONE)
    def invoke(self, arg, from_tty):
    print ’Hello from Python’
    MyCommand() # register the command
    gdb
    $ PYTHONPATH="" gdb
    GNU gdb (Ubuntu/Linaro 7.3-0ubuntu2) 7.3-2011.08
    ...
    (gdb) python import gdbdemo
    (gdb) mycmd
    Hello from Python
    (gdb)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 32 / 47

    View full-size slide

  64. Problem #4: quicker turnaround time (2)
    Annoying to develop
    Several steps to do every time:
    start gdb
    start the program
    run until a certain point
    try your command
    antocuni (EuroPython 2012) Python White Magic July 3 2012 33 / 47

    View full-size slide

  65. Spell #4: reload & rebind __class__
    gdbdemo.py
    import gdb
    class MyCommand(gdb.Command):
    def __init__(self):
    gdb.Command.__init__(self, "mycmd", gdb.COMMAND_NONE)
    def invoke(self, arg, from_tty):
    import gdbdemo2
    reload(gdbdemo2)
    self.__class__ = gdbdemo2.MyCommand
    self.do_invoke(arg, from_tty)
    def do_invoke(self, arg, from_tty):
    print ’Hello from Python’
    MyCommand() # register the command
    antocuni (EuroPython 2012) Python White Magic July 3 2012 34 / 47

    View full-size slide

  66. Method lookup

    x
    y
    __class__
    ...

    __init__()
    invoke()
    do_invoke()
    ...
    MyCommand().invoke(...)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

    View full-size slide

  67. Method lookup

    x
    y
    __class__
    ...

    __init__()
    invoke()
    do_invoke()
    ...
    MyCommand().invoke(...)

    x
    y
    __class__
    ...
    antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

    View full-size slide

  68. Method lookup

    x
    y
    __class__
    ...

    __init__()
    invoke()
    do_invoke()
    ...
    MyCommand().invoke(...)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

    View full-size slide

  69. Method lookup

    x
    y
    __class__
    ...

    __init__()
    invoke()
    do_invoke()
    ...
    MyCommand().invoke(...)
    invoke()
    antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

    View full-size slide

  70. Method lookup

    x
    y
    __class__
    ...

    __init__()
    invoke()
    do_invoke()
    ...
    MyCommand().invoke(...)
    reload(gdbdemo)

    __init__()
    invoke()
    do_invoke()
    ...
    antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

    View full-size slide

  71. Method lookup

    x
    y
    __class__
    ...

    __init__()
    invoke()
    do_invoke()
    ...
    MyCommand().invoke(...)
    reload(gdbdemo)

    __init__()
    invoke()
    do_invoke()
    ...
    self.__class__ = gdbdemo.MyCommand
    antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

    View full-size slide

  72. Method lookup

    x
    y
    __class__
    ...

    __init__()
    invoke()
    do_invoke()
    ...
    MyCommand().invoke(...)
    reload(gdbdemo)

    __init__()
    invoke()
    do_invoke()
    ...
    self.__class__ = gdbdemo.MyCommand
    self.do_invoke(...)
    do_invoke()
    antocuni (EuroPython 2012) Python White Magic July 3 2012 35 / 47

    View full-size slide

  73. Problem #4 solved
    Only during development
    Pros
    Quicker turnaround
    Useful in various situations
    Cons
    Fragile
    Confusion++ (two different classes with the same name)
    Does not work in more complex cases
    antocuni (EuroPython 2012) Python White Magic July 3 2012 36 / 47

    View full-size slide

  74. Bonus spell
    How much time is left?
    bonus.py
    import time
    time.go_back(minutes=10)
    How much time is left?
    antocuni (EuroPython 2012) Python White Magic July 3 2012 37 / 47

    View full-size slide

  75. Bonus spell
    How much time is left?
    bonus.py
    import time
    time.go_back(minutes=10)
    How much time is left?
    antocuni (EuroPython 2012) Python White Magic July 3 2012 37 / 47

    View full-size slide

  76. Bonus spell
    How much time is left?
    bonus.py
    import time
    time.go_back(minutes=10)
    How much time is left?
    antocuni (EuroPython 2012) Python White Magic July 3 2012 37 / 47

    View full-size slide

  77. Problem #5: implicit global state
    elixir
    declarative layer on top of SQLAlchemy
    (precursor of sqlalchemy.declarative)
    model.py
    from elixir import (Entity, Field, Unicode, Integer,
    metadata, setup_all, create_all)
    metadata.bind = "sqlite:///db.sqlite"
    class User(Entity):
    name = Field(Unicode)
    age = Field(Integer)
    setup_all()
    create_all()
    # ...
    antocuni (EuroPython 2012) Python White Magic July 3 2012 38 / 47

    View full-size slide

  78. Problem #5: implicit global state
    elixir
    declarative layer on top of SQLAlchemy
    (precursor of sqlalchemy.declarative)
    model.py
    from elixir import (Entity, Field, Unicode, Integer,
    metadata, setup_all, create_all)
    metadata.bind = "sqlite:///db.sqlite"
    class User(Entity):
    name = Field(Unicode)
    age = Field(Integer)
    setup_all()
    create_all()
    # ...
    antocuni (EuroPython 2012) Python White Magic July 3 2012 38 / 47

    View full-size slide

  79. Global state
    Global state is evil
    that’s it
    implicit is even worse
    Hard to test
    no isolation for tests
    persistent side effects (e.g. DB)
    Goal
    multiple, isolated, independent DBs
    one DB per test (or group of tests)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 39 / 47

    View full-size slide

  80. Global state
    Global state is evil
    that’s it
    implicit is even worse
    Hard to test
    no isolation for tests
    persistent side effects (e.g. DB)
    Goal
    multiple, isolated, independent DBs
    one DB per test (or group of tests)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 39 / 47

    View full-size slide

  81. Global state
    Global state is evil
    that’s it
    implicit is even worse
    Hard to test
    no isolation for tests
    persistent side effects (e.g. DB)
    Goal
    multiple, isolated, independent DBs
    one DB per test (or group of tests)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 39 / 47

    View full-size slide

  82. A step forward (1)
    model.py
    import sqlalchemy
    from elixir import (Entity, Field, Unicode, Integer,
    setup_entities, GlobalEntityCollection)
    __session__ = sqlalchemy.orm.scoped_session(
    sqlalchemy.orm.sessionmaker())
    __metadata__ = sqlalchemy.MetaData(bind="sqlite:///db.sqlite")
    __collection__ = GlobalEntityCollection()
    class User(Entity):
    name = Field(Unicode)
    age = Field(Integer)
    if __name__ == ’__main__’:
    setup_entities(__collection__)
    __metadata__.create_all()
    antocuni (EuroPython 2012) Python White Magic July 3 2012 40 / 47

    View full-size slide

  83. A step forward (2)
    Still global state
    but explicit
    Goal: turn global state into local
    N indepentent
    __metadata__ & co.
    model.User
    Multiple copies of the same module
    Modules are singleton
    Cannot rely on import
    antocuni (EuroPython 2012) Python White Magic July 3 2012 41 / 47

    View full-size slide

  84. A step forward (2)
    Still global state
    but explicit
    Goal: turn global state into local
    N indepentent
    __metadata__ & co.
    model.User
    Multiple copies of the same module
    Modules are singleton
    Cannot rely on import
    antocuni (EuroPython 2012) Python White Magic July 3 2012 41 / 47

    View full-size slide

  85. import logic (simplified)
    import foo
    find foo.py
    this is a whole mess of its own :-)
    import foo
    mod = types.ModuleType(’foo’) # 1. create the module object
    mod.__file__ = ’/path/to/foo.py’
    sys.modules[’foo’] = mod # 2. update sys.modules
    src = open(’/path/to/foo.py’).read() # 3. compile&exec the code
    exec src in mod.__dict__
    antocuni (EuroPython 2012) Python White Magic July 3 2012 42 / 47

    View full-size slide

  86. import logic (simplified)
    import foo
    find foo.py
    this is a whole mess of its own :-)
    import foo
    mod = types.ModuleType(’foo’) # 1. create the module object
    mod.__file__ = ’/path/to/foo.py’
    sys.modules[’foo’] = mod # 2. update sys.modules
    src = open(’/path/to/foo.py’).read() # 3. compile&exec the code
    exec src in mod.__dict__
    antocuni (EuroPython 2012) Python White Magic July 3 2012 42 / 47

    View full-size slide

  87. Spell #5: import_model (1)
    db.py
    class Database(object):
    def __init__(self, url, modelfile):
    self.url = url
    self.session = sqlalchemy.orm.scoped_session(
    sqlalchemy.orm.sessionmaker())
    self.metadata = sqlalchemy.MetaData(bind=url)
    self.collection = elixir.GlobalEntityCollection()
    self.import_model(modelfile)
    elixir.setup_entities(self.collection)
    def import_model(self, filename):
    ...
    def create_all(self, *args, **kwds):
    self.metadata.create_all(*args, **kwds)
    db = Database(’sqlite:///db.sqlite’)
    db.create_all()
    myuser = db.model.User(’antocuni’, 30)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 43 / 47

    View full-size slide

  88. Spell #5: import_model (2)
    db.py
    def import_model(self, filename):
    # 1. create the module object
    self.model = types.ModuleType(’model’)
    self.model.__file__ = filename
    # 2. update sys.modules
    assert ’model’ not in sys.modules
    sys.modules[’model’] = self.model
    try:
    # inject the "local" state!
    self.model.__session__ = self.session
    self.model.__metadata__ = self.metadata
    self.model.__collection__ = self.collection
    # 3. compile&exec the code
    src = open(filename).read()
    code = compile(src, filename, ’exec’)
    exec src in self.model.__dict__
    finally:
    # (Python doesn’t do it usually :-))
    del sys.modules[’model’]
    antocuni (EuroPython 2012) Python White Magic July 3 2012 44 / 47

    View full-size slide

  89. Problem #5 solved
    test_model.py
    def test_user():
    url = "sqlite:///tmp/db.tmp"
    db = Database(url, ’/path/to/model.py’)
    db.create_all()
    myuser = db.model.User( ’antocuni’, 30)
    ...
    Pros
    testable
    multiple DB
    model.py kept simple
    magic well contained
    Cons
    complex
    db1.model.User != db2.model.User
    antocuni (EuroPython 2012) Python White Magic July 3 2012 45 / 47

    View full-size slide

  90. Problem #5 solved
    test_model.py
    def test_user():
    url = "sqlite:///tmp/db.tmp"
    db = Database(url, ’/path/to/model.py’)
    db.create_all()
    myuser = db.model.User( ’antocuni’, 30)
    ...
    Pros
    testable
    multiple DB
    model.py kept simple
    magic well contained
    Cons
    complex
    db1.model.User != db2.model.User
    antocuni (EuroPython 2012) Python White Magic July 3 2012 45 / 47

    View full-size slide

  91. Conclusion
    Magic might be helpful
    code reuse
    readability
    testability
    Use with care
    only if you know what you are doing
    lots of comments, please :-)
    Debugging is twice as hard as writing the code in the first place.
    Therefore, if you write the code as cleverly as possible, you are, by
    definition, not smart enough to debug it.
    (Brian Kernighan)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 46 / 47

    View full-size slide

  92. Conclusion
    Magic might be helpful
    code reuse
    readability
    testability
    Use with care
    only if you know what you are doing
    lots of comments, please :-)
    Debugging is twice as hard as writing the code in the first place.
    Therefore, if you write the code as cleverly as possible, you are, by
    definition, not smart enough to debug it.
    (Brian Kernighan)
    antocuni (EuroPython 2012) Python White Magic July 3 2012 46 / 47

    View full-size slide

  93. Contacts, Q&A
    http://bitbucket.org/antocuni/whitemagic
    twitter: @antocuni
    Available for consultancy & training:
    http://antocuni.eu
    [email protected]
    Any question?
    antocuni (EuroPython 2012) Python White Magic July 3 2012 47 / 47

    View full-size slide