Tweet
Tweet

Slide 1

Slide 1 text

Copyright (C) 2013, http://www.dabeaz.com Python 3 Metaprogramming 1 David Beazley @dabeaz http://www.dabeaz.com Presented at PyCon'2013, Santa Clara, CA March 14, 2013

Slide 2

Slide 2 text

Copyright (C) 2013, http://www.dabeaz.com Requirements 2 • Python 3.3 or more recent • Don't even attempt on any earlier version • Support files: http://www.dabeaz.com/py3meta

Slide 3

Slide 3 text

Copyright (C) 2013, http://www.dabeaz.com Welcome! • An advanced tutorial on two topics • Python 3 • Metaprogramming • Honestly, can you have too much of either? • No! 3

Slide 4

Slide 4 text

Copyright (C) 2013, http://www.dabeaz.com Metaprogramming • In a nutshell: code that manipulates code • Common examples: • Decorators • Metaclasses • Descriptors • Essentially, it's doing things with code 4

Slide 5

Slide 5 text

Copyright (C) 2013, http://www.dabeaz.com Why Would You Care? • Extensively used in frameworks and libraries • Better understanding of how Python works • It's fun • It solves a practical problem 5

Slide 6

Slide 6 text

Copyright (C) 2013, http://www.dabeaz.com 6 DRY

Slide 7

Slide 7 text

Copyright (C) 2013, http://www.dabeaz.com 7 DRY Don't Repeat Yourself

Slide 8

Slide 8 text

Copyright (C) 2013, http://www.dabeaz.com 8 DRY Don't Repeat Yourself Don't Repeat Yourself

Slide 9

Slide 9 text

Copyright (C) 2013, http://www.dabeaz.com Don't Repeat Yourself • Highly repetitive code sucks • Tedious to write • Hard to read • Difficult to modify 9

Slide 10

Slide 10 text

Copyright (C) 2013, http://www.dabeaz.com This Tutorial • A modern journey of metaprogramming • Highlight unique aspects of Python 3 • Explode your brain 10

Slide 11

Slide 11 text

Copyright (C) 2013, http://www.dabeaz.com Target Audience • Framework/library builders • Anyone who wants to know how things work 11

Slide 12

Slide 12 text

Copyright (C) 2013, http://www.dabeaz.com Reading 12 • Tutorial loosely based on content in "Python Cookbook, 3rd Ed." • Published May, 2013 • You'll find even more information in the book

Slide 13

Slide 13 text

Copyright (C) 2013, http://www.dabeaz.com Caution • If you do everything in this tutorial all at once • You will either be fired • Or have permanent job security • It's a fine line 13

Slide 14

Slide 14 text

Copyright (C) 2013, http://www.dabeaz.com Preliminaries 14

Slide 15

Slide 15 text

Copyright (C) 2013, http://www.dabeaz.com Basic Building Blocks 15 def func(args): statement1 statement2 statement3 ... class A: def method1(self, args): statement1 statement2 def method2(self, args): statement1 statement2 ... statement1 statement2 statement3 ... Code

Slide 16

Slide 16 text

Copyright (C) 2013, http://www.dabeaz.com Statements 16 statement1 statement2 statement3 ... • Perform the actual work of your program • Always execute in two scopes • globals - Module dictionary • locals - Enclosing function (if any) • exec(statements [, globals [, locals]])

Slide 17

Slide 17 text

Copyright (C) 2013, http://www.dabeaz.com Functions 17 def func(x, y, z): statement1 statement2 statement3 ... • The fundamental unit of code in most programs • Module-level functions • Methods of classes

Slide 18

Slide 18 text

Copyright (C) 2013, http://www.dabeaz.com Calling Conventions 18 def func(x, y, z): statement1 statement2 statement3 ... • Positional arguments func(1, 2, 3) • Keyword arguments func(x=1, z=3, y=2)

Slide 19

Slide 19 text

Copyright (C) 2013, http://www.dabeaz.com Default Arguments 19 def func(x, debug=False, names=None): if names is None: names = [] ... func(1) func(1, names=['x', 'y']) • Default values set at definition time • Only use immutable values (e.g., None)

Slide 20

Slide 20 text

Copyright (C) 2013, http://www.dabeaz.com *args and **kwargs 20 def func(*args, **kwargs): # args is tuple of position args # kwargs is dict of keyword args ... func(1, 2, x=3, y=4, z=5) args = (1, 2) kwargs = { 'x': 3, 'y': 4, 'z': 5 }

Slide 21

Slide 21 text

Copyright (C) 2013, http://www.dabeaz.com *args and **kwargs 21 args = (1, 2) kwargs = { 'x': 3, 'y': 4, 'z': 5 } func(*args, **kwargs) func(1, 2, x=3, y=4, z=5) same as

Slide 22

Slide 22 text

Copyright (C) 2013, http://www.dabeaz.com Keyword-Only Args 22 def recv(maxsize, *, block=True): ... def sum(*args, initial=0): ... • Named arguments appearing after '*' can only be passed by keyword recv(8192, block=False) # Ok recv(8192, False) # Error

Slide 23

Slide 23 text

Copyright (C) 2013, http://www.dabeaz.com Closures 23 def make_adder(x, y): def add(): return x + y return add • You can make and return functions • Local variables are captured >>> a = make_adder(2, 3) >>> b = make_adder(10, 20) >>> a() 5 >>> b() 30 >>>

Slide 24

Slide 24 text

Copyright (C) 2013, http://www.dabeaz.com Classes 24 class Spam: a = 1 def __init__(self, b): self.b = b def imethod(self): pass >>> Spam.a # Class variable 1 >>> s = Spam(2) >>> s.b # Instance variable 2 >>> s.imethod() # Instance method >>>

Slide 25

Slide 25 text

Copyright (C) 2013, http://www.dabeaz.com Different Method Types 25 class Spam: def imethod(self): pass @classmethod def cmethod(cls): pass @staticmethod def smethod(): pass s = Spam() s.imethod() Spam.cmethod() Spam.smethod() Usage

Slide 26

Slide 26 text

Copyright (C) 2013, http://www.dabeaz.com Special Methods 26 class Array: def __getitem__(self, index): ... def __setitem__(self, index, value): ... def __delitem__(self, index): ... def __contains__(self, item): ... • Almost everything can be customized

Slide 27

Slide 27 text

Copyright (C) 2013, http://www.dabeaz.com Inheritance 27 class Base: def spam(self): ... class Foo(Base): def spam(self): ... # Call method in base class r = super().spam()

Slide 28

Slide 28 text

Copyright (C) 2013, http://www.dabeaz.com Dictionaries 28 class Spam: def __init__(self, x, y): self.x = x self.y = y def foo(self): pass • Objects are layered on dictionaries • Example: >>> s = Spam(2,3) >>> s.__dict__ {'y': 3, 'x': 2} >>> Spam.__dict__['foo'] >>>

Slide 29

Slide 29 text

Copyright (C) 2013, http://www.dabeaz.com 29 "I love the smell of debugging in the morning." Metaprogramming Basics

Slide 30

Slide 30 text

Copyright (C) 2013, http://www.dabeaz.com Problem: Debugging 30 • Will illustrate basics with a simple problem • Debugging • Not the only application, but simple enough to fit on slides

Slide 31

Slide 31 text

Copyright (C) 2013, http://www.dabeaz.com Debugging with Print 31 def add(x, y): return x + y • A function • A function with debugging def add(x, y): print('add') return x + y • The one and only true way to debug...

Slide 32

Slide 32 text

Copyright (C) 2013, http://www.dabeaz.com Many Functions w/ Debug 32 def add(x, y): print('add') return x + y def sub(x, y): print('sub') return x - y def mul(x, y): print('mul') return x * y def div(x, y): print('div') return x / y

Slide 33

Slide 33 text

Copyright (C) 2013, http://www.dabeaz.com Many Functions w/ Debug 33 def add(x, y): print('add') return x + y def sub(x, y): print('sub') return x - y def mul(x, y): print('mul') return x * y def div(x, y): print('div') return x / y Bleah!

Slide 34

Slide 34 text

Copyright (C) 2013, http://www.dabeaz.com Decorators • A decorator is a function that creates a wrapper around another function • The wrapper is a new function that works exactly like the original function (same arguments, same return value) except that some kind of extra processing is carried out 34

Slide 35

Slide 35 text

Copyright (C) 2013, http://www.dabeaz.com A Debugging Decorator 35 from functools import wraps def debug(func): msg = func.__qualname__ @wraps(func) def wrapper(*args, **kwargs): print(msg) return func(*args, **kwargs) return wrapper • Application (wrapping) func = debug(func)

Slide 36

Slide 36 text

Copyright (C) 2013, http://www.dabeaz.com A Debugging Decorator 36 from functools import wraps def debug(func): msg = func.__qualname__ @wraps(func) def wrapper(*args, **kwargs): print(msg) return func(*args, **kwargs) return wrapper A decorator creates a "wrapper" function

Slide 37

Slide 37 text

Copyright (C) 2013, http://www.dabeaz.com A Debugging Decorator 37 from functools import wraps def debug(func): msg = func.__qualname__ @wraps(func) def wrapper(*args, **kwargs): print(msg) return func(*args, **kwargs) return wrapper A decorator creates a "wrapper" function A decorator creates a "wrapper" function Around a function that you provide

Slide 38

Slide 38 text

Copyright (C) 2013, http://www.dabeaz.com Function Metadata 38 from functools import wraps def debug(func): msg = func.__qualname__ @wraps(func) def wrapper(*args, **kwargs): print(msg) return func(*args, **kwargs) return wrapper • @wraps copies metadata • Name and doc string • Function attributes

Slide 39

Slide 39 text

Copyright (C) 2013, http://www.dabeaz.com The Metadata Problem • If you don't use @wraps, weird things happen 39 def add(x,y): "Adds x and y" return x+y add = debug(add) >>> add.__qualname__ 'wrapper' >>> add.__doc__ >>> help(add) Help on function wrapper in module __main__: wrapper(*args, **kwargs) >>>

Slide 40

Slide 40 text

Copyright (C) 2013, http://www.dabeaz.com Decorator Syntax • The definition of a function and wrapping almost always occur together 40 • @decorator syntax performs the same steps def add(x,y): return x+y add = debug(add) @debug def add(x,y): return x+y

Slide 41

Slide 41 text

Copyright (C) 2013, http://www.dabeaz.com Example Use 41 @debug def add(x, y): return x + y @debug def sub(x, y): return x - y @debug def mul(x, y): return x * y @debug def div(x, y): return x / y

Slide 42

Slide 42 text

Copyright (C) 2013, http://www.dabeaz.com @debug def add(x, y): return x + y @debug def sub(x, y): return x - y @debug def mul(x, y): return x * y @debug def div(x, y): return x / y Example Use 42 Each function is decorated, but there are no other implementation details

Slide 43

Slide 43 text

Copyright (C) 2013, http://www.dabeaz.com Big Picture • Debugging code is isolated to single location • This makes it easy to change (or to disable) • User of a decorator doesn't worry about it • That's really the whole idea 43

Slide 44

Slide 44 text

Copyright (C) 2013, http://www.dabeaz.com Variation: Logging 44 from functools import wraps import logging def debug(func): log = logging.getLogger(func.__module__) msg = func.__qualname__ @wraps(func) def wrapper(*args, **kwargs): log.debug(msg) return func(*args, **kwargs) return wrapper

Slide 45

Slide 45 text

Copyright (C) 2013, http://www.dabeaz.com Variation: Optional Disable 45 from functools import wraps import os def debug(func): if 'DEBUG' not in os.environ: return func msg = func.__qualname__ @wraps(func) def wrapper(*args, **kwargs): print(msg) return func(*args, **kwargs) return wrapper • Key idea: Can change decorator independently of code that uses it

Slide 46

Slide 46 text

Copyright (C) 2013, http://www.dabeaz.com Debugging with Print 46 • Everyone knows you really need a prefix • A function with debugging def add(x, y): print('add') return x + y def add(x, y): print('***add') return x + y • You know, for grepping...

Slide 47

Slide 47 text

Copyright (C) 2013, http://www.dabeaz.com Decorators with Args 47 • Evaluates as • Calling convention @decorator(args) def func(): pass func = decorator(args)(func) • It's a little weird--two levels of calls

Slide 48

Slide 48 text

Copyright (C) 2013, http://www.dabeaz.com Decorators with Args 48 from functools import wraps def debug(prefix=''): def decorate(func): msg = prefix + func.__qualname__ @wraps(func) def wrapper(*args, **kwargs): print(msg) return func(*args, **kwargs) return wrapper return decorate • Usage @debug(prefix='***') def add(x,y): return x+y

Slide 49

Slide 49 text

Copyright (C) 2013, http://www.dabeaz.com Decorators with Args 49 from functools import wraps def debug(prefix=''): def decorate(func): msg = prefix + func.__qualname__ @wraps(func) def wrapper(*args, **kwargs): print(msg) return func(*args, **kwargs) return wrapper return decorate Outer function defines variables for use in regular decorator Normal decorator function

Slide 50

Slide 50 text

Copyright (C) 2013, http://www.dabeaz.com A Reformulation 50 from functools import wraps, partial def debug(func=None, *, prefix=''): if func is None: return partial(debug, prefix=prefix) msg = prefix + func.__qualname__ @wraps(func) def wrapper(*args, **kwargs): print(msg) return func(*args, **kwargs) return wrapper • A test of your function calling skills...

Slide 51

Slide 51 text

Copyright (C) 2013, http://www.dabeaz.com Usage 51 @debug def add(x, y): return x + y • Use as a simple decorator • Or as a decorator with optional configuration @debug(prefix='***') def add(x, y): return x + y

Slide 52

Slide 52 text

Copyright (C) 2013, http://www.dabeaz.com Debug All Of This 52 class Spam: @debug def grok(self): pass @debug def bar(self): pass @debug def foo(self): pass • Debug all of the methods of a class • Can you decorate all methods at once?

Slide 53

Slide 53 text

Copyright (C) 2013, http://www.dabeaz.com Class Decorator 53 def debugmethods(cls): for name, val in vars(cls).items(): if callable(val): setattr(cls, name, debug(val)) return cls • Idea: • Walk through class dictionary • Identify callables (e.g., methods) • Wrap with a decorator

Slide 54

Slide 54 text

Copyright (C) 2013, http://www.dabeaz.com Example Use 54 @debugmethods class Spam: def grok(self): pass def bar(self): pass def foo(self): pass • One decorator application • Covers all definitions within the class • It even mostly works...

Slide 55

Slide 55 text

Copyright (C) 2013, http://www.dabeaz.com Limitations 55 @debugmethods class BrokenSpam: @classmethod def grok(cls): # Not wrapped pass @staticmethod def bar(): # Not wrapped pass • Only instance methods get wrapped • Why? An exercise for the reader...

Slide 56

Slide 56 text

Copyright (C) 2013, http://www.dabeaz.com Variation: Debug Access 56 def debugattr(cls): orig_getattribute = cls.__getattribute__ def __getattribute__(self, name): print('Get:', name) return orig_getattribute(self, name) cls.__getattribute__ = __getattribute__ return cls • Rewriting part of the class itself

Slide 57

Slide 57 text

Copyright (C) 2013, http://www.dabeaz.com Example 57 @debugattr class Point: def __init__(self, x, y): self.x = x self.y = y >>> p = Point(2, 3) >>> p.x Get: x 2 >>> p.y Get: y 3 >>>

Slide 58

Slide 58 text

Copyright (C) 2013, http://www.dabeaz.com Debug All The Classes 58 @debugmethods class Base: ... @debugmethods class Spam(Base): ... @debugmethods class Grok(Spam): ... @debugmethods class Mondo(Grok): ... • Many classes with debugging • Didn't we just solve this? • Bleah!!

Slide 59

Slide 59 text

Copyright (C) 2013, http://www.dabeaz.com Solution: A Metaclass 59 class debugmeta(type): def __new__(cls, clsname, bases, clsdict): clsobj = super().__new__(cls, clsname, bases, clsdict) clsobj = debugmethods(clsobj) return clsobj class Base(metaclass=debugmeta): ... class Spam(Base): ... • Usage

Slide 60

Slide 60 text

Copyright (C) 2013, http://www.dabeaz.com Solution: A Metaclass 60 class debugmeta(type): def __new__(cls, clsname, bases, clsdict): clsobj = super().__new__(cls, clsname, bases, clsdict) clsobj = debugmethods(clsobj) return clsobj • Idea • Class gets created normally

Slide 61

Slide 61 text

Copyright (C) 2013, http://www.dabeaz.com Solution: A Metaclass 61 class debugmeta(type): def __new__(cls, clsname, bases, clsdict): clsobj = super().__new__(cls, clsname, bases, clsdict) clsobj = debugmethods(clsobj) return clsobj • Idea • Class gets created normally • Immediately wrapped by class decorator

Slide 62

Slide 62 text

Copyright (C) 2013, http://www.dabeaz.com 62

Slide 63

Slide 63 text

Copyright (C) 2013, http://www.dabeaz.com Types • All values in Python have a type • Example: >>> x = 42 >>> type(x) >>> s = "Hello" >>> type(s) >>> items = [1,2,3] >>> type(items) >>> 63

Slide 64

Slide 64 text

Copyright (C) 2013, http://www.dabeaz.com Types and Classes • Classes define new types class Spam: pass >>> s = Spam() >>> type(s) >>> 64 • The class is the type of instances created • The class is a callable that creates instances

Slide 65

Slide 65 text

Copyright (C) 2013, http://www.dabeaz.com Types of Classes • Classes are instances of types >>> type(int) >>> type(list) >>> type(Spam) >>> isinstance(Spam, type) True >>> 65 • This requires some thought, but it should make some sense (classes are types)

Slide 66

Slide 66 text

Copyright (C) 2013, http://www.dabeaz.com Creating Types • Types are their own class (builtin) class type: ... >>> type >>> 66 • This class creates new "type" objects • Used when defining classes

Slide 67

Slide 67 text

Copyright (C) 2013, http://www.dabeaz.com Classes Deconstructed • Consider a class: 67 class Spam(Base): def __init__(self, name): self.name = name def bar(self): print "I'm Spam.bar" • What are its components? • Name ("Spam") • Base classes (Base,) • Functions (__init__,bar)

Slide 68

Slide 68 text

Copyright (C) 2013, http://www.dabeaz.com Class Definition Process • What happens during class definition? • Step1: Body of class is isolated body = ''' def __init__(self, name): self.name = name def bar(self): print "I'm Spam.bar" ''' 68 class Spam(Base): def __init__(self, name): self.name = name def bar(self): print "I'm Spam.bar"

Slide 69

Slide 69 text

Copyright (C) 2013, http://www.dabeaz.com Class Definition • Step 2: The class dictionary is created clsdict = type.__prepare__('Spam', (Base,)) • This dictionary serves as local namespace for statements in the class body • By default, it's a simple dictionary (more later) 69

Slide 70

Slide 70 text

Copyright (C) 2013, http://www.dabeaz.com Class Definition • Step 3: Body is executed in returned dict exec(body, globals(), clsdict) • Afterwards, clsdict is populated >>> clsdict {'__init__': , 'bar': } >>> 70

Slide 71

Slide 71 text

Copyright (C) 2013, http://www.dabeaz.com Class Definition • Step 4: Class is constructed from its name, base classes, and the dictionary >>> Spam = type('Spam', (Base,), clsdict) >>> Spam >>> s = Spam('Guido') >>> s.bar() I'm Spam.bar >>> 71

Slide 72

Slide 72 text

Copyright (C) 2013, http://www.dabeaz.com Changing the Metaclass • metaclass keyword argument • Sets the class used for creating the type class Spam(metaclass=type): def __init__(self,name): self.name = name def bar(self): print "I'm Spam.bar" 72 • By default, it's set to 'type', but you can change it to something else

Slide 73

Slide 73 text

Copyright (C) 2013, http://www.dabeaz.com Defining a New Metaclass • You typically inherit from type and redefine __new__ or __init__ class mytype(type): def __new__(cls, name, bases, clsdict): clsobj = super().__new__(cls, name, bases, clsdict) return clsobj 73 • To use class Spam(metaclass=mytype): ...

Slide 74

Slide 74 text

Copyright (C) 2013, http://www.dabeaz.com Using a Metaclass • Metaclasses get information about class definitions at the time of definition • Can inspect this data • Can modify this data • Essentially, similar to a class decorator • Question: Why would you use one? 74

Slide 75

Slide 75 text

Copyright (C) 2013, http://www.dabeaz.com Inheritance • Metaclasses propagate down hierarchies class Base(metaclass=mytype): ... class Spam(Base): # metaclass=mytype ... class Grok(Spam): # metaclass=mytype ... 75 • Think of it as a genetic mutation

Slide 76

Slide 76 text

Copyright (C) 2013, http://www.dabeaz.com Solution: Reprise 76 class debugmeta(type): def __new__(cls, clsname, bases, clsdict): clsobj = super().__new__(cls, clsname, bases, clsdict) clsobj = debugmethods(clsobj) return clsobj • Idea • Class gets created normally • Immediately wrapped by class decorator

Slide 77

Slide 77 text

Copyright (C) 2013, http://www.dabeaz.com Debug The Universe 77 class Base(metaclass=debugmeta): ... class Spam(Base): ... class Grok(Spam): ... class Mondo(Grok): ... • Debugging gets applied across entire hierarchy • Implicitly applied in subclasses

Slide 78

Slide 78 text

Copyright (C) 2013, http://www.dabeaz.com Big Picture • It's mostly about wrapping/rewriting • Decorators : Functions • Class Decorators: Classes • Metaclasses : Class hierarchies • You have the power to change things 78

Slide 79

Slide 79 text

Copyright (C) 2013, http://www.dabeaz.com Interlude 79

Slide 80

Slide 80 text

Copyright (C) 2013, http://www.dabeaz.com Journey So Far 80 • Have seen "classic" metaprogramming • Already widely used in Python 2 • Only a few Python 3 specific changes

Slide 81

Slide 81 text

Copyright (C) 2013, http://www.dabeaz.com Journey to Come 81 • Let's build something more advanced • Using techniques discussed • And more...

Slide 82

Slide 82 text

Copyright (C) 2013, http://www.dabeaz.com Problem : Structures 82 class Stock: def __init__(self, name, shares, price): self.name = name self.shares = shares self.price = price class Point: def __init__(self, x, y): self.x = x self.y = y class Host: def __init__(self, address, port): self.address = address self.port = port

Slide 83

Slide 83 text

Copyright (C) 2013, http://www.dabeaz.com Problem : Structures 83 class Stock: def __init__(self, name, shares, price): self.name = name self.shares = shares self.price = price class Point: def __init__(self, x, y): self.x = x self.y = y class Host: def __init__(self, address, port): self.address = address self.port = port Why must I keep writing these boilerplate init methods?

Slide 84

Slide 84 text

Copyright (C) 2013, http://www.dabeaz.com A Solution : Inheritance 84 class Structure: _fields = [] def __init__(self, *args): if len(args) != self._fields: raise TypeError('Wrong # args') for name, val in zip(self._fields, args): setattr(self, name, val) class Stock(Structure): _fields = ['name', 'shares', 'price'] class Point(Structure): _fields = ['x', 'y'] class Host(Structure): _fields = ['address', 'port'] A generalized __init__()

Slide 85

Slide 85 text

Copyright (C) 2013, http://www.dabeaz.com Usage 85 >>> s = Stock('ACME', 50, 123.45) >>> s.name 'ACME' >>> s.shares 50 >>> s.price 123.45 >>> p = Point(4, 5) >>> p.x 4 >>> p.y 5 >>>

Slide 86

Slide 86 text

Copyright (C) 2013, http://www.dabeaz.com Some Issues 86 >>> s = Stock('ACME', price=123.45, shares=50) Traceback (most recent call last): File "", line 1, in TypeError: __init__() got an unexpected keyword argument 'shares' >>> • No support for keyword args • Missing calling signatures >>> import inspect >>> print(inspect.signature(Stock)) (*args) >>>

Slide 87

Slide 87 text

Copyright (C) 2013, http://www.dabeaz.com 87 Put a Signature on It

Slide 88

Slide 88 text

Copyright (C) 2013, http://www.dabeaz.com New Approach: Signatures 88 from inspect import Parameter, Signature fields = ['name', 'shares', 'price'] parms = [ Parameter(name, Parameter.POSITIONAL_OR_KEYWORD) for name in fields] sig = Signature(parms) • Build a function signature object • Signatures are more than just metadata

Slide 89

Slide 89 text

Copyright (C) 2013, http://www.dabeaz.com Signature Binding 89 def func(*args, **kwargs): bound_args = sig.bind(*args, **kwargs) for name, val in bound_args.arguments.items(): print(name, '=', val) • Argument binding • sig.bind() binds positional/keyword args to signature • .arguments is an OrderedDict of passed values

Slide 90

Slide 90 text

Copyright (C) 2013, http://www.dabeaz.com Signature Binding 90 • Example use: >>> func('ACME', 50, 91.1) name = ACME shares = 50 price = 91.1 >>> func('ACME', price=91.1, shares=50) name = ACME shares = 50 price = 91.1 • Notice: both positional/keyword args work

Slide 91

Slide 91 text

Copyright (C) 2013, http://www.dabeaz.com Signature Binding 91 • Error handling >>> func('ACME', 50) Traceback (most recent call last): ... TypeError: 'price' parameter lacking default value >>> func('ACME', 50, 91.1, 92.3) Traceback (most recent call last): ... TypeError: too many positional arguments >>> • Binding: it just "works"

Slide 92

Slide 92 text

Copyright (C) 2013, http://www.dabeaz.com Solution w/Signatures 92 from inspect import Parameter, Signature def make_signature(names): return Signature( Parameter(name, Parameter.POSITIONAL_OR_KEYWORD) for name in names) class Structure: __signature__ = make_signature([]) def __init__(self, *args, **kwargs): bound = self.__signature__.bind( *args, **kwargs) for name, val in bound.arguments.items(): setattr(self, name, val)

Slide 93

Slide 93 text

Copyright (C) 2013, http://www.dabeaz.com Solution w/Signatures 93 class Stock(Structure): __signature__ = make_signature( ['name','shares','price']) class Point(Structure): __signature__ = make_signature(['x', 'y']) class Host(Structure): __signature__ = make_signature( ['address', 'port'])

Slide 94

Slide 94 text

Copyright (C) 2013, http://www.dabeaz.com Solution w/Signatures 94 >>> s = Stock('ACME', shares=50, price=91.1) >>> s.name 'ACME' >>> s.shares 50 >>> s.price 91.1 >>> import inspect >>> print(inspect.signature(Stock)) (name, shares, price) >>>

Slide 95

Slide 95 text

Copyright (C) 2013, http://www.dabeaz.com New Problem 95 class Stock(Structure): __signature__ = make_signature( ['name','shares','price']) class Point(Structure): __signature__ = make_signature(['x', 'y']) class Host(Structure): __signature__ = make_signature( ['address', 'port']) • This is rather annoying • Can't it be simplified in some way?

Slide 96

Slide 96 text

Copyright (C) 2013, http://www.dabeaz.com Solutions 96 • Ah, a problem involving class definitions • Class decorators • Metaclasses • Which seems more appropriate? • Let's explore both options

Slide 97

Slide 97 text

Copyright (C) 2013, http://www.dabeaz.com Class Decorators 97 def add_signature(*names): def decorate(cls): cls.__signature__ = make_signature(names) return cls return decorate • Usage: @add_signature('name','shares','price') class Stock(Structure): pass @add_signature('x','y') class Point(Structure): pass

Slide 98

Slide 98 text

Copyright (C) 2013, http://www.dabeaz.com Metaclass Solution 98 class StructMeta(type): def __new__(cls, name, bases, clsdict): clsobj = super().__new__(cls, name, bases, clsdict) sig = make_signature(clsobj._fields) setattr(clsobj, '__signature__', sig) return clsobj class Structure(metaclass=StructMeta): _fields = [] def __init__(self, *args, **kwargs): bound = self.__signature__.bind( *args, **kwargs) for name, val in bound.arguments.items(): setattr(self, name, val)

Slide 99

Slide 99 text

Copyright (C) 2013, http://www.dabeaz.com Metaclass Solution 99 class StructMeta(type): def __new__(cls, name, bases, clsdict): clsobj = super().__new__(cls, name, bases, clsdict) sig = make_signature(clsobj._fields) setattr(clsobj, '__signature__', sig) return clsobj class Structure(metaclass=StructMeta): _fields = [] def __init__(self, *args, **kwargs): bound = self.__signature__.bind( *args, **kwargs) for name, val in bound.arguments.items(): setattr(self, name, val) Read _fields attribute and make a proper signature out of it

Slide 100

Slide 100 text

Copyright (C) 2013, http://www.dabeaz.com Usage 100 class Stock(Structure): _fields = ['name', 'shares', 'price'] class Point(Structure): _fields = ['x', 'y'] class Host(Structure): _fields = ['address', 'port'] • It's back to original 'simple' solution • Signatures are created behind scenes

Slide 101

Slide 101 text

Copyright (C) 2013, http://www.dabeaz.com Considerations 101 class Structure(metaclass=StructMeta): _fields = [] ... def __repr__(self): args = ', '.join(repr(getattr(self, name)) for name in self._fields) return type(self).__name__ + \ '(' + args + ')' • How much will the Structure class be expanded? • Example: supporting methods • Is type checking important? isinstance(s, Structure)

Slide 102

Slide 102 text

Copyright (C) 2013, http://www.dabeaz.com Advice 102 • Use a class decorator if the goal is to tweak classes that might be unrelated • Use a metaclass if you're trying to perform actions in combination with inheritance • Don't be so quick to dismiss techniques (e.g., 'metaclasses suck so .... blah blah') • All of the tools are meant to work together

Slide 103

Slide 103 text

Copyright (C) 2013, http://www.dabeaz.com Owning the Dot 103 Q: "Who's in charge here?" A: "In charge? I don't know, man."

Slide 104

Slide 104 text

Copyright (C) 2013, http://www.dabeaz.com Problem : Correctness 104 >>> s = Stock('ACME', 50, 91.1) >>> s.name = 42 >>> s.shares = 'a heck of a lot' >>> s.price = (23.45 + 2j) >>> • Types like a duck, rhymes with ... • Bah, real programmers use Haskell!

Slide 105

Slide 105 text

Copyright (C) 2013, http://www.dabeaz.com Properties 105 class Stock(Structure): _fields = ['name', 'shares', 'price'] @property def shares(self): return self._shares @shares.setter def shares(self, value): if not isinstance(value, int): raise TypeError('expected int') if value < 0: raise ValueError('Must be >= 0') self._shares = value • You can upgrade attributes to have checks (getter) (setter)

Slide 106

Slide 106 text

Copyright (C) 2013, http://www.dabeaz.com Properties 106 >>> s = Stock('ACME', 50, 91.1) >>> s.shares = 'a lot' Traceback (most recent call last): ... TypeError: expected int >>> s.shares = -10 Traceback (most recent call last): ... ValueError: Must be >= 0 >>> s.shares = 37 >>> s.shares 37 >>> • Example use:

Slide 107

Slide 107 text

Copyright (C) 2013, http://www.dabeaz.com An Issue 107 @property def shares(self): return self._shares @shares.setter def shares(self, value): if not isinstance(value, int): raise TypeError('expected int') if value < 0: raise ValueError('Must be >= 0') self._shares = value • It works, but it quickly gets annoying • Imagine writing same code for many attributes

Slide 108

Slide 108 text

Copyright (C) 2013, http://www.dabeaz.com A Complexity 108 • Want to simplify, but how? • Two kinds of checking are intertwined • Type checking: int, float, str, etc. • Validation: >, >=, <, <=, !=, etc. • Question: How to structure it?

Slide 109

Slide 109 text

Copyright (C) 2013, http://www.dabeaz.com Descriptor Protocol 109 • Properties are implemented via descriptors class Descriptor: def __get__(self, instance, cls): ... def __set__(self, instance, value): ... def __delete__(self, instance) ... • Customized processing of attribute access

Slide 110

Slide 110 text

Copyright (C) 2013, http://www.dabeaz.com Descriptor Protocol 110 • Example: class Spam: x = Descriptor() s = Spam() s.x # x.__get__(s, Spam) s.x = value # x.__set__(s, value) del s.x # x.__delete__(s) • Customized handling of a specific attribute

Slide 111

Slide 111 text

Copyright (C) 2013, http://www.dabeaz.com A Basic Descriptor 111 class Descriptor: def __init__(self, name=None): self.name = name def __get__(self, instance, cls): if instance is None: return self else: return instance.__dict__[self.name] def __set__(self, instance, value): instance.__dict__[self.name] = value def __delete__(self, instance): del instance.__dict__[self.name]

Slide 112

Slide 112 text

Copyright (C) 2013, http://www.dabeaz.com A Basic Descriptor 112 class Descriptor: def __init__(self, name=None): self.name = name def __get__(self, instance, cls): if instance is None: return self else: return instance.__dict__[self.name] def __set__(self, instance, value): instance.__dict__[self.name] = value def __delete__(self, instance): del instance.__dict__[self.name] name of attribute being stored. A key in the instance dict.

Slide 113

Slide 113 text

Copyright (C) 2013, http://www.dabeaz.com A Basic Descriptor 113 class Descriptor: def __init__(self, name=None): self.name = name def __get__(self, instance, cls): if instance is None: return self else: return instance.__dict__[self.name] def __set__(self, instance, value): instance.__dict__[self.name] = value def __delete__(self, instance): del instance.__dict__[self.name] Direct manipulation of the instance dict.

Slide 114

Slide 114 text

Copyright (C) 2013, http://www.dabeaz.com A Simpler Descriptor 114 class Descriptor: def __init__(self, name=None): self.name = name def __set__(self, instance, value): instance.__dict__[self.name] = value def __delete__(self, instance): raise AttributeError("Can't delete") • You don't need __get__() if it merely returns the normal dictionary value

Slide 115

Slide 115 text

Copyright (C) 2013, http://www.dabeaz.com Descriptor Usage 115 class Stock(Structure): _fields = ['name', 'shares', 'price'] name = Descriptor('name') shares = Descriptor('shares') price = Descriptor('price') • If it works, will capture set/delete operations >>> s = Stock('ACME', 50, 91.1) >>> s.shares 50 >>> s.shares = 50 # shares.__set__(s, 50) >>> del s.shares Traceback (most recent call last): ... AttributeError: Can't delete >>>

Slide 116

Slide 116 text

Copyright (C) 2013, http://www.dabeaz.com Type Checking 116 class Typed(Descriptor): ty = object def __set__(self, instance, value): if not isinstance(value, self.ty): raise TypeError('Expected %s' % self.ty) super().__set__(instance, value) • Specialization class Integer(Typed): ty = int class Float(Typed): ty = float class String(Typed): ty = str

Slide 117

Slide 117 text

Copyright (C) 2013, http://www.dabeaz.com Usage 117 • Example: >>> s = Stock('ACME', 50, 91.1) >>> s.shares = 'a lot' Traceback (most recent call last): ... TypeError: Expected >>> s.name = 42 Traceback (most recent call last): ... TypeError: Expected >>> class Stock(Structure): _fields = ['name', 'shares', 'price'] name = String('name') shares = Integer('shares') price = Float('price')

Slide 118

Slide 118 text

Copyright (C) 2013, http://www.dabeaz.com Value Checking 118 class Positive(Descriptor): def __set__(self, instance, value): if value < 0: raise ValueError('Expected >= 0') super().__set__(instance, value) • Use as a mixin class class PosInteger(Integer, Positive): pass class PosFloat(Float, Positive): pass

Slide 119

Slide 119 text

Copyright (C) 2013, http://www.dabeaz.com Usage 119 • Example: >>> s = Stock('ACME', 50, 91.1) >>> s.shares = -10 Traceback (most recent call last): ... ValueError: Expected >= 0 >>> s.shares = 'a lot' Traceback (most recent call last): ... TypeError: Expected >>> class Stock(Structure): _fields = ['name', 'shares', 'price'] name = String('name') shares = PosInteger('shares') price = PosFloat('price')

Slide 120

Slide 120 text

Copyright (C) 2013, http://www.dabeaz.com Building Blocks! 120 class PosInteger(Integer, Positive): pass super()

Slide 121

Slide 121 text

Copyright (C) 2013, http://www.dabeaz.com Understanding the MRO 121 class PosInteger(Integer, Positive): pass >>> PosInteger.__mro__ (, , , , , ) >>> This chain defines the order in which the value is checked by different __set__() methods • Base order matters (e.g., int before < 0)

Slide 122

Slide 122 text

Copyright (C) 2013, http://www.dabeaz.com Length Checking 122 class Sized(Descriptor): def __init__(self, *args, maxlen, **kwargs): self.maxlen = maxlen super().__init__(*args, **kwargs) def __set__(self, instance, value): if len(value) > self.maxlen: raise ValueError('Too big') super().__set__(instance, value) • Use: class SizedString(String, Sized): pass

Slide 123

Slide 123 text

Copyright (C) 2013, http://www.dabeaz.com Usage 123 • Example: >>> s = Stock('ACME', 50, 91.1) >>> s.name = 'ABRACADABRA' Traceback (most recent call last): ... ValueError: Too big >>> class Stock(Structure): _fields = ['name', 'shares', 'price'] name = SizedString('name', maxlen=8) shares = PosInteger('shares') price = PosFloat('price')

Slide 124

Slide 124 text

Copyright (C) 2013, http://www.dabeaz.com Pattern Checking 124 import re class Regex(Descriptor): def __init__(self, *args, pat, **kwargs): self.pat = re.compile(pat) super().__init__(*args, **kwargs) def __set__(self, instance, value): if not self.pat.match(value): raise ValueError('Invalid string') super().__set__(instance, value) • Use: class SizedRegexString(String, Sized, Regex): pass

Slide 125

Slide 125 text

Copyright (C) 2013, http://www.dabeaz.com Usage 125 • Example: >>> s = Stock('ACME', 50, 91.1) >>> s.name = 'Head Explodes!' Traceback (most recent call last): ... ValueError: Invalid string >>> class Stock(Structure): _fields = ['name', 'shares', 'price'] name = SizedRegexString('name', maxlen=8, pat='[A-Z]+$') shares = PosInteger('shares') price = PosFloat('price')

Slide 126

Slide 126 text

Copyright (C) 2013, http://www.dabeaz.com Whoa, Whoa, Whoa 126 • Mixin classes with __init__() functions? class SizedRegexString(String, Sized, Regex): pass • Each with own unique signature a = String('name') b = Sized(maxlen=8) c = Regex(pat='[A-Z]+$') • This works, how?

Slide 127

Slide 127 text

Copyright (C) 2013, http://www.dabeaz.com Keyword-only Args 127 class Descriptor: def __init__(self, name=None): ... class Sized(Descriptor): def __init__(self, *args, maxlen, **kwargs): ... super().__init__(*args, **kwargs) class Regex(Descriptor): def __init__(self, *args, pat, **kwargs): ... super().__init__(*args, **kwargs) SizedRegexString('name', maxlen=8, pat='[A-Z]+$')

Slide 128

Slide 128 text

Copyright (C) 2013, http://www.dabeaz.com Keyword-only Args 128 class Descriptor: def __init__(self, name=None): ... class Sized(Descriptor): def __init__(self, *args, maxlen, **kwargs): ... super().__init__(*args, **kwargs) class Regex(Descriptor): def __init__(self, *args, pat, **kwargs): ... super().__init__(*args, **kwargs) SizedRegexString('name', maxlen=8, pat='[A-Z]+$') Keyword-only argument is isolated and removed from all other passed args

Slide 129

Slide 129 text

Copyright (C) 2013, http://www.dabeaz.com 129 "Awesome, man!"

Slide 130

Slide 130 text

Copyright (C) 2013, http://www.dabeaz.com Annoyance 130 • Still quite a bit of repetition • Signatures and type checking not unified • Maybe we can push it further class Stock(Structure): _fields = ['name', 'shares', 'price'] name = SizedRegexString('name', maxlen=8, pat='[A-Z]+$') shares = PosInteger('shares') price = PosFloat('price')

Slide 131

Slide 131 text

Copyright (C) 2013, http://www.dabeaz.com 131

Slide 132

Slide 132 text

Copyright (C) 2013, http://www.dabeaz.com A New Metaclass 132 from collections import OrderedDict class StructMeta(type): @classmethod def __prepare__(cls, name, bases): return OrderedDict() def __new__(cls, name, bases, clsdict): fields = [ key for key, val in clsdict.items() if isinstance(val, Descriptor) ] for name in fields: clsdict[name].name = name clsobj = super().__new__(cls, name, bases, dict(clsdict)) sig = make_signature(fields) setattr(clsobj, '__signature__', sig) return clsobj

Slide 133

Slide 133 text

Copyright (C) 2013, http://www.dabeaz.com New Usage 133 • Oh, that's rather nice... class Stock(Structure): name = SizedRegexString(maxlen=8,pat='[A-Z]+$') shares = PosInteger() price = PosFloat()

Slide 134

Slide 134 text

Copyright (C) 2013, http://www.dabeaz.com New Metaclass 134 from collections import OrderedDict class StructMeta(type): @classmethod def __prepare__(cls, name, bases): return OrderedDict() def __new__(cls, name, bases, clsdict): fields = [ key for key, val in clsdict.items() if isinstance(val, Descriptor) ] for name in fields: clsdict[name].name = name clsobj = super().__new__(cls, name, bases, dict(clsdict)) sig = make_signature(fields) setattr(clsobj, '__signature__', sig) return clsobj __prepare__() creates and returns dictionary to use for execution of the class body. An OrderedDict will preserve the definition order.

Slide 135

Slide 135 text

Copyright (C) 2013, http://www.dabeaz.com Ordering of Definitions 135 class Stock(Structure): name = SizedRegexString(maxlen=8,pat='[A-Z]+$') shares = PosInteger() price = PosFloat() clsdict = OrderedDict( ('name', ), ('shares', ), ('price', ) )

Slide 136

Slide 136 text

Copyright (C) 2013, http://www.dabeaz.com Duplicate Definitions 136 class NoDupOrderedDict(OrderedDict): def __setitem__(self, key, value): if key in self: raise NameError('%s already defined' % key) super().__setitem__(key, value) • If inclined, you could do even better • Make a new kind of dict • Use in place of OrderedDict

Slide 137

Slide 137 text

Copyright (C) 2013, http://www.dabeaz.com Duplicate Definitions 137 class Stock(Structure): name = String() shares = PosInteger() price = PosFloat() shares = PosInteger() Traceback (most recent call last): File "", line 1, in File "", line 5, in Stock File "./typestruct.py", line 107, in __setitem__ raise NameError('%s already defined' % key) NameError: shares already defined • Won't pursue further, but you get the idea

Slide 138

Slide 138 text

Copyright (C) 2013, http://www.dabeaz.com New Metaclass 138 from collections import OrderedDict class StructMeta(type): @classmethod def __prepare__(cls, name, bases): return OrderedDict() def __new__(cls, name, bases, clsdict): fields = [ key for key, val in clsdict.items() if isinstance(val, Descriptor) ] for name in fields: clsdict[name].name = name clsobj = super().__new__(cls, name, bases, dict(clsdict)) sig = make_signature(fields) setattr(clsobj, '__signature__', sig) return clsobj Collect Descriptors and set their names

Slide 139

Slide 139 text

Copyright (C) 2013, http://www.dabeaz.com Name Setting 139 class Stock(Structure): _fields = ['name', 'shares', 'price'] name = SizedRegexString('name', ...) shares = PosInteger('shares') price = PosFloat('price') • Old code • New Code class Stock(Structure): name = SizedRegexString(...) shares = PosInteger() price = PosFloat() Names are set from dict keys

Slide 140

Slide 140 text

Copyright (C) 2013, http://www.dabeaz.com New Metaclass 140 from collections import OrderedDict class StructMeta(type): @classmethod def __prepare__(cls, name, bases): return OrderedDict() def __new__(cls, name, bases, clsdict): fields = [ key for key, val in clsdict.items() if isinstance(val, Descriptor) ] for name in fields: clsdict[name].name = name clsobj = super().__new__(cls, name, bases, dict(clsdict)) sig = make_signature(fields) setattr(clsobj, '__signature__', sig) return clsobj Make the class and signature exactly as before.

Slide 141

Slide 141 text

Copyright (C) 2013, http://www.dabeaz.com New Metaclass 141 from collections import OrderedDict class StructMeta(type): @classmethod def __prepare__(cls, name, bases): return OrderedDict() def __new__(cls, name, bases, clsdict): fields = [ key for key, val in clsdict.items() if isinstance(val, Descriptor) ] for name in fields: clsdict[name].name = name clsobj = super().__new__(cls, name, bases, dict(clsdict)) sig = make_signature(fields) setattr(clsobj, '__signature__', sig) return clsobj A technicality: Must create a proper dict for class contents

Slide 142

Slide 142 text

Copyright (C) 2013, http://www.dabeaz.com 142 Performance

Slide 143

Slide 143 text

Copyright (C) 2013, http://www.dabeaz.com The Costs 143 • Option 1 : Simple class Stock(Structure): name = SizedRegexString(...) shares = PosInteger() price = PosFloat() class Stock: def __init__(self, name, shares, price): self.name = name self.shares = shares self.price = price • Option 2 : Meta

Slide 144

Slide 144 text

Copyright (C) 2013, http://www.dabeaz.com A Few Tests 144 • Instance creation s = Stock('ACME', 50, 91.1) • Attribute lookup s.price • Attribute assignment s.price = 10.0 • Attribute assignment s.name = 'ACME' 1.07s 91.8s 0.08s 0.08s 0.11s 3.40s 0.14s 8.14s Simple Meta (86x) (31x) (58x)

Slide 145

Slide 145 text

Copyright (C) 2013, http://www.dabeaz.com A Few Tests 145 • Instance creation s = Stock('ACME', 50, 91.1) • Attribute lookup s.price • Attribute assignment s.price = 10.0 • Attribute assignment s.name = 'ACME' Simple Meta A bright spot 1.07s 91.8s 0.08s 0.08s 0.11s 3.40s 0.14s 8.14s (86x) (31x) (58x)

Slide 146

Slide 146 text

Copyright (C) 2013, http://www.dabeaz.com Thoughts 146 • Several large bottlenecks • Signature enforcement • Multiple inheritance/super in descriptors • Can anything be done without a total rewrite?

Slide 147

Slide 147 text

Copyright (C) 2013, http://www.dabeaz.com Code Generation 147 def _make_init(fields): code = 'def __init__(self, %s):\n' % \ ', '.join(fields) for name in fields: code += ' self.%s = %s\n' % (name, name) return code • Example: >>> code = _make_init(['name','shares','price']) >>> print(code) def __init__(self, name, shares, price): self.name = name self.shares = shares self.price = price >>>

Slide 148

Slide 148 text

Copyright (C) 2013, http://www.dabeaz.com Code Generation 148 class StructMeta(type): ... def __new__(cls, name, bases, clsdict): fields = [ key for key, val in clsdict.items() if isinstance(val, Descriptor) ] for name in fields: clsdict[name].name = name if fields: exec(_make_init(fields),globals(),clsdict) clsobj = super().__new__(cls, name, bases, dict(clsdict)) setattr(clsobj, '_fields', fields) return clsobj

Slide 149

Slide 149 text

Copyright (C) 2013, http://www.dabeaz.com Code Generation 149 class StructMeta(type): ... def __new__(cls, name, bases, clsdict): fields = [ key for key, val in clsdict.items() if isinstance(val, Descriptor) ] for name in fields: clsdict[name].name = name if fields: exec(_make_init(fields),globals(),clsdict) clsobj = super().__new__(cls, name, bases, dict(clsdict)) setattr(clsobj, '_fields', fields) return clsobj No signature, but set _fields for code that wants it

Slide 150

Slide 150 text

Copyright (C) 2013, http://www.dabeaz.com New Code 150 class Structure(metaclass=StructMeta): pass class Stock(Structure): name = SizedRegexString(...) shares = PosInteger() price = PosFloat() Instance creation: Simple 1.1s Old Meta (w/signatures) 91.8s New Meta (w/exec) 17. 6s

Slide 151

Slide 151 text

Copyright (C) 2013, http://www.dabeaz.com New Thought 151 class Descriptor: ... def __set__(self, instance, value): instance.__dict__[self.name] = value class Typed(Descriptor): def __set__(self, instance, value): if not isinstance(value, self.ty): raise TypeError('Expected %s' % self.ty) super().__set__(instance, value) class Positive(Descriptor): def __set__(self, instance, value): if value < 0: raise ValueError('Expected >= 0') super().__set__(instance, value) Could you merge this code together?

Slide 152

Slide 152 text

Copyright (C) 2013, http://www.dabeaz.com Reformulation 152 class Descriptor(metaclass=DescriptorMeta): def __init__(self, name=None): self.name = name @staticmethod def set_code(): return [ 'instance.__dict__[self.name] = value' ] def __delete__(self, instance): raise AttributeError("Can't delete") • Change __set__ to a method that returns source • Introduce a new metaclass (later)

Slide 153

Slide 153 text

Copyright (C) 2013, http://www.dabeaz.com Reformulation 153 class Typed(Descriptor): ty = object @staticmethod def set_code(): return [ 'if not isinstance(value, self.ty):', ' raise TypeError("Expected %s"%self.ty)' ] class Positive(Descriptor): @staticmethod def set_code(self): return [ 'if value < 0:', ' raise ValueError("Expected >= 0")' ]

Slide 154

Slide 154 text

Copyright (C) 2013, http://www.dabeaz.com Reformulation 154 class Sized(Descriptor): def __init__(self, *args, maxlen, **kwargs): self.maxlen = maxlen super().__init__(*args, **kwargs) @staticmethod def set_code(): return [ 'if len(value) > self.maxlen:', ' raise ValueError("Too big")' ]

Slide 155

Slide 155 text

Copyright (C) 2013, http://www.dabeaz.com Reformulation 155 import re class RegexPattern(Descriptor): def __init__(self, *args, pat, **kwargs): self.pat = re.compile(pat) super().__init__(*args, **kwargs) @staticmethod def set_code(): return [ 'if not self.pat.match(value):', ' raise ValueError("Invalid string")' ]

Slide 156

Slide 156 text

Copyright (C) 2013, http://www.dabeaz.com Generating a Setter 156 def _make_setter(dcls): code = 'def __set__(self, instance, value):\n' for d in dcls.__mro__: if 'set_code' in d.__dict__: for line in d.set_code(): code += ' ' + line + '\n' return code • Takes a descriptor class as input • Walks its MRO and collects output of set_code() • Concatenate to make a __set__() method

Slide 157

Slide 157 text

Copyright (C) 2013, http://www.dabeaz.com Example Setters 157 >>> print(_make_setter(Descriptor)) def __set__(self, instance, value): instance.__dict__[self.name] = value >>> print(_make_setter(PosInteger)) def __set__(self, instance, value): if not isinstance(value, self.ty): raise TypeError("Expected %s" % self.ty) if value < 0: raise ValueError("Expected >= 0") instance.__dict__[self.name] = value >>>

Slide 158

Slide 158 text

Copyright (C) 2013, http://www.dabeaz.com Descriptor Metaclass 158 class DescriptorMeta(type): def __init__(self, clsname, bases, clsdict): if '__set__' not in clsdict: code = _make_setter(self) exec(code, globals(), clsdict) setattr(self, '__set__', clsdict['__set__']) else: raise TypeError('Define set_code()') class Descriptor(metaclass=DescriptorMeta): ... • For each Descriptor class, create setter code • exec() and drop result onto created class

Slide 159

Slide 159 text

Copyright (C) 2013, http://www.dabeaz.com Just to be Clear 159 class Stock(Structure): name = SizedRegexString(...) shares = PosInteger() price = PosFloat() • User has no idea about this code generation • They're just using the same code as before • It's an implementation detail of descriptors

Slide 160

Slide 160 text

Copyright (C) 2013, http://www.dabeaz.com New Performance 160 • Instance creation s = Stock('ACME',50,91.1) • Attribute lookup s.price • Attribute assignment s.price = 10.0 • Attribute assignment s.name = 'ACME' 1.07s 91.8s 7.19s 0.08s 0.08s 0.08s 0.11s 3.40s 1.11s 0.14s 8.14s 2.95s Simple Meta (86x) (6.7x) (31x) (10x) (58x) (21x) Exec

Slide 161

Slide 161 text

Copyright (C) 2013, http://www.dabeaz.com 161 The Horror! The Horror! @alex_gaynor

Slide 162

Slide 162 text

Copyright (C) 2013, http://www.dabeaz.com Remaining Problem 162 class Stock(Structure): name = SizedRegexString(maxlen=8, pat='[A-Z]+$') shares = PosInteger() price = PosFloat() class Point(Structure): x = Integer() y = Integer() class Address(Structure): hostname = String() port = Integer() • Convincing a manager about all of this

Slide 163

Slide 163 text

Copyright (C) 2013, http://www.dabeaz.com Solution: XML 163 name shares price x y hostname port

Slide 164

Slide 164 text

Copyright (C) 2013, http://www.dabeaz.com Solution: XML 164 name shares price x y hostname port +5 extra credit Regex + XML

Slide 165

Slide 165 text

Copyright (C) 2013, http://www.dabeaz.com XML to Classes 165 from xml.etree.ElementTree import parse def _xml_to_code(filename): doc = parse(filename) code = 'import typestruct as _ts\n' for st in doc.findall('structure'): code += _xml_struct_code(st) return code • XML Parsing • Continued...

Slide 166

Slide 166 text

Copyright (C) 2013, http://www.dabeaz.com XML to Classes 166 def _xml_struct_code(st): stname = st.get('name') code = 'class %s(_ts.Structure):\n' % stname for field in st.findall('field'): name = field.text.strip() dtype = '_ts.' + field.get('type') kwargs = ', '.join('%s=%s' % (key, val) for key, val in field.items() if key != 'type') code += ' %s = %s(%s)\n' % \ (name, dtype, kwargs) return code

Slide 167

Slide 167 text

Copyright (C) 2013, http://www.dabeaz.com Example 167 >>> code = _xml_to_code('data.xml') >>> print(code) import typestruct as _ts class Stock(_ts.Structure): name = _ts.SizedRegexString(maxlen=8, pat='[A-Z]+ shares = _ts.PosInteger() price = _ts.PosFloat() class Point(_ts.Structure): x = _ts.Integer() y = _ts.Integer() class Address(_ts.Structure): hostname = _ts.String() port = _ts.Integer() >>>

Slide 168

Slide 168 text

Copyright (C) 2013, http://www.dabeaz.com $$!!@!&!**!!! 168 • Now WHAT!?!? • Allow structure .xml files to be imported • Using the import statement • Yes!

Slide 169

Slide 169 text

Copyright (C) 2013, http://www.dabeaz.com Import Hooks 169 >>> import sys >>> sys.meta_path [, , ] >>> • sys.meta_path • A collection of importer/finder instances

Slide 170

Slide 170 text

Copyright (C) 2013, http://www.dabeaz.com An Experiment 170 class MyImporter: def find_module(self, fullname, path=None): print('*** Looking for', fullname) return None >>> sys.meta_path.append(MyImporter()) >>> import foo *** Looking for foo Traceback (most recent call last): File "", line 1, in ImportError: No module named 'foo' >>> • Yes, you've plugged into the import statement

Slide 171

Slide 171 text

Copyright (C) 2013, http://www.dabeaz.com Structure Importer 171 class StructImporter: def __init__(self, path): self._path = path def find_module(self, fullname, path=None): name = fullname.rpartition('.')[-1] if path is None: path = self._path for dn in path: filename = os.path.join(dn, name+'.xml') if os.path.exists(filename): return StructXmlLoader(filename) return None

Slide 172

Slide 172 text

Copyright (C) 2013, http://www.dabeaz.com Structure Importer 172 class StructImporter: def __init__(self, path): self._path = path def find_module(self, fullname, path=None): name = fullname.rpartition('.')[-1] if path is None: path = self._path for dn in path: filename = os.path.join(dn, name+'.xml') if os.path.exists(filename): return StructXmlLoader(filename) return None Fully qualified module name Package path (if any)

Slide 173

Slide 173 text

Copyright (C) 2013, http://www.dabeaz.com Structure Importer 173 class StructImporter: def __init__(self, path): self._path = path def find_module(self, fullname, path=None): name = fullname.rpartition('.')[-1] if path is None: path = self._path for dn in path: filename = os.path.join(dn, name+'.xml') if os.path.exists(filename): return StructXmlLoader(filename) return None Walk path, check for existence of .xml file and return a loader

Slide 174

Slide 174 text

Copyright (C) 2013, http://www.dabeaz.com XML Module Loader 174 import imp class StructXMLLoader: def __init__(self, filename): self._filename = filename def load_module(self, fullname): mod = sys.modules.setdefault(fullname, imp.new_module(fullname)) mod.__file__ = self._filename mod.__loader__ = self code = _xml_to_code(self._filename) exec(code, mod.__dict__, mod.__dict__) return mod

Slide 175

Slide 175 text

Copyright (C) 2013, http://www.dabeaz.com XML Module Loader 175 import imp class StructXMLLoader: def __init__(self, filename): self._filename = filename def load_module(self, fullname): mod = sys.modules.setdefault(fullname, imp.new_module(fullname)) mod.__file__ = self._filename mod.__loader__ = self code = _xml_to_code(self._filename) exec(code, mod.__dict__, mod.__dict__) return mod Create a new module and put in sys.modules

Slide 176

Slide 176 text

Copyright (C) 2013, http://www.dabeaz.com XML Module Loader 176 import imp class StructXMLLoader: def __init__(self, filename): self._filename = filename def load_module(self, fullname): mod = sys.modules.setdefault(fullname, imp.new_module(fullname)) mod.__file__ = self._filename mod.__loader__ = self code = _xml_to_code(self._filename) exec(code, mod.__dict__, mod.__dict__) return mod Convert XML to code and exec() resulting source

Slide 177

Slide 177 text

Copyright (C) 2013, http://www.dabeaz.com Installation and Use 177 • Add to sys.meta_path def install_importer(path=sys.path): sys.meta_path.append(StructImporter(path)) install_importer() • From this point, structure .xml files will import >>> import datadefs >>> s = datadefs.Stock('ACME', 50, 91.1) >>> s.name 'ACME' >>> datadefs >>>

Slide 178

Slide 178 text

Copyright (C) 2013, http://www.dabeaz.com Look at the Source 178 >>> datadefs >>> >>> import inspect >>> print(inspect.getsource(datadefs)) name shares price ...

Slide 179

Slide 179 text

Copyright (C) 2013, http://www.dabeaz.com 179 Final Thoughts (probably best to start packing up)

Slide 180

Slide 180 text

Copyright (C) 2013, http://www.dabeaz.com Extreme Power 180 class Stock(Structure): name = SizedRegexString(maxlen=8, pat='[A-Z]+$') shares = PosInteger() price = PosFloat() • Think about all of the neat things we did • Descriptors as building blocks • Hiding of annoying details (signatures, etc.) • Dynamic code generation • Even customizing import

Slide 181

Slide 181 text

Copyright (C) 2013, http://www.dabeaz.com Hack or by Design? 181 • Python 3 is designed to do this sort of stuff • More advanced metaclasses (e.g., __prepare__) • Signatures • Import hooks • Keyword-only args • Observe: I didn't do any mind-twisting "hacks" to work around a language limitation.

Slide 182

Slide 182 text

Copyright (C) 2013, http://www.dabeaz.com Python 3 FTW! 182 • Python 3 makes a lot of little things easier • Example : Python 2 keyword-only args def __init__(self, *args, **kwargs): self.maxlen = kwargs.pop('maxlen') ... • In Python 3 def __init__(self, *args, maxlen, **kwargs): self.maxlen = maxlen ... • There are a lot of little things like this

Slide 183

Slide 183 text

Copyright (C) 2013, http://www.dabeaz.com Just the Start 183 • We've only scratched the surface • Function annotations def add(x:int, y:int) -> int: return x + y • Non-local variables def outer(): x = 0 def inner(): nonlocal x x = newvalue ...

Slide 184

Slide 184 text

Copyright (C) 2013, http://www.dabeaz.com Just the Start 184 • Context managers with m: ... • Frame-hacks import sys f = sys._getframe(1) • Parsing/AST-manipulation import ast

Slide 185

Slide 185 text

Copyright (C) 2013, http://www.dabeaz.com You Can, But Should You? 185 • Metaprogramming is not for "normal" coding • Frameworks/libraries are a different story • If using a framework, you may be using this features without knowing it • You can do a lot of cool stuff • OTOH: Keeping it simple is not a bad strategy

Slide 186

Slide 186 text

Copyright (C) 2013, http://www.dabeaz.com That is All! 186 • Thanks for listening • Hope you learned a few new things • Buy the "Python Cookbook, 3rd Ed." (O'Reilly) • Twitter: @dabeaz