Induction in metaclasses

A head first guide to metaclasses in Python. The presentation will show you how metaclasses work, how they affect attribute resolution and few potential usecases for them. At the end you'll have a pretty good idea of how Django models and other metaclass (ab)using libraries work.

HTML5 version: http://blog.ionelmc.ro/presentations/metaclase/
Presented at: http://conference.ropython.org/

Transcript

1. Induction in metaclasses
3. Induction in metaclasses ME TA CL ASS Ionel Cristian Mărieș — Python / OSS enthusiast blog.ionelmc.ro github.com/ionelmc
5. Why? Lots of unwarranted fear ... ʺitʹs magicʺ, ʺthey are badʺ etc They present lots of interesting opportunities for: Reducing boilerplate. Nicer APIs Especially useful in designs that rely on subclassing (subclasses will use the same metaclass). They have a tax: Introspection issues, unhappy linters. Need to understand them to use effectively. Because some interactions are not explicit.
7. Few basic things Objects allow overriding certain object interfaces and operations Like the function call operator: >>> class Funky: ... def __call__(self): ... print("Look at me, I work like a function!") >>> f = Funky() >>> f() Look at me, I work like a function! Or a ribute access: >>> class MagicHat: ... def __getattr__(self, name): ... return "Imagine a %a ..." % name >>> hat = MagicHat() >>> hat.rabbit "Imagine a 'rabbit' ..."
9. But there are some caveats Magic methods are looked up (resolved) on the class, not on the instance. [1] >>> class Simple: ... pass >>> s = Simple() Doesnʹt work: >>> s.__call__ = lambda self: "x" >>> s() Traceback (most recent call last): ... TypeError: 'Simple' object is not callable Aha! It is resolved on the class: >>> Simple.__call__ = lambda self: "x" >>> s() 'x' [1] Exceptions: Python 2 old‑style objects and few special‑cased methods.
11. Constructors and initialisers Instance creation is just a magic method you can override: >>> class WithConstructor(object): ... def __new__(cls, value): ... print("Creating the instance") ... return super().__new__(cls) ... ... def __init__(self, value): ... print("Initialising the instance") ... self.value = value >>> WithConstructor(1) Creating the instance Initialising the instance <__main__.WithConstructor object at 0x...> Note that if __new__ returns a different type of object then a different __init__ would be called (from the other typeʹs class).
13. Types and instances A class is just an instance of something else, a metaclass.
15. You can have a custom metaclass So, if a class is just an instance, we can customise the creation process: >>> class Meta(type): ... pass >>> class Complex(metaclass=Meta): ... pass >>> Complex() <__main__.Complex object at 0x...> Normally, the metaclass should be a subclass of type . More on this later.
17. The instantiation dance in detail Remember that: __init__ does not create instances, __new__ does. Magic methods are resolved on the metaclass. Complex() is equivalent to Meta.__call__ .
19. The full interface __prepare__(mcs, name, bases, **kwargs) ‑ New in Python 3, returns the namespace dictionary __new__(mcs, name, bases, attrs, **kwargs) ‑ Returns an instance of Meta __init__(cls, name, bases, attrs, **kwargs) ‑ Runs initialisation code, typeʹs __init__ doesnʹt do anything kwargs: only in Python 3, example: class Class(object, metaclass=Meta, a=1, b=2, c=3): pass __call__ ‑ Returns and instance of Complex (which in turn is instance of Meta). Because magic methods are resolved on the class.
21. Syntactic sugar >>> class Simple(Base): ... foo = 'bar' Is equivalent to: >>> Simple = type('Simple', (Base, ), {'foo': 'bar'}) ~ >>> class Simple(Base, metaclass=Meta): ... foo = 'bar' Is equivalent to: >>> Simple = Meta('Simple', (Base, ), {'foo': 'bar'})
23. Going back to the type of the metaclass Normally you inherit from type because it implements all the necessary interface for a well functioning class. But you can use just as well a plain callable. And let horrible stuff like this happen: >>> class NotAnymore(metaclass=print): ... pass NotAnymore () {'__qualname__': 'NotAnymore', '__module__': '__main__'} >>> repr(NotAnymore) 'None'
25. Why can the metaclass be any callable
27. Why can the metaclass be any callable
29. Useless tricks >>> type(type) is type True But how about: >>> class mutable(type): # dummy type, so the instance has a __dict__ ... pass # (mutable in other words) ... >>> class typeish(type, metaclass=mutable): ... pass ... >>> typeish.__class__ = typeish >>> print(type(typeish) is typeish) True
31. New in Python 3: __prepare__ Allows users to customize the class creation before the body of the class is executed. Basically allows you to return a different object as the namespace (instead of a dict). What can you do with it? Lots of interesting stuff: Single dispatch Duplicate validators Field order aware objects
33. Disallow overrides with __prepare__ >>> class StrictDict(dict): ... def __setitem__(self, name, value): ... if name in self: ... raise RuntimeError('You already defined %r!' % name) ... super().__setitem__(name, value) ... >>> class StrictMeta(type): ... def __prepare__(name, bases): ... return StrictDict() ... >>> class Strict(metaclass=StrictMeta): ... a = 1 ... a = 2 # Ooops. Will ever anyone notice this? Traceback (most recent call last): ... RuntimeError: You already defined 'a'!
35. Single dispatch with __prepare__ ﴾1/3﴿ Suppose we have this code: >>> class Int(int): ... def __repr__(self): ... return "Int(%d)" % self ... ... @dispatching.dispatch ... def __add__(self, other:int): ... return Int(int.__add__(self, other)) ... ... @__add__.dispatch ... def __add__(self, other:str): ... return Int(int.__add__(self, int(other))) >>> i = Int(5) >>> i + 1 Int(6) >>> i + "2" Int(7)
37. Single dispatch with __prepare__ ﴾2/3﴿ We can make it seamless using __prepare__ : >>> class Int(int, metaclass=SingleDispatchMeta): ... def __repr__(self): ... return "Int(%d)" % self ... ... def __add__(self, other:int): ... return Int(int.__add__(self, other)) ... ... def __add__(self, other:str): ... return Int(int.__add__(self, int(other))) >>> i = Int(5) >>> i + 1 Int(6) >>> i + "2" Int(7)
39. Single dispatch with __prepare__ ﴾3/3﴿ >>> import dispatching >>> class SingleDispatchCollector(dict): ... def __setitem__(self, name, value): ... if callable(value): ... if name in self: ... self[name].dispatch(value) ... else: ... super().__setitem__(name, ... dispatching.dispatch(value)) ... else: ... super().__setitem__(name, value) >>> class SingleDispatchMeta(type): ... def __prepare__(name, bases): ... return SingleDispatchCollector()
41. What else can you do with metaclasses Docstring fixers DSLs, Models Class or usage validators Subclass registry systems All sorts of behavior changing Warning/deprecation systems Automatic function decoration
43. Metaclasses vs class decorators A class decorator takes a class as input and, hopefully, returns a class. The decorator usually returns the same class after making some changes to it (eg: monkey‑patching). Disadvantages of a decorator: Lack of flexibility. They are equivalent to a __init__ method in the metaclass. When accessing methods through the class or instance they become bound functions (which is not the same thing as the original function). Solution is to pull them out of __dict__ . It becomes boilerplate. Doesnʹt work well with subclassing.
45. Known uses Pythonʹs abc Django SQLAlchemy fields $ pip install fields >>> from fields import Fields >>> class Pair(Fields.a.b): ... pass ... >>> Pair(1, 2) Pair(a=1, b=2)
47. Where do we draw the line? Metaclasses have some disadvantages. What is a good tradeoff? Whatʹs worse, boilerplate or implicit behavior? Two important things to consider: Is the implicit behavior intuitive? Does the abstraction leak?
49. And last .... Q: What does the metaclass say to the class? A: Youʹre __new__ to me.