You don't know decorators

A talk about the common problems about implementing your own decorators and the solutions in Python. Some examples on how decorators are used in the community and finally a few statistics about the use of them.

Transcript

1. You Don’t Know Decorators June 12, 2019 Zoli Szeredi engineer@syneto @zoliszeredi
2. Outline • Motivation • Problems and Solutions • Implementing Custom Decorators • Functions Objects • Real World Examples • Stats
3. PEP 318: Why do we need decorators? • In Python 2.2 there was staticmethod, classmethod but no syntactic support for generic function transforms • Patching the function after the definition is suboptimal in readability • Syntactic brittleness • Structure information in simple and intuitive way • Delayed by syntax bikeshedding • Python 2.4 implemented PEP 318 on November 30, 2004 • PEP-3129 (class decorators) backported to 2.7 from 3.X
4. Back in the old days... class Data(object): def __init__(self, **options): self.options = options # alternate constructor def from_json(self, json_string): instance = self() instance.options = json.loads(json_string) return instance from_json = classmethod(from_json) >>> x = Data.from_json(‘{“a”: 42}’) >>> x.options {‘a’: 42}
5. Duplication, verbose… many not too happy class Data(object): def __init__(self, **options): self.options = options # alternate constructor def from_json(self, json_string): instance = self() instance.options = json.loads(json_string) return instance from_json = classmethod(from_json) # same for staticmethod and property :( >>> x = Data.from_json(‘{“a”: 42}’) >>> x.options {‘a’: 42}
6. A more elegant syntax for a more civilized age class Data(object): def __init__(self, **options): self.options = options @classmethod def from_json(cls, json_string): instance = cls() instance.options = json.loads(json_string) return instance
7. Built-in ones • property • classmethod • staticmethod • *.setter • unittest.mock.patch • dataclass # lacking in 3.6; not great, or terrible • functools.wraps • functools.lru_cache • asyncio.coroutine
8. @ decorator? • not originating from the Decorator pattern from the Design Patterns by GoF • Comes from the notion of Decorated (or annotated) Parse Tree from compilers terminology: the process of evaluating attributes is called annotation, or DECORATION, of the parse tree
9. How are decorators used? Profiling and Probing • Tracers • Loggers • Timers • NewRelic • Metadata Middleware • Authentication • Throttling • Processing • Separation of concern • Error Handling • Monkey Patching • Testing Application Config • Flask app.route('/') • Hook atexit.register • Signals • Registries • Conditional Exec
10. What are good use-cases @newrelic.agent.wsgi_application() def application(environ, start_response): pass @etag @login_required def my_view(request): """Django based view""" @dataclass class InventoryItem: '''Class for keeping track of an item in inventory.''' name: str unit_price: float quantity_on_hand: int = 0 @app.route('/') def hello_world(): return 'Hello, World!' @pytest.mark.parametrize( ‘x’, [0, None, 1, 42, 667] ) def test_code(x): “Trust by verify.”
11. Syntax @wrapper def function(): pass # Is similar to... def function(): pass function = wrapper(function)
12. @decorator def target(arg, option=None): “The body of the function” • Decorator: the callable that takes a function as an argument • Target: The base function and its definition • Resulting function (invisible) Components
13. Example def fib(n): "Computes the fibonacci sequence." return fib(n-1) + fib(n-2) if n > 2 else n >>> fib(10) 89 >>> fib(38) # ... long pause 63245986 %timeit fib(38) 4.83 s ± 17.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
14. Tracing for more info def trace(target): def wrapper(*args, **kwargs): print (args, kwargs) return target(*args, **kwargs) return wrapper @trace def fib(n): "Computes the fibonacci sequence." return fib(n-1) + fib(n-2) if n > 2 else n >>> fib(6) (6,) {} (5,) {} (4,) {} (3,) {} (2,) {} (1,) {} (2,) {} (3,) {} (2,) {} (1,) {} (4,) {} (3,) {} (2,) {} (1,) {} (2,) {}
15. Let’s disassemble with dis.dis @trace def fib(n): return fib(n-1) + fib(n-2) if n > 2 else n 3 8 LOAD_NAME 2 (trace) 10 LOAD_CONST 0 (<code obj>) 12 LOAD_CONST 1 ('fib') 14 MAKE_FUNCTION 0 16 CALL_FUNCTION 1 18 STORE_NAME 3 (fib) 20 LOAD_CONST 2 (None) 22 RETURN_VALUE
16. Let’s disassemble def fib(n): return fib(n-1) + fib(n-2) if n > 2 else n fib = trace(fib) 3 8 LOAD_CONST 0 (<code obj>) 10 LOAD_CONST 1 ('fib') 12 MAKE_FUNCTION 0 14 STORE_NAME 3 (fib) 5 16 LOAD_NAME 2 (trace) 18 LOAD_NAME 3 (fib) 20 CALL_FUNCTION 1 22 STORE_NAME 3 (fib) 24 LOAD_CONST 2 (None) 26 RETURN_VALUE
17. Memoize where possible def trace(target): def wrapper(*args, **kwargs): print (args, kwargs) return target(*args, **kwargs) return wrapper @trace # this is before so we can still trace @functools.lru_cache() def fib(n): "Computes the fibonacci sequence." return fib(n-1) + fib(n-2) if n > 2 else n >>> fib(6) (6,) {} (5,) {} (4,) {} (3,) {} (2,) {} (1,) {} (2,) {} (3,) {} (4,) {} 13 >> fib(6) (6,) {} 13
18. But there are problems... def trace(target): def wrapper(*args, **kwargs): print (args, kwargs) return target(*args, **kwargs) return wrapper @trace @functools.lru_cache() def fib(n): "Computes the fibonacci sequence." return fib(n-1) + fib(n-2) if n > 2 else n In [35]: ?? fib Signature: fib(*args, **kwargs) Docstring: <no docstring> Source: def wrapper(*args, **kwargs): print (args, kwargs) return target(*args, **kwargs) File: ~/<ipython-input-31-ee3828d11e4b> Type: function
19. Some solutions def trace(target): @functools.wraps(target) def wrapper(*args, **kwargs): print (args, kwargs) return target(*args, **kwargs) return wrapper @trace @functools.lru_cache() def fib(n): "Computes the fibonacci sequence." return fib(n-1) + fib(n-2) if n > 2 else n In [39]: ?? fib Signature: fib(n) Source: @trace @functools.lru_cache() def fib(n): "Computes the fibonacci sequence." return fib(n-1) + fib(n-2) if n > 2 else n File: ~/<ipython-input-40-32d58494e625> Type: function
20. functools.wraps • Facade for functools.update_wrapper • Preserves function attributes from functools.WRAPPER_ASSIGNMENTS: ◦ __module__ ◦ __name__ ◦ __qualname__ ◦ __doc__ ◦ __annotations__ • Avoids the laborious manual copying of above attributes • Future proofs your code
21. Decorators with arguments • Example with nested scopes • Example with functools.partial random.seed(42); [random.random() for _ in range(5)] def with_seed(seed): def inner_decorator(target): def surrogate(*args, **kwargs): random.seed(seed) return target(*args, **kwargs) return surrogate return inner_decorator @with_seed(42) def rando(count=5): return [random.random() for _ in range(count)] >>> rando() [0.6394267984578837, 0.025010755222666936, 0.27502931836911926, 0.22321073814882275, 0.7364712141640124] >>> rando() [0.6394267984578837, 0.025010755222666936, 0.27502931836911926, 0.22321073814882275, 0.7364712141640124]
22. What if we decorate the class or methods? @trace class Fib: def __init__(self): self._data = {} @trace def __getitem__(self, lookup): if lookup in {0, 1}: return lookup if self._data.get(lookup, None) is None: result = self[lookup-1] + self[lookup-2] self._data[lookup] = result return self._data[lookup] >>> for num in Fib(): >>> if num > 1: >>> break >>> print (val) () {} (<__main__.Fib object at 0x7f2256c>, 0) {} 0 (<__main__.Fib object at 0x7f2256c>, 1) {} 1 (<__main__.Fib object at 0x7f2256c>, 2) {} (<__main__.Fib object at 0x7f2256c>, 1) {} (<__main__.Fib object at 0x7f2256c>, 0) {} 1 (<__main__.Fib object at 0x7f2256c>, 3) {} (<__main__.Fib object at 0x7f2256c>, 2) {} (<__main__.Fib object at 0x7f2256c>, 1) {}
23. Counting Calls def count_arg(target): "Counts the calls for a given argument." target._calls = collections.defaultdict(int) @functools.wraps(target) def surrogate(n): try: return target(n) finally: target._calls[n] += 1 surrogate._target = target return surrogate @count_arg def fib(n): "Computes the fibonacci sequence." return fib(n-1) + fib(n-2) if n > 2 else n >>> fib(15) # 987 >>> fib._calls defaultdict(int, {2: 377, 1: 233, 3: 233, 4: 144, 5: 89, 6: 55, 7: 34, 8: 21, 9: 13, 10: 8, 11: 5, 12: 3, 13: 2, 14: 1, 15: 1})
24. What if there’s another way? class count_arg2: "Counts the calls for a given argument." def __init__(self, target): self._calls = collections.defaultdict(int) self.target = target functools.update_wrapper(self, target) def __call__(self, n): try: return self.target(n) finally: self._calls[n] += 1 @count_arg2 def fib(n): "Computes the fibonacci sequence." return fib(n-1) + fib(n-2) if n > 2 else n >>> fib(15) # 987 >>> fib._calls defaultdict(int, {2: 377, 1: 233, 3: 233, 4: 144, 5: 89, 6: 55, 7: 34, 8: 21, 9: 13, 10: 8, 11: 5, 12: 3, 13: 2, 14: 1, 15: 1})
25. User defined decorators are not easy • Implementation not universal • Lots of cases to consider: ◦ Class decorator ◦ Static method ◦ Instance Method ◦ Class method ◦ Module function
26. Functions Recap • __name__ __doc__ __module__ __annotations__ • First class citizen • are objects • call signature • Bound and unbound • Descriptor interface __get__ and __set__ • Use of functools.wraps • inspect.getargspec(fun)
27. wrapt package https://github.com/GrahamDumpleton/wrapt Graham Dumpleton: NewRelic, mod_wsgi, import wrapt @wrapt.decorator def pass_through(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs) @pass_through def function(): pass
28. With arguments import wrapt def with_arguments(myarg1, myarg2): @wrapt.decorator def wrapper(wrapped, instance, args, kwargs): return wrapped(*args, **kwargs) return wrapper @with_arguments(1, 2) def function(): pass
29. Universal Decorator import inspect @wrapt.decorator def universal(wrapped, instance, args, kwargs): if instance is None: if inspect.isclass(wrapped): # Decorator was applied to a class. return wrapped(*args, **kwargs) else: return wrapped(*args, **kwargs) # Decorator was applied to a function or staticmethod. else: if inspect.isclass(instance): return wrapped(*args, **kwargs) # Decorator was applied to a classmethod. else: return wrapped(*args, **kwargs) # Decorator applied to an instancemethod.
30. Decorator for propper class methods def on_methods(decorator): @wrapt.decorator def surrogate(target, instance, args, kwargs): for name, attribute in target.__dict__.items(): if callable(attribute): setattr(target, name, decorator(attribute)) return target(*args, **kwargs) return surrogate
31. Apply class method decorator @on_methods(trace) class URL: def __init__(self, address): self.address = address def get(self): return urllib.request.urlopen(self.address) >>> URL('http://feeds.reuters.com/Reuters/worldNews').get() __init__ ('http://feeds.reuters.com/Reuters/worldNews',) {} get () {} <http.client.HTTPResponse at 0x7f45b0ae7c50>
32. Apply decorator on all class methods @on_methods(trace) class Fib: def __init__(self, _check=False): self._data = {} def __getitem__(self, lookup): if isinstance(lookup, slice): return (self[index] for index in range(lookup.start or 0, lookup.stop, lookup.step or 1)) if lookup in {0, 1}: return lookup if self._data.get(lookup, None) is None: result = self[lookup-1] + self[lookup-2] self._data[lookup] = result return self._data[lookup] def on_methods(decorator): @wrapt.decorator def surrogate(target, instance, args, kwargs): items = target.__dict__.items() for name, attribute in items: if callable(attribute): setattr(target, name, decorator(attribute)) return target(*args, **kwargs) return surrogate
33. What it does @on_methods(trace) class Fib: def __init__(self, _check=False): self._data = {} def __getitem__(self, lookup): if isinstance(lookup, slice): return (self[index] for index in range(lookup.start or 0, lookup.stop, lookup.step or 1)) if lookup in {0, 1}: return lookup if self._data.get(lookup, None) is None: self._data[lookup] = self[lookup-1] + self[lookup-2] return self._data[lookup] for x in Fib()[:5:2]: print (x) __init__ () {} __getitem__ (slice(None, 5, 2),) {} __getitem__ (0,) {} 0 __getitem__ (2,) {} __getitem__ (1,) {} __getitem__ (0,) {} 1 __getitem__ (4,) {} __getitem__ (3,) {} __getitem__ (2,) {} __getitem__ (1,) {} __getitem__ (2,) {} 3
34. Testing code with HTTP: VCR.py @vcr.use_cassette('fixtures/vcr_cassettes/synopsis.yaml') def test_iana(): response = urllib2.urlopen('http://www.iana.org/domains/reserved').read() assert 'Example domains' in response • Run this test once, and VCR.py will record the HTTP request. • Run it again, and VCR.py will replay the response
35. Conditional Execution @six.add_metaclass(Meta) class Compatible(object): "Same code in python2 and 3" # this will be equivalent to: class Compatible(object, metaclass=Meta): "In python3" class Compatible(object): "In python2" __metaclass__ = Meta @pytest.mark.xfail(raises=IOError) def test_fetch_feed(): "Verifies the retrieval of the feed object"
36. Stats 1 head -13 django.decorators 976 override_settings 926 setup 438 property 427 classmethod 293 skipUnlessDBFeature 150 cached_property 106 isolate_apps 98 mock.patch 85 register.filter 80 unittest.skipUnless 75 modify_settings 57 register.tag 51 staticmethod head -13 flask.decorators 226 app.route 31 pytest.mark.parametrize 28 bp.route 26 app.errorhandler 25 pytest.fixture 21 setupmethod 14 property 13 app.teardown_request 11 click.option 10 app.before_request 9 staticmethod 7 pytest.mark.skipif 7 app.teardown_appcontext
37. Stats 2 head -13 cpython.decorators 755 unittest.skipUnless 616 classmethod 498 property 409 unittest.skipIf 257 dataclass 199 staticmethod 124 bench 123 bigmemtest 115 mock.patch 110 support.cpython_only 102 abstractmethod 81 contextlib.contextmanager 63 requireAttrs head -13 sqlalchemy.decorators 1297 classmethod 584 property 300 testing.provide_metadata 180 event.listens_for 142 util.memoized_property 111 testing.fails_on 99 declared_attr 83 testing.emits_warning 80 _generative 78 reflection.cache 63 compiles 46 util.dependencies 42 profiling.function_call_count
38. Conclusions • Use decorators when you want to compose behaviour or profile code • Great for post-processing of results: i.e. json-serialize • Increase readability and signal intent • Don’t overuse them (more than 3 decorators) • Essential for deterministic testing • Can limit scope of monkey patching to functions
39. Thank you!