Share
Embed
Python descriptors in detail
In this PyCon ES 2015 talk we will review the descriptors protocol, use cases and direct application in Python's standard library; properties and slots.
Pablo E
November 21, 2015
Transcript
45 slides
-
{ PYTHON DESCRIPTORS PROTOCOL IN DETAIL PYCON ES 2015 PABLO
ENFEDAQUE VIDAL @pablitoev56 -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || In this talk we will review the descriptors protocol, use cases and direct application in Python'ʹs standard library; properties and slots Welcome! -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Python descriptors protocol -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs implement a descriptor from random import randint class RandAttr: # Hello! I'm a descriptor def __init__(self, n=10): self.n = n def __get__(self, inst, cls): print("Called __get__") return randint(0, self.n) -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs use the descriptor from random import randint class RandAttr: # Hello! I'm a descriptor def __init__(self, n=10): self.n = n def __get__(self, inst, cls): print("Called __get__") return randint(0, self.n) class RandAttrClass(object): rand_attr = RandAttr(20) # Descriptors are class attributes inst = RandAttrClass() # Let's instantiate and access it print(inst.rand_attr) Called __get__ 12 print(inst.rand_attr) Called __get__ 5 -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs use the descriptor from random import randint class RandAttr: # Hello! I'm a descriptor def __init__(self, n=10): self.n = n def __get__(self, inst, cls): print("Called __get__") return randint(0, self.n) class RandAttrClass(object): rand_attr = RandAttr(20) # Descriptors are class attributes inst = RandAttrClass() # Let's instantiate and access it print(inst.rand_attr) Called __get__ 12 print(inst.rand_attr) Called __get__ 5 -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || A descriptor is an object attribute with binding behaviour, one whose attribute access has been overridden by methods in the descriptor protocol Python descriptors -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || A descriptor is an object attribute with binding behaviour, one whose attribute access has been overridden by methods in the descriptor protocol It may intercept attribute retrieval, modification and/or deletion Python descriptors -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Descriptors protocol requires implementing at least one of: Introduced in Python 2.2 Python descriptors protocol descr.__get__(self, obj, type=None) --> value descr.__set__(self, obj, value) --> None descr.__delete__(self, obj) --> None -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Descriptors protocol __get__ -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs understand __get__ class VerboseGetDescriptor: def __get__(self, inst, cls): print("Called __get__ on", self.__class__) print("\tself:", self) print("\tinst:", inst) print("\tcls:", cls) class VerboseGetClass: attr = VerboseGetDescriptor() -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs understand __get__ class VerboseGetDescriptor: def __get__(self, inst, cls): print("Called __get__ on", self.__class__) print("\tself:", self) print("\tinst:", inst) print("\tcls:", cls) class VerboseGetClass: attr = VerboseGetDescriptor() inst = VerboseGetClass() print(inst.attr) Called __get__ on <class VerboseGetDescriptor> self: <VerboseGetDescriptor object at 0x10e62f310> inst: <VerboseGetClass object at 0x10e6cb990> cls: <class VerboseGetClass'> None -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs understand __get__ class VerboseGetDescriptor: def __get__(self, inst, cls): print("Called __get__ on", self.__class__) print("\tself:", self) print("\tinst:", inst) print("\tcls:", cls) class VerboseGetClass: attr = VerboseGetDescriptor() inst = VerboseGetClass() print(inst.attr) Called __get__ on <class VerboseGetDescriptor> self: <VerboseGetDescriptor object at 0x10e62f310> inst: <VerboseGetClass object at 0x10e6cb990> cls: <class VerboseGetClass'> None -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || __get__ with a class attribute class VerboseGetDescriptor: def __get__(self, inst, cls): print("Called __get__ on", self.__class__) print("\tself:", self) print("\tinst:", inst) print("\tcls:", cls) class VerboseGetClass: attr = VerboseGetDescriptor() inst = VerboseGetClass() print(VerboseGetClass.attr) # Now with the class attribute Called __get__ on <class VerboseGetDescriptor> self: <VerboseGetDescriptor object at 0x10e62f310> inst: None cls: <class 'VerboseGetClass'> None -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || __get__ with a class attribute class VerboseGetDescriptor: def __get__(self, inst, cls): print("Called __get__ on", self.__class__) print("\tself:", self) print("\tinst:", inst) print("\tcls:", cls) class VerboseGetClass: attr = VerboseGetDescriptor() inst = VerboseGetClass() print(VerboseGetClass.attr) # Now with the class attribute Called __get__ on <class VerboseGetDescriptor> self: <VerboseGetDescriptor object at 0x10e62f310> inst: None cls: <class 'VerboseGetClass'> None -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || __get__ in real life class LazyAttr: def __init__(self, name, func): self.name = name self.func = func def __get__(self, inst, cls): print("Called __get__") # Let's add the result to the INSTANCE inst.__dict__[self.name] = self.func(inst) return inst.__dict__[self.name] from time import sleep class LazyAttrUser: def expensive_func(self): print("\tCalled expensive func") sleep(9999) return 12345 # We give the same name than the descriptor attr = LazyAttr ('attr', expensive_func) -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || __get__ in real life class LazyAttr: def __init__(self, name, func): self.name = name self.func = func def __get__(self, inst, cls): print("Called __get__") # Let's add the result to the INSTANCE inst.__dict__[self.name] = self.func(inst) return inst.__dict__[self.name] from time import sleep class LazyAttrUser: def expensive_func(self): print("\tCalled expensive func") sleep(9999) return 12345 # We give the same name than the descriptor attr = LazyAttr ('attr', expensive_func) # Let's instantiate the class inst = LazyAttrUser() print(inst.__dict__) {} # Let's retrieve it print(inst.attr) Called __get__ Called expensive func 12345 # Let's do it again print(inst.attr) 12345 print(inst.__dict__) {'attr': 12345} -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || __get__ in real life class LazyAttr: def __init__(self, name, func): self.name = name self.func = func def __get__(self, inst, cls): print("Called __get__") # Let's add the result to the INSTANCE inst.__dict__[self.name] = self.func(inst) return inst.__dict__[self.name] from time import sleep class LazyAttrUser: def expensive_func(self): print("\tCalled expensive func") sleep(9999) return 12345 # We give the same name than the descriptor attr = LazyAttr ('attr', expensive_func) # Let's instantiate the class inst = LazyAttrUser() print(inst.__dict__) {} # Let's retrieve it print(inst.attr) Called __get__ Called expensive func 12345 # Let's do it again print(inst.attr) 12345 print(inst.__dict__) {'attr': 12345} -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Descriptors protocol __set__ -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs understand __set__ class VerboseSetDescriptor: def __set__(self, inst, val): print("Called __get__ on", self.__class__) print("\tself:", self) print("\tinst:", inst) print("\tval:", val) # New value instead of the class class VerboseSetClass: attr = VerboseSetDescriptor() -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs understand __set__ class VerboseSetDescriptor: def __set__(self, inst, val): print("Called __get__ on", self.__class__) print("\tself:", self) print("\tinst:", inst) print("\tval:", val) # New value instead of the class class VerboseSetClass: attr = VerboseSetDescriptor() inst = VerboseSetClass() inst.attr = 12345 Called __get__ on <class VerboseSetDescriptor> self: <VerboseSetDescriptor object at 0x10e6cb490> inst: <VerboseSetClass object at 0x10e6cb5d0> val: 12345 print(inst.__dict__) # NO modification done {} -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs understand __set__ class VerboseSetDescriptor: def __set__(self, inst, val): print("Called __get__ on", self.__class__) print("\tself:", self) print("\tinst:", inst) print("\tval:", val) # New value instead of the class class VerboseSetClass: attr = VerboseSetDescriptor() inst = VerboseSetClass() inst.attr = 12345 Called __get__ on <class VerboseSetDescriptor> self: <VerboseSetDescriptor object at 0x10e6cb490> inst: <VerboseSetClass object at 0x10e6cb5d0> val: 12345 print(inst.__dict__) # NO modification done {} -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs understand __set__ class VerboseSetDescriptor: def __set__(self, inst, val): print("Called __get__ on", self.__class__) print("\tself:", self) print("\tinst:", inst) print("\tval:", val) # New value instead of the class class VerboseSetClass: attr = VerboseSetDescriptor() inst = VerboseSetClass() print(VerboseSetClass.__dict__) {<VerboseSetDescriptor object at 0x10e6cb490>, ...} VerboseSetClass.attr = 12345 print(inst.__dict__) print(VerboseSetClass.__dict__) # We "replaced" it {} {'attr': 12345, ...} -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || __set__ in real life class TypedAttrib: def __init__(self, name, typ): self.name = name self.typ = typ # Avoid using 'type' as attribute name! def __set__(self, inst, val): if isinstance(val, self.typ): inst.__dict__[self.name] = val else: raise TypeError("Wrong type {0}".format(type(val))) class MyTypedClass: string_attr = TypedAttrib('string', str) integer_attr = TypedAttrib('integer', int) inst = MyTypedClass() inst.string_attr = "123" print(inst.__dict__) {'string_attr': '123'} print(inst.string_attr) 123 -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || __set__ in real life class TypedAttrib: def __init__(self, name, typ): self.name = name self.typ = typ # Avoid using 'type' as attribute name! def __set__(self, inst, val): if isinstance(val, self.typ): inst.__dict__[self.name] = val else: raise TypeError("Wrong type {0}".format(type(val))) class MyTypedClass: string_attr = TypedAttrib('string', str) integer_attr = TypedAttrib('integer', int) inst = MyTypedClass() try: inst.string_attr = 123 except TypeError, e: print(e) Wrong type <class 'int'> -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Don'ʹt do this… duck typing class TypedAttrib: def __init__(self, name, typ): self.name = name self.typ = typ # Avoid using 'type' as attribute name! def __set__(self, inst, val): if isinstance(val, self.typ): inst.__dict__[self.name] = val else: raise TypeError("Wrong type {0}".format(type(val))) class MyTypedClass: string_attr = TypedAttrib('string', str) integer_attr = TypedAttrib('integer', int) inst = MyTypedClass() try: inst.string_attr = 123 except TypeError, e: print(e) Wrong type <class 'int'> -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Descriptors protocol __delete__ -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || No time today -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Descriptors in standard library -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Python slots -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Welcome to slots class SlottedClass: __slots__ = ['attr_x', 'attr_y'] # These are slots def __init__(self, x, y): self.attr_x = x self.attr_y = y def sum_content(self): return self.attr_x + self.attr_y inst = SlottedClass(12345, 67890) print(inst.attr_x) 12345 print(inst.attr_y) # We can access its attributes 67890 inst.attr_x = 54321 # We can modify its attributes print(inst.sum_content()) # We can call its methods 122211 -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || However… class SlottedClass: __slots__ = ['attr_x', 'attr_y'] # These are slots def __init__(self, x, y): self.attr_x = x self.attr_y = y def sum_content(self): return self.attr_x + self.attr_y inst = SlottedClass(12345, 67890) inst.attr_xyz = 54321 Traceback (most recent call last): File "<console>", line 1, in <module> AttributeError: 'SlottedClass' object has no attribute 'attr_xyz' print(inst.__dict__) Traceback (most recent call last): File "<console>", line 1, in <module> AttributeError: 'SlottedClass' object has no attribute '__dict__' -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || class SlottedClass: __slots__ = ['attr_x', 'attr_y'] # These are slots def __init__(self, x, y): self.attr_x = x self.attr_y = y def sum_content(self): return self.attr_x + self.attr_y inst = SlottedClass(12345, 67890) print(SlottedClass.__dict__) {'attr_x': <member 'attr_x' of 'SlottedClass' objects>, 'attr_y': <member 'attr_y' of 'SlottedClass' objects>, '__slots__': ['attr_x', 'attr_y']...} print(SlottedClass.__dict__['attr_x'].__get__(inst, SlottedClass)) 54321 # Slots are descriptors! SlottedClass.__dict__['attr_x'].__set__(inst, 100001) print(inst.attr_x) 100001 -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || • Light control of objects structure (can be bypassed!) • Faster access • Smaller instances memory footprint • Improve efficiency (storage and speed) • Really useful to handle huge amounts of objects • Slots are implemented with descriptors • Don'ʹt forget there is no instance __dict__ when using slots • There are other restrictions. Check the docs! Python slots -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Python properties -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || First, some useful methods def f_to_c(f): # Fahrenheit to Celsius return (f - 32.0) * 5.0 / 9.0 def c_to_f(c): # Celsius to Fahrenheit return c * 9.0 / 5.0 + 32.0 def k_to_c(k): # Kelvin to Celsius return k - 273.15 def c_to_k(c): # Celsius to Kelvin return c + 273.15 -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Welcome to properties class TempClass: def __init__(self, temp=None): self._temp = temp or 0.0 def get_temp_c (self): return self._temp def set_temp_c (self, val): self._temp = val # This is a property! temp_celsius = property(get_temp_c, set_temp_c) def get_temp_f (self): return c_to_f(self._temp) def set_temp_f (self, val): self._temp = f_to_c(val) temp_fahrenheit = property(get_temp_f, set_temp_f) def get_temp_k(self): return c_to_k(self._temp) temp_kelvin = property(get_temp_k) # No setter! -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Welcome to properties class TempClass: def __init__(self, temp=None): self._temp = temp or 0.0 def get_temp_c (self): return self._temp def set_temp_c (self, val): self._temp = val # This is a property! temp_celsius = property(get_temp_c, set_temp_c) def get_temp_f (self): return c_to_f(self._temp) def set_temp_f (self, val): self._temp = f_to_c(val) temp_fahrenheit = property(get_temp_f, set_temp_f) def get_temp_k(self): return c_to_k(self._temp) temp_kelvin = property(get_temp_k) # No setter! -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Let'ʹs use these properties class TempClass: def __init__(self, temp=None): self._temp = temp or 0.0 def get_temp_c (self): return self._temp def set_temp_c (self, val): self._temp = val # This is a property! temp_celsius = property(get_temp_c, set_temp_c) def get_temp_f (self): return c_to_f(self._temp) def set_temp_f (self, val): self._temp = f_to_c(val) temp_fahrenheit = property(get_temp_f, set_temp_f) def get_temp_k(self): return c_to_k(self._temp) temp_kelvin = property(get_temp_k) # No setter! inst = TempClass(20) print(inst.temp_fahrenheit) 68.0 print(inst.temp_kelvin) 293.15 inst.temp_fahrenheit = 32 print(inst.temp_celsius) 0.0 inst.temp_kelvin = 200 # No setter Traceback (most recent call last): File "<console>", line 1, in <module> AttributeError: can't set attribute -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || print(inst.__dict__) {'_temp': 0.0} print(TempClass.__dict__) {'get_temp_celsius': <function get_temp_celsius at 0x10e6cdcf8>, 'set_temp_celsius': <function set_temp_celsius at 0x10e6cd758>, 'set_temp_fahrenheit': <function set_temp_fahrenheit at 0x10e6cdde8>, 'get_temp_fahrenheit': <function get_temp_fahrenheit at 0x10e6cdc08>, 'temp_celsius': <property object at 0x10e6d8470>, 'temp_fahrenheit': <property object at 0x10e6d84c8>, 'temp_kelvin': <property object at 0x10e6d8520>, 'get_temp_kelvin': <function get_temp_kelvin at 0x10e6cde60>...} print(TempClass.__dict__['temp_fahrenheit']) <property object at 0x10e6d84c8> print(TempClass.__dict__['temp_fahrenheit'].__get__(inst, TempClass)) # This a descriptor! Yay! 32.0 TempClass.__dict__['temp_celsius'].__set__(inst, 100) print(inst.__dict__) {'_temp': 100} -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || There is a @property decorator class TempClass: def __init__(self, temp=None): self._temp = temp or 0.0 # This is a property @property def temp_celsius(self): " Temperature in Celsius " return self._temp # This is the setter for tmp_celsius @temp_celsius.setter def temp_celsius_set(self, val): self._temp = val # temp_kelvin does not have setter @property def temp_kelvin(self): " Temperature in Kelvin " return c_to_k(self._temp) -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || There is a @property decorator class TempClass: def __init__(self, temp=None): self._temp = temp or 0.0 # This is a property @property def temp_celsius(self): " Temperature in Celsius " return self._temp # This is the setter for tmp_celsius @temp_celsius.setter def temp_celsius_set(self, val): self._temp = val # temp_kelvin does not have setter @property def temp_kelvin(self): " Temperature in Kelvin " return c_to_k(self._temp) -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || • Properties are a specialized implementation of descriptors • Define getters, setters, deleters and even docstrings used instead of direct attribute access • Alternative declaration with decorators • Use them to keep backwards compatibility with direct attribute access • Use them when you need knowledge of class internals • Use them for more generic logic or less coupled solutions Python properties -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || Q&A Thanks for coming! Slides: https://speakerdeck.com/pablito56/python-‐‑ descriptors-‐‑in-‐‑detail -
|| PYTHON DESCRIPTORS PROTOCOL IN DETAIL || PYCON ES 2015
|| PABLO ENFEDAQUE VIDAL || @pablitoev56 || • https://docs.python.org/3/reference/datamodel.html • https://docs.python.org/3/howto/descriptor.html • http://docs.python.org/3/reference/datamodel.html#slots • http://docs.python.org/3/library/functions.html#property References