The 8 Things that Happen at the "." Between an Object and the Attribute

Presented at PyGotham 2017

Overview
We rarely think about the dot “.” between our objects and their fields, but there are quite a lot of things that happen every time we use one in Python. This talk will include the commonly known functions, (dict, getattr, etc), but especially focus on the Descriptor Protocol.

Description
We rarely think about the dot “.” between our objects and their fields, but there are quite a lot of things that happen every time we use one in Python. This talk will explore the details of what happens, how the descriptor protocol works, and how it can be used to alter the Python object model.

Actions explored:
* instance dicts
* magic methods
* the descriptor protocol

Examples include altering method binding and property behaviors, as well as supporting method implementation reuse via Descriptors.

Conference site:
https://2017.pygotham.org/talks/the-8-things-that-happen-at-the-between-an-object-and-the-attribute/
Video:
https://youtu.be/Bg_cEIkOS9g
PyVideo.org:
http://pyvideo.org/pygotham-2017/the-8-things-that-happen-at-the-between-an-object-and-the-attribute.html

Transcript

1. Attribute access & descriptors 8 things that happen at the dot: PyGotham 2017 October 7, 2017 Andy Fundinger Senior Software Engineer © 2017 Bloomberg Finance L.P. All rights reserved.
2. Who am I? • New Father • Python Developer since v2.4 o This talk is all Python 3, but it's not something that has changed • Previously approached the use of Descriptors for typing in a talk at PyCaribbean 2016 • Worked in industrial automation, metaverse development, and financial industry • Currently at Bloomberg working in the Data License group o We deliver financial information to clients o We use Flask, Celery, pycsvw, Jena, RIOT, PHP, Perl, JavaScript, C/C++ o We are enterprise scale, but extremely agile
3. © 2017 Bloomberg Finance L.P. All rights reserved. Bloomberg at a glance • Bloomberg Terminal • Trading Solutions • Tradebook • Enterprise Data & Content • News • Media • Bloomberg Law/BNA • Bloomberg New Energy Finance • Bloomberg Government
4. © 2017 Bloomberg Finance L.P. All rights reserved. Bloomberg technology by the numbers • 5,000+ software engineers • 150+ technologists and data scientists devoted to machine learning • One of the largest private networks in the world • 100 billion tick messages per day, with a peak of more than 10 million messages/second • 1.5 million news stories ingested / published each day (that's 500 news stories ingested/second) • News content from 125K+ sources • Over 400 million messages and 17 million IB chats exchanged daily
5. © 2017 Bloomberg Finance L.P. All rights reserved. Consider our friend the humble . Quietly sitting between an instance(o) and an attribute(x) it serves as syntactic sugar in place of getattr(o,'x'). In the simplest case this reduces further to o.__dict__['x'] {'x': 3} #1. self.__dict__[key] class MyObject: pass o = MyObject() o.__dict__['x'] = 3 o.__dict__ o.x 3
6. © 2017 Bloomberg Finance L.P. All rights reserved. Of course it could be a class attribute 4 #2. self.__class__.__dict__[key] class BaseObject: pass class MyObject(BaseObject): pass o = MyObject() MyObject.y = 4 o.__class__.__dict__ o.y mappingproxy({'__doc__': None, '__module__': '__main__', 'y': 4})
7. © 2017 Bloomberg Finance L.P. All rights reserved. Or from a parent class 4 #3. self.__class__{BaseClasses}__dict__[key] o = MyObject() BaseObject.t = 4 o.__class__.__dict o.t o.__class__.__bases__[0].__dict mappingproxy({'__dict__': <attribute '__dict__' of 'BaseObject' objects>, '__doc__': None, '__module__': '__main__', '__weakref__': <attribute '__weakref__' of 'BaseObject' objects>, 't': 4}) mappingproxy({'__doc__': None, '__module__': '__main__', 'y': 4})
8. © 2017 Bloomberg Finance L.P. All rights reserved. Basic overrides This is great but we know Python and there's way more we can do, this is old and boring stuff, but look at how powerful it is! called __getattr__ print p before set __getattr__(r) called class Probe: def __getattr__(self,attr): ret = "__getattr__(%s)"%attr print('called __getattr__') if attr=='r': return ret raise AttributeError #4. self.__class__.__getattr__(self,attr) p = Probe() print("print p before set",p.r,"called") p.r = 3 print("print p after set",p.r,"access is direct") print p after set 3 access is direct
9. © 2017 Bloomberg Finance L.P. All rights reserved. On the left side we have __setattr__ {} class Probe: def __setattr__(self,attr,value): print('called __setattr__') #5. self.__class__.__setattr__(self,attr,value) p = Probe() p.t = 3 p.__dict__ p.__dict__ called __setattr
10. © 2017 Bloomberg Finance L.P. All rights reserved. __getattribute__() is farther reaching __getattribute__() actually calls __getattr__() __getattribute__(s) print p after set Anyvar class Probe: def __getattribute__(self, attr): print('__getattribute__(%s)'%attr) return 'Anyvar' #6. self.__class__.__getattribute__(self,attr) p = Probe() print("print p before set",p.s) p.s = "hello" print("print p after set",p.s) __getattribute__(s) print p before set Anyvar
11. © 2017 Bloomberg Finance L.P. All rights reserved. object.__getattribute__ does more Actually much more, particularly it looks at what’s found in the class.__dict__ __get__(<__main__.MyObject object at 0x7f74a84ff470>,<class '__main__.MyObject'>) class ProbeND: def __get__(self, obj, type=None): ret = '__get__(%s,%s)'%(obj,type) return ret #7. self.__class__.__dict__[attr].__get__(self,cls) o = MyObject() MyObject.z = ProbeND() # adding to __class__.__dict__ o.z o.__dict__['z'] = None o.z None
12. © 2017 Bloomberg Finance L.P. All rights reserved. ... and it checks what it finds If we add a __set__() method and do the same the behavior changes a little __get__(<__main__.MyObject object at 0x7f74a852d550>,<class '__main__.MyObject'>) class ProbeDD(ProbeND): def __set__(self, obj, value): print('__set__(%s,%s)'%(obj,value)) #8. self.__class__.__dict__[attr].__set__(self,cls) MyObject.z = ProbeDD() o = MyObject() o.__dict__['z'] = None o.z o.__dict__ {'z': None}
13. © 2017 Bloomberg Finance L.P. All rights reserved. __set__() is called as expected If we add a __set__() method and do the same the behavior changes a little __set__(<__main__.MyObject object at 0x7f74a852d550>,Something) class ProbeDD(ProbeND): def __set__(self, obj, value): print('__set__(%s,%s)'%(obj,value)) #8. self.__class__.__dict__[attr].__set__(self,cls) MyObject.z = ProbeDD() o = MyObject() o.__dict__['z'] = None o.z = 'Something' o.z o.__dict__ __get__(<__main__.MyObject object at 0x7f74a852d550>,<class '__main__.MyObject'> {'z': None}
14. © 2017 Bloomberg Finance L.P. All rights reserved. The 8 things 1. self.__dict__[key] 2. self.__class__.__dict__[key] 3. self.__class__{BaseClasses}__dict__[key] 4. self.__class__.__getattr__(self,attr) 5. self.__class__.__setattr__(self,attr) 6. self.__class__.__getattribute__(self,attr) 7. self.__class__.__dict__[attr].__get__(self,cls) 8. self.__class__.__dict__[attr].__set__(self,cls) Descriptor Protocol
15. © 2017 Bloomberg Finance L.P. All rights reserved. The Descriptor Protocol Initially included with Python 2.2 “New Style Classes” class Descriptor(object): def __get__(self, obj, objtype): # obj is None when called on the class return None def __set__(self, obj, val): pass def __delete__(self, obj): pass # Python 3.6 def __set_name__(self, owner, name): # owner is class, name is attr name self.name = name Data Descriptor Non-Data Descriptor
16. © 2017 Bloomberg Finance L.P. All rights reserved. Method binding This is the only way instance methods work at all. method class Greeter(): def greeting(self): print('Hi') Greeter.__dict__['greeting'].__class__ Greeter.__dict__['greeting'].__get__ function bound = Greeter().greeting bound.__class__ <method-wrapper '__get__' of function object at 0x7f74a850e598> Hi bound()
17. © 2017 Bloomberg Finance L.P. All rights reserved. What else? • Variant method bindings o staticmethods o classmethods o properties • Framework tricks o SqlAlchemy o flask.config • Lazy Execution • Proxying • Monitoring • Runtime type checking
18. © 2017 Bloomberg Finance L.P. All rights reserved. Example: Alias Two names for one attribute. ('Hi', 'Hi', 'PyGotham', True) class Alias(object): def __init__(self, alias_to): super(Alias, self).__init__() self.alias_to = alias_to def __get__(self, instance, owner): if instance is None: return self return getattr(instance, self.alias_to) def __set__(self, instance, value): setattr(instance, self.alias_to, value) class DataClass: data_x = Alias('legacy_x') data_y = Alias('legacy_y') data_z = Alias('legacy_z') def __init__(self,x,y,z): self.legacy_x = x self.legacy_y = y self.legacy_z = z dc = DataClass("Hi","PyGotham",2017) dc.data_x, dc.legacy_x, dc.legacy_y, dc.data_z is dc.legacy_z ('Hello', 'Hello', 'PyGotham', True) dc.data_x = 'Hello' dc.data_x, dc.legacy_x, dc.data_y, dc.data_z is dc.legacy_z
19. © 2017 Bloomberg Finance L.P. All rights reserved. Example: defaultedmethod These are both class and instance methods, which create an instance as needed. from functools import partial class defaultedmethod(object): def __init__(self, fn): self.fn = fn def __get__(self, obj, cls=None): if obj is None: return partial(self.fn, cls()) return self.fn.__get__(obj,cls) class NamePrinter: def __init__(self, name='Default Name'): self.name = name def name_print(self): print(self.name) @defaultedmethod def print_name(self): print(self.name) Andy Andy andy = NamePrinter('Andy') andy.print_name() andy.name_print() Default Name TypeError: name_print() missing 1 required positional argument: 'self NamePrinter.print_name() NamePrinter.name_print()
20. © 2017 Bloomberg Finance L.P. All rights reserved. Questions? Technique Example Usage Normal Attribute Access Data Storage __getattr__ Caching, Lazy Execution __getattribute__ Proxy access to an unknown, remote, API Non Data Descriptors (__get__) Method Binding Data Descriptors (__get__ & __set__) Runtime typing Which should I use and when? Andy Fundinger AFundinger1@bloomberg.net