8 Things that happen at the dot

Transcript

1. © 2019 Bloomberg Finance L.P. All rights reserved. 8 things that happen at the dot: Attribute access & descriptors PyCon 2019 May 4, 2019 Andy Fundinger Senior Software Engineer
2. © 2019 Bloomberg Finance L.P. All rights reserved. Who am I?  Python developer since v2.4 o Descriptors have not changed since Python 2.2, though we’re using Python 3 today  Worked in industrial automation, metaverse development, and financial industry  Previously spoken about descriptors in talks at PyCaribbean 2016, PyGotham 2017, EuroPython, PyCascades, and PyTennesssee  Currently working at Bloomberg in the Data License group o We deliver financial data to clients o We use Flask, Celery, pycsvw, Redis, Jena, Hadoop, Perl, JavaScript, C/C++, Fortran o We are enterprise scale, but extremely agile
3. © 2019 Bloomberg Finance L.P. All rights reserved. Bloomberg at a glance Bloomberg Terminal Trading Solutions Tradebook Enterprise Data & Content News Media Bloomberg Law Bloomberg New Energy Finance Bloomberg Government
4. © 2019 Bloomberg Finance L.P. All rights reserved. Bloomberg technology by the numbers • 5,000+ software engineers • 150+ technologists and data scientists devoted to machine learning • One of the largest private networks in the world • 120 billion pieces of data from the financial markets each day, with a peak of more than 10 million messages/second • 2 million news stories ingested / published each day (500+ news stories ingested/second) • News content ingested from 125K+ sources • Over 1 billion messages and Instant Bloomberg (IB) chats handled daily
5. © 2019 Bloomberg Finance L.P. All rights reserved. . Quietly sitting between an instance(o) and an attribute(x) it serves as syntactic sugar in place of getattr(o,'x'). In the simplest case this reduces further to o.__dict__['x'] {'x': 3} #1. self.__dict__[key] class MyObject: pass o = MyObject() o.__dict__['x'] = 3 o.__dict__ o.x 3
6. © 2019 Bloomberg Finance L.P. All rights reserved. .. It could be a class attribute 4 #2. self.__class__.__dict__[key] class BaseObject: pass class MyObject(BaseObject): pass o = MyObject() MyObject.y = 4 o.__class__.__dict__ o.y mappingproxy({'__doc__': None, '__module__': '__main__', 'y': 4})
7. … or from a parent class 4 #3. self.__class__{BaseClasses}__dict__[key] o = MyObject() BaseObject.t = 4 o.__class__.__dict__ o.t o.__class__.__bases__[0].__dict__ mappingproxy({'__dict__': <attribute '__dict__' of 'BaseObject' objects>, '__doc__': None, '__module__': '__main__', '__weakref__': <attribute '__weakref__' of 'BaseObject' objects>, 't': 4}) mappingproxy({'__doc__': None, '__module__': '__main__', 'y': 4})
8. …. This is great, but we know Python and there's way more we can do. This is old and boring stuff, but look at how powerful it is! called __getattr__ print p before set __getattr__(r) called class Probe: def __getattr__(self,attr): ret = "__getattr__(%s)"%attr print('called __getattr__') if attr=='r': return ret raise AttributeError #4. self.__class__.__getattr__(self,attr) p = Probe() print("print p before set",p.r,"called") p.r = 3 print("print p after set",p.r,"access is direct") print p after set 3 access is direct
9. © 2019 Bloomberg Finance L.P. All rights reserved. Calling __setattr__() On the left side, we have __setattr__ {} class Probe: def __setattr__(self,attr,value): print('called __setattr__') #self.__class__.__setattr__(self,attr,value) p = Probe() p.t = 3 p.__dict__ p.__dict__ called __setattr
10. ….. __getattribute__() actually calls __getattr__() __getattribute__(s) print p after set Anyvar class Probe: def __getattribute__(self, attr): print('__getattribute__(%s)'%attr) return 'Anyvar' #5. self.__class__.__getattribute__(self,attr) p = Probe() print("print p before set",p.s) p.s = "hello" print("print p after set",p.s) __getattribute__(s) print p before set Anyvar
11. …... Actually much more, particularly it looks at what’s found in the class.__dict__ __get__(<__main__.MyObject object at 0x7f74a84ff470>,<class '__main__.MyObject'>) class ProbeND: def __get__(self, obj, type=None): ret = '__get__(%s,%s)'%(obj,type) return ret #6. self.__class__.__dict__[attr].__get__(self,cls) o = MyObject() MyObject.z = ProbeND() # adding to __class__.__dict__ o.z o.__dict__['z'] = None o.z None
12. ……. If we add a __set__() method and do the same, the behavior changes a little __get__(<__main__.MyObject object at 0x7f74a852d550>,<class '__main__.MyObject'>) class ProbeDD(ProbeND): def __set__(self, obj, value): print('__set__(%s,%s)'%(obj,value)) #7. self.__class__.__dict__[attr].__set__(self,cls) MyObject.z = ProbeDD() o = MyObject() o.__dict__['z'] = None o.z o.__dict__ {'z': None}
13. Calling __set__() __set__(<__main__.MyObject object at 0x7f74a852d550>,Something) class ProbeDD(ProbeND): def __set__(self, obj, value): print('__set__(%s,%s)'%(obj,value)) #self.__class__.__dict__[attr].__set__(self,cls) MyObject.z = ProbeDD() o = MyObject() o.__dict__['z'] = None o.z = 'Something' o.z o.__dict__ __get__(<__main__.MyObject object at 0x7f74a852d550>,<class '__main__.MyObject'> {'z': None}
14. © 2019 Bloomberg Finance L.P. All rights reserved. The 8 things 1. self.__dict__[key] 2. self.__class__.__dict__[key] 3. self.__class__{BaseClasses}__dict__[key] 4. self.__class__.__getattr__(self,attr) 5. self.__class__.__getattribute__(self,attr) 6. self.__class__.__dict__[attr].__get__(self,cls) 7. self.__class__.__dict__[attr].__set__(self,cls) 8. raise AttributeError Descriptor Protocol
15. © 2019 Bloomberg Finance L.P. All rights reserved. The Descriptor Protocol Initially included with Python 2.2 “New Style Classes” class Descriptor(object): def __get__(self, obj, objtype): # obj is None when called on the class return None def __set__(self, obj, val): pass def __delete__(self, obj): pass # Python 3.6 def __set_name__(self, owner, name): # owner is class, name is attr name self.name = name Data Descriptor Non-Data Descriptor
16. Method binding This is the only way instance methods work at all method class Greeter(): def greeting(self): print('Hi') Greeter.__dict__['greeting'].__class__ Greeter.__dict__['greeting'].__get__ function bound = Greeter().greeting bound.__class__ <method-wrapper '__get__' of function object at 0x7f74a850e598> Hi bound()
17. © 2019 Bloomberg Finance L.P. All rights reserved. What else requires or uses Descriptors? • Variant method bindings —staticmethods —classmethods —properties • Framework tricks —SqlAlchemy —flask.config • Lazy Execution • Proxying • Monitoring • Runtime type checking
18. Example: Alias Two names for one attribute ('Hi', 'Hi', 'PyCon', True) class Alias(object): def __init__(self, alias_to): super(Alias, self).__init__() self.alias_to = alias_to def __get__(self, instance, owner): if instance is None: return self return getattr(instance, self.alias_to) def __set__(self, instance, value): setattr(instance, self.alias_to, value) class DataClass: data_x = Alias('legacy_x') data_y = Alias('legacy_y') data_z = Alias('legacy_z') def __init__(self,x,y,z): self.legacy_x = x self.legacy_y = y self.legacy_z = z dc = DataClass("Hi","PyCon",2019) dc.data_x, dc.legacy_x, dc.legacy_y, dc.data_z is dc.legacy_z ('Hello', 'Hello', 'PyCon', True) dc.data_x = 'Hello' dc.data_x, dc.legacy_x, dc.data_y, dc.data_z is dc.legacy_z
19. Example: defaultedmethod These are both class and instance methods, which create an instance as needed. from functools import partial class defaultedmethod(object): def __init__(self, fn): self.fn = fn def __get__(self, obj, cls=None): if obj is None: return partial(self.fn, cls()) return self.fn.__get__(obj,cls) class NamePrinter: def __init__(self, name='Default Name'): self.name = name def name_print(self): print(self.name) @defaultedmethod def print_name(self): print(self.name) Andy Andy andy = NamePrinter('Andy') andy.print_name() andy.name_print() Default Name TypeError: name_print() missing 1 required positional argument: 'self NamePrinter.print_name() NamePrinter.name_print()
20. © 2019 Bloomberg Finance L.P. All rights reserved. Which should I use and when? Any metaprogramming technique is a shovel – a technique to get you out of a hole which is best held back until you are sure the benefit is worth the added complexity Technique Example Usage Normal Attribute Access Data Storage __getattr__ Caching, Lazy Execution __getattribute__ Proxy access to an unknown, remote, API Non Data Descriptors (__get__) Method Binding Data Descriptors (__get__ & __set__) Runtime typing
21. © 2019 Bloomberg Finance L.P. All rights reserved. Questions? Andy Fundinger AFundinger1@bloomberg.net Slides on SpeakerDeck https://speakerdeck.com/ciemaar