Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding Python attributes

4ef16ff17bbfb331c657d584ea0dc135?s=47 reuven
April 30, 2022

Understanding Python attributes

Attributes in Python, which we use dozens of times each day, seem boring, obvious, and not worthy of attention. But it turns out that they're key to the Python language: Every time you say a.b in Python, that little dot is hiding a lot of work, from searching across multiple objects to silently rewriting things. And it turns out that what happens with attributes, while not always obvious to developers, determines a great deal of behavior in the Python language.

In this talk from PyCon US 2022, I discuss what attributes are (and aren't), what Python does when you use a dot (.) in your code, and how you can take advantage of it. I talk about attribute lookup, about inheritance, and about methods vs. functions. I also look into properties, and how they allow us to have attributes that look like data but behave like setters and getters. Finally, I look at the descriptor protocol, which makes so much of Python's functionality possible, including the automatic insertion of "self" as the first argument in method calls.

4ef16ff17bbfb331c657d584ea0dc135?s=128

reuven

April 30, 2022
Tweet

More Decks by reuven

Other Decks in Technology

Transcript

  1. Understanding attributes (Or: They're not nearly as boring as you

    think!) Reuven M. Lerner • PyCon US 2022 reuven@lerner.co.il • @reuvenmlerner
  2. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Corporate • Video • Hybrid • Weekly Python Exercise • Books: • Python Workout • Pandas Workout • Come see my booth in the expo hall for swag + discounts! I teach Python! 2
  3. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes x

    = 100 • This does not mean: Put the value of 100 in the memory location called “x” • This does mean: The name “x” should refer to the integer 100 Let’s assign to a variable! 3
  4. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    MyClass: # do-nothing class pass x = MyClass() # create an instance, assign to x x.y = 100 # assign 100 to x.y • y is not a variable. It’s an attribute. Let’s get fancier 4
  5. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Every object in Python has attributes • They’re basically a private dictionary — but you use a dot (.) rather than [] (square brackets). • The attribute doesn’t exist on the variable, but rather on the object that the variable refers to. Variables vs. attributes 5
  6. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    We retrieve values from attributes quite a bit: • sys.version • str.upper('abc') • random.randint(0, 100) • Just like variables, attributes can contain any Python object — both data and functions. Reading from attributes 6
  7. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    With rare exception, you can set any attribute on any object x.y = 100 >>> sys.version = '4.00' >>> sys.version '4.00' >>> sys.version = '4.20' >>> sys.version '4.20' Setting attributes 7
  8. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    In fact, the __init__ method is meant to set attributes! class Person: def __init__(self, name): self.name = name p1 = Person('name1') p2 = Person('name2') We set attributes all the time 8
  9. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes When

    __init__ is running 9
  10. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes After

    __init__ returns 10
  11. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Person: def __init__(self, name): self.name = name p1 = Person('name1') p2 = Person('name2') • Let’s keep track of the population as we add people! What’s missing from this program? 11
  12. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes population

    = 0 class Person: def __init__(self, name): population += 1 self.name = name def greet(self): return f'Hello, {self.name}!' print(f'Before, {population=}') p1 = Person(‘name1’) p2 = Person(‘name2’) print(f'After, {population=}’) Idea: A global variable 12
  13. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes UnboundLocalError

    13
  14. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes population

    = 0 class Person: def __init__(self, name): population += 1 self.name = name def greet(self): return f'Hello, {self.name}!' print(f'Before, {population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {population=}') The problem? 14
  15. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes population

    = 0 class Person: def __init__(self, name): global population population += 1 self.name = name def greet(self): return f'Hello, {self.name}!' print(f'Before, {population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {population=}') print(p1.greet()) print(p2.greet()) “global” to the rescue! 15
  16. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Everything in Python is an object • Every object has attributes • We can set and retrieve attributes on every object • What if we set an attribute on the Person class? A better alternative 16
  17. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Person: def __init__(self, name): Person.population += 1 self.name = name def greet(self): return f'Hello, {self.name}!' Person.population = 0 print(f'Before, {Person.population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {Person.population=}') print(p1.greet()) print(p2.greet()) Class attribute 17
  18. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    First we’re creating the Person class • Then, after we’re done, we add the “population” attribute • It works, but this can’t be the best way to do it This is pretty ugly 18
  19. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes Quiz

    time! 19
  20. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes print('A')

    class Person: print('B') def __init__(self, name): print('C') self.name = name print('D') print('E') p1 = Person('name1') p2 = Person('name2') What will this code print? 20
  21. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes print('A')

    class Person: print('B') def __init__(self, name): print('C') self.name = name print('D') print('E') p1 = Person('name1') p2 = Person('name2') The answer 21 A B D E C C 1 2 3 4 5
  22. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Function and class de f initions look similar, but they aren’t • A function body doesn’t execute when it’s de f ined • A class body, though, must execute when it’s de f ined class Person: def __init__(self, name): self.name = name p1 = Person('name1') p2 = Person('name2') Why? 22 Before this can happen This has to happen
  23. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    But wait — def normally does two things: • Creates a function object • Assigns the object to a variable, the function’s name • But here, def is de f ining __init__, which is… what? 
 What about de f ? 23
  24. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes Methods

    are class attributes 24
  25. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Functions de f ined in a class body are actually class attributes • Inside of the class, it looks like a variable • Outside of the class, it looks like an attribute on the class • Where have we seen this before? Modules! • When we import a module, any global variables in the module are available as attributes on the module object. • So any functions we de f ine inside a class aren’t variables, but class attributes. • And any assignments we make in a class don’t create variables, but rather class attributes. Classes are f ile-less modules 25
  26. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Person: population = 0 # not a variable definition! def __init__(self, name): Person.population += 1 self.name = name print(f'Before, {Person.population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {Person.population=}') Class attribute, nicer edition 26
  27. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    People who come from other languages sometimes call class attributes “static variables.” • Don’t do this! • Static variables are shared across the class and instances • But a class attribute is a name that exists only on the class • Python doesn’t have the concept of a “shared attribute.” • However, many different attributes (and variables) might refer to the same object. Static variables?!? 27
  28. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Person: population = 0 def __init__(self, name): Person.population += 1 self.name = name print(f'Before, {Person.population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {Person.population=}') print(f'After, {p1.population=}') print(f’After, {p2.population=}’) Let’s see that they’re different 28 2 2 2
  29. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    It worked. • And we got the same values. • Maybe they are actually shared? Uh oh 29
  30. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Nope! • We’ve stumbled onto the attribute lookup rule in Python, which I call ICPO: • I: First Python looks on the instance in the expression. • C: If it doesn’t f ind the attribute on the speci f ied instance, it looks on it’s class. • P: Not on the class? Python looks on the class’s parent • It keeps looking up and up, on each class’s parent • O: The f inal stop is object, the top of Python’s class hierarchy ICPO 30
  31. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Person: population = 0 # not a variable definition! def __init__(self, name): Person.population += 1 self.name = name print(f'Before, {Person.population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {Person.population=}') print(f'After, {p1.population=}') print(f'After, {p2.population=}') Let’s walk through this 31 I: Does Person have the attribute “population”? YES, 2 I: Does p1 have the attribute “population”? NO. C: Does Person have the attribute “population”? YES, 2 I: Does p2 have the attribute “population”? NO. C: Does Person have the attribute “population”? YES, 2
  32. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes ICPO

    explains method lookup 32
  33. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Person: def __init__(self, name): self.name = name def greet(self): return f'Hello, {self.name}!' p1 = Person('name1') p2 = Person('name2') print(p1.greet()) print(p2.greet()) A new version of Person 33 I: Does p1 have the attribute “greet”? NO. C: Does Person have the attribute “greet”? YES! I: Does p2 have the attribute “greet”? NO. C: Does Person have the attribute “greet”? YES!
  34. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Methods are class attributes… • …but we normally call them via the instance • ICPO explains how this can be. ICPO explains method lookup 34
  35. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Employee, which is almost exactly like Person • Employees are created with two attributes, name and id_number • Otherwise, they’re the same as people We need a new class 35
  36. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Person: def __init__(self, name): self.name = name def greet(self): return f'Hello, {self.name}!' p1 = Person('name1') p2 = Person('name2') print(p1.greet()) print(p2.greet()) Person 36
  37. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Employee: def __init__(self, name, id_number): self.name = name self.id_number = id_number def greet(self): return f'Hello, {self.name}!' e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) Let’s de f ine Employee! 37
  38. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes Hello,

    name1 Hello, name2 Hello, emp1 Hello, emp2 Does it work? 38
  39. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Employee(Person): def __init__(self, name, id_number): self.name = name self.id_number = id_number def greet(self): return f'Hello, {self.name}!' e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) Try again, but with inheritance 39 This inserts “Person” into the ICPO attribute search
  40. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    The “greet” method is exactly the same in both Person and Employee • So we don’t need to implement it in Employee! Let’s use inheritance 40
  41. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Employee(Person): def __init__(self, name, id_number): self.name = name self.id_number = id_number e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) Better use of inheritance 41
  42. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Do we need to set self.name in Employee.__init__? • After all, we set it in Person.__init__, right? What about name? 42
  43. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Employee(Person): def __init__(self, name, id_number): self.id_number = id_number e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) What happens now? 43 Assume it’ll be set in Person.__init__
  44. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes AttributeError:

    'Employee' object has no attribute ‘name' • Huh? Let’s check, using “vars”: >>> print(vars(e1)) {'id_number': 1} >>> print(vars(e2)) {'id_number': 2} • The “name” attribute was never set! It doesn’t go well… 44
  45. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    When we created each instance of Employee: • I: We asked the new instance, “Do you have __init__?” No. • C: We asked Employee, “Do you have __init__?” Yes. • Employee.__init__ ran, setting self.id_number • Because Employee.__init__ was found, Person.__init__ never ran • Class de f initions in Python aren’t merged! If Person.__init__ doesn’t run, then the attributes it adds aren’t added • We end up with nameless employees… • …which ends badly when “greet” asks for self.name The problem? 45
  46. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Employee(Person): def __init__(self, name, id_number): Person.__init__(self, name) self.id_number = id_number e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) Solution: Explicitly call Person.__init__ 46 # Hello, emp1! # Hello, emp2!
  47. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    Employee(Person): def __init__(self, name, id_number): super().__init__(name) self.id_number = id_number e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) Better: Use super 47 # Hello, emp1! # Hello, emp2!
  48. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    What happens when I print(p1) or print(e1)? >>> print(p1) <__main__.Person object at 0x10902c5e0> >>> print(e1) <__main__.Employee object at 0x108f14910> Printing our objects 48
  49. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Calling “print” invokes str, which calls the __str__ method. • With p1: • I: Does p1 have __str__? No. • C: Does p1’s class, Person, have __str__? No. • O: Does object have __str__? Yes! • With e1: • I: Does e1 have __str__? No. • C: Does e1’s class, Employee, have __str__? No. • P: Does e1’s class’s parent, Person, have __str__? No. • O: Does object have __str__? Yes! Why so ugly? 49
  50. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Attribute lookup is the reason that operator overloading works • De f ine __str__ in your class, and it is found earlier than “object” Operator overloading! 50
  51. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Python loves to be explicit, and to work without magic. • But there’s a big piece of magic that we all know, and don’t complain about: Method rewriting. >>> s = 'abcd' >>> s.upper() 'ABCD' >>> str.upper(s) 'ABCD' Something is still missing 51
  52. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    We know that methods are stored in class attributes • We know that we can retrieve a class attribute either via the class or via the instance • But somehow, we’re getting different behavior depending on how we retrieve it! Something is weird here 52
  53. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>>

    Person.greet <function __main__.Person.greet(self)> >>> p1.greet <bound method Person.greet of <__main__.Person object at 0x10902c5e0>> And indeed: 53
  54. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    Normally, when I retrieve a class attribute, I get the object that is stored in the attribute. • But if: • The attribute’s class has a __get__ method • Then the result of __get__ is returned instead Descriptors 54
  55. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    LoudNumber: def __init__(self, n): print(f'In LoudNumber.__init__, {n=}') self.n = n def __get__(self, *args): print(f'In LoudNumber.__get__, {self.n=}') return self.n • If I assign an instance of LoudNumber to a variable, nothing special happens. • But if I assign it to a class attribute, magic happens. Example 55
  56. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>>

    class Person: age = LoudNumber(30) In LoudNumber.__init__, n=30 >>> p = Person() Use it in a class attribute 56
  57. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes 57

  58. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>>

    p.age In LoudNumber.__get__, self.n=30 30 >>> Person.age In LoudNumber.__get__, self.n=30 30 Retrieve the value 58
  59. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    I retrieve the attribute • I don’t use parentheses • Despite this, a method has run! Notice 59
  60. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class

    LoudNumber: def __init__(self, n): print(f'In LoudNumber.__init__, {n=}') self.n = n def __get__(self, obj, objtype): print(f'Also: {obj=}, {objtype=}') print(f'In LoudNumber.__get__, {self.n=}') return self.n What are __get__’s parameters? 60 From what instance is this attribute being retrieved? In what class is this attribute stored?
  61. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>>

    class Person: age = LoudNumber(30) In LoudNumber.__init__, n=30 De f ine the class again 61
  62. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>>

    p = Person() >>> p.age Also: obj=<__main__.Person object at 0x108f15240>, objtype=<class '__main__.Person'> In LoudNumber.__get__, self.n=30 30 • We retrieve via p, the instance — that’s obj • The class attribute is de f ined on Person — that’s objtype What happens now? 62
  63. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>>

    Person.age Also: obj=None, objtype=<class '__main__.Person'> In LoudNumber.__get__, self.n=30 30 • We retrieve via Person, the the class — so obj is None! • The class attribute is de f ined on Person — that’s objtype And from the class? 63
  64. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    When we retrieve a method via the class, Python returns the original function that we de f ined. • It can tell, because obj is set to None • Thus, we need to supply an instance as the f irst argument • When we retrieve a method via the instance, Python does its magic rewriting • It takes obj, the instance on which we ran the method • It then calls the original function that we de f ined • It sticks obj into the f irst location — what is assigned to self, and returns a partial function Methods are descriptors 64
  65. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    A partial function is a function with some arguments “pre- loaded” into it from functools import partial def add(a, b): return a + b add5 = partial(add, 5) add5(10) Partial? 65
  66. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>>

    Person.greet <function __main__.Person.greet(self)> >>> p1.greet <bound method Person.greet of <__main__.Person object at 0x10902c5e0>> Remember this? 66
  67. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    We now understand that methods are class attributes • But where is our original function? • Stored on another class attribute, __dict__ • The method name (as a string) is the key • The original function is the value • Based on whether obj is None (i.e., we’re running things via the instance or the class), Python either returns the original function or the function with our f irst argument inserted there Where is our original function? 67
  68. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    When we ask for a.b in Python, a lot is going on! • Python looks for our attribute via ICPO • When we f inally f ind an attribute, it might be a descriptor • We use descriptors every day without knowing it • They allow us to use methods via either the instance or the class — whichever we prefer. Whew! 68
  69. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes •

    E-mail me: reuven@lerner.co.il • Follow me on Twitter: @reuvenmlerner • Free, weekly newsletters about Python: • About Python: BetterDevelopersWeekly.com • Come see me at my booth, for T shirts and a raffle of my books! Questions or comments? 69