Understanding Python attributes

Attributes in Python, which we use dozens of times each day, seem boring, obvious, and not worthy of attention. But it turns out that they're key to the Python language: Every time you say a.b in Python, that little dot is hiding a lot of work, from searching across multiple objects to silently rewriting things. And it turns out that what happens with attributes, while not always obvious to developers, determines a great deal of behavior in the Python language.

In this talk from PyCon US 2022, I discuss what attributes are (and aren't), what Python does when you use a dot (.) in your code, and how you can take advantage of it. I talk about attribute lookup, about inheritance, and about methods vs. functions. I also look into properties, and how they allow us to have attributes that look like data but behave like setters and getters. Finally, I look at the descriptor protocol, which makes so much of Python's functionality possible, including the automatic insertion of "self" as the first argument in method calls.

Transcript

1. Understanding attributes (Or: They're not nearly as boring as you think!) Reuven M. Lerner • PyCon US 2022 reuven@lerner.co.il • @reuvenmlerner
2. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Corporate • Video • Hybrid • Weekly Python Exercise • Books: • Python Workout • Pandas Workout • Come see my booth in the expo hall for swag + discounts! I teach Python! 2
3. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes x = 100 • This does not mean: Put the value of 100 in the memory location called “x” • This does mean: The name “x” should refer to the integer 100 Let’s assign to a variable! 3
4. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class MyClass: # do-nothing class pass x = MyClass() # create an instance, assign to x x.y = 100 # assign 100 to x.y • y is not a variable. It’s an attribute. Let’s get fancier 4
5. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Every object in Python has attributes • They’re basically a private dictionary — but you use a dot (.) rather than [] (square brackets). • The attribute doesn’t exist on the variable, but rather on the object that the variable refers to. Variables vs. attributes 5
6. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • We retrieve values from attributes quite a bit: • sys.version • str.upper('abc') • random.randint(0, 100) • Just like variables, attributes can contain any Python object — both data and functions. Reading from attributes 6
7. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • With rare exception, you can set any attribute on any object x.y = 100 >>> sys.version = '4.00' >>> sys.version '4.00' >>> sys.version = '4.20' >>> sys.version '4.20' Setting attributes 7
8. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • In fact, the __init__ method is meant to set attributes! class Person: def __init__(self, name): self.name = name p1 = Person('name1') p2 = Person('name2') We set attributes all the time 8
9. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes When __init__ is running 9
10. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes After __init__ returns 10
11. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Person: def __init__(self, name): self.name = name p1 = Person('name1') p2 = Person('name2') • Let’s keep track of the population as we add people! What’s missing from this program? 11
12. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes population = 0 class Person: def __init__(self, name): population += 1 self.name = name def greet(self): return f'Hello, {self.name}!' print(f'Before, {population=}') p1 = Person(‘name1’) p2 = Person(‘name2’) print(f'After, {population=}’) Idea: A global variable 12
13. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes UnboundLocalError 13
14. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes population = 0 class Person: def __init__(self, name): population += 1 self.name = name def greet(self): return f'Hello, {self.name}!' print(f'Before, {population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {population=}') The problem? 14
15. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes population = 0 class Person: def __init__(self, name): global population population += 1 self.name = name def greet(self): return f'Hello, {self.name}!' print(f'Before, {population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {population=}') print(p1.greet()) print(p2.greet()) “global” to the rescue! 15
16. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Everything in Python is an object • Every object has attributes • We can set and retrieve attributes on every object • What if we set an attribute on the Person class? A better alternative 16
17. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Person: def __init__(self, name): Person.population += 1 self.name = name def greet(self): return f'Hello, {self.name}!' Person.population = 0 print(f'Before, {Person.population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {Person.population=}') print(p1.greet()) print(p2.greet()) Class attribute 17
18. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • First we’re creating the Person class • Then, after we’re done, we add the “population” attribute • It works, but this can’t be the best way to do it This is pretty ugly 18
19. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes Quiz time! 19
20. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes print('A') class Person: print('B') def __init__(self, name): print('C') self.name = name print('D') print('E') p1 = Person('name1') p2 = Person('name2') What will this code print? 20
21. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes print('A') class Person: print('B') def __init__(self, name): print('C') self.name = name print('D') print('E') p1 = Person('name1') p2 = Person('name2') The answer 21 A B D E C C 1 2 3 4 5
22. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Function and class de f initions look similar, but they aren’t • A function body doesn’t execute when it’s de f ined • A class body, though, must execute when it’s de f ined class Person: def __init__(self, name): self.name = name p1 = Person('name1') p2 = Person('name2') Why? 22 Before this can happen This has to happen
23. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • But wait — def normally does two things: • Creates a function object • Assigns the object to a variable, the function’s name • But here, def is de f ining __init__, which is… what? 
 What about de f ? 23
24. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes Methods are class attributes 24
25. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Functions de f ined in a class body are actually class attributes • Inside of the class, it looks like a variable • Outside of the class, it looks like an attribute on the class • Where have we seen this before? Modules! • When we import a module, any global variables in the module are available as attributes on the module object. • So any functions we de f ine inside a class aren’t variables, but class attributes. • And any assignments we make in a class don’t create variables, but rather class attributes. Classes are f ile-less modules 25
26. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Person: population = 0 # not a variable definition! def __init__(self, name): Person.population += 1 self.name = name print(f'Before, {Person.population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {Person.population=}') Class attribute, nicer edition 26
27. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • People who come from other languages sometimes call class attributes “static variables.” • Don’t do this! • Static variables are shared across the class and instances • But a class attribute is a name that exists only on the class • Python doesn’t have the concept of a “shared attribute.” • However, many different attributes (and variables) might refer to the same object. Static variables?!? 27
28. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Person: population = 0 def __init__(self, name): Person.population += 1 self.name = name print(f'Before, {Person.population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {Person.population=}') print(f'After, {p1.population=}') print(f’After, {p2.population=}’) Let’s see that they’re different 28 2 2 2
29. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • It worked. • And we got the same values. • Maybe they are actually shared? Uh oh 29
30. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Nope! • We’ve stumbled onto the attribute lookup rule in Python, which I call ICPO: • I: First Python looks on the instance in the expression. • C: If it doesn’t f ind the attribute on the speci f ied instance, it looks on it’s class. • P: Not on the class? Python looks on the class’s parent • It keeps looking up and up, on each class’s parent • O: The f inal stop is object, the top of Python’s class hierarchy ICPO 30
31. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Person: population = 0 # not a variable definition! def __init__(self, name): Person.population += 1 self.name = name print(f'Before, {Person.population=}') p1 = Person('name1') p2 = Person('name2') print(f'After, {Person.population=}') print(f'After, {p1.population=}') print(f'After, {p2.population=}') Let’s walk through this 31 I: Does Person have the attribute “population”? YES, 2 I: Does p1 have the attribute “population”? NO. C: Does Person have the attribute “population”? YES, 2 I: Does p2 have the attribute “population”? NO. C: Does Person have the attribute “population”? YES, 2
32. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes ICPO explains method lookup 32
33. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Person: def __init__(self, name): self.name = name def greet(self): return f'Hello, {self.name}!' p1 = Person('name1') p2 = Person('name2') print(p1.greet()) print(p2.greet()) A new version of Person 33 I: Does p1 have the attribute “greet”? NO. C: Does Person have the attribute “greet”? YES! I: Does p2 have the attribute “greet”? NO. C: Does Person have the attribute “greet”? YES!
34. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Methods are class attributes… • …but we normally call them via the instance • ICPO explains how this can be. ICPO explains method lookup 34
35. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Employee, which is almost exactly like Person • Employees are created with two attributes, name and id_number • Otherwise, they’re the same as people We need a new class 35
36. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Person: def __init__(self, name): self.name = name def greet(self): return f'Hello, {self.name}!' p1 = Person('name1') p2 = Person('name2') print(p1.greet()) print(p2.greet()) Person 36
37. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Employee: def __init__(self, name, id_number): self.name = name self.id_number = id_number def greet(self): return f'Hello, {self.name}!' e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) Let’s de f ine Employee! 37
38. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes Hello, name1 Hello, name2 Hello, emp1 Hello, emp2 Does it work? 38
39. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Employee(Person): def __init__(self, name, id_number): self.name = name self.id_number = id_number def greet(self): return f'Hello, {self.name}!' e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) Try again, but with inheritance 39 This inserts “Person” into the ICPO attribute search
40. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • The “greet” method is exactly the same in both Person and Employee • So we don’t need to implement it in Employee! Let’s use inheritance 40
41. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Employee(Person): def __init__(self, name, id_number): self.name = name self.id_number = id_number e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) Better use of inheritance 41
42. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Do we need to set self.name in Employee.__init__? • After all, we set it in Person.__init__, right? What about name? 42
43. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Employee(Person): def __init__(self, name, id_number): self.id_number = id_number e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) What happens now? 43 Assume it’ll be set in Person.__init__
44. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes AttributeError: 'Employee' object has no attribute ‘name' • Huh? Let’s check, using “vars”: >>> print(vars(e1)) {'id_number': 1} >>> print(vars(e2)) {'id_number': 2} • The “name” attribute was never set! It doesn’t go well… 44
45. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • When we created each instance of Employee: • I: We asked the new instance, “Do you have __init__?” No. • C: We asked Employee, “Do you have __init__?” Yes. • Employee.__init__ ran, setting self.id_number • Because Employee.__init__ was found, Person.__init__ never ran • Class de f initions in Python aren’t merged! If Person.__init__ doesn’t run, then the attributes it adds aren’t added • We end up with nameless employees… • …which ends badly when “greet” asks for self.name The problem? 45
46. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Employee(Person): def __init__(self, name, id_number): Person.__init__(self, name) self.id_number = id_number e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) Solution: Explicitly call Person.__init__ 46 # Hello, emp1! # Hello, emp2!
47. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class Employee(Person): def __init__(self, name, id_number): super().__init__(name) self.id_number = id_number e1 = Employee('emp1', 1) e2 = Employee('emp2', 2) print(e1.greet()) print(e2.greet()) Better: Use super 47 # Hello, emp1! # Hello, emp2!
48. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • What happens when I print(p1) or print(e1)? >>> print(p1) <__main__.Person object at 0x10902c5e0> >>> print(e1) <__main__.Employee object at 0x108f14910> Printing our objects 48
49. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Calling “print” invokes str, which calls the __str__ method. • With p1: • I: Does p1 have __str__? No. • C: Does p1’s class, Person, have __str__? No. • O: Does object have __str__? Yes! • With e1: • I: Does e1 have __str__? No. • C: Does e1’s class, Employee, have __str__? No. • P: Does e1’s class’s parent, Person, have __str__? No. • O: Does object have __str__? Yes! Why so ugly? 49
50. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Attribute lookup is the reason that operator overloading works • De f ine __str__ in your class, and it is found earlier than “object” Operator overloading! 50
51. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Python loves to be explicit, and to work without magic. • But there’s a big piece of magic that we all know, and don’t complain about: Method rewriting. >>> s = 'abcd' >>> s.upper() 'ABCD' >>> str.upper(s) 'ABCD' Something is still missing 51
52. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • We know that methods are stored in class attributes • We know that we can retrieve a class attribute either via the class or via the instance • But somehow, we’re getting different behavior depending on how we retrieve it! Something is weird here 52
53. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>> Person.greet <function __main__.Person.greet(self)> >>> p1.greet <bound method Person.greet of <__main__.Person object at 0x10902c5e0>> And indeed: 53
54. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • Normally, when I retrieve a class attribute, I get the object that is stored in the attribute. • But if: • The attribute’s class has a __get__ method • Then the result of __get__ is returned instead Descriptors 54
55. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class LoudNumber: def __init__(self, n): print(f'In LoudNumber.__init__, {n=}') self.n = n def __get__(self, *args): print(f'In LoudNumber.__get__, {self.n=}') return self.n • If I assign an instance of LoudNumber to a variable, nothing special happens. • But if I assign it to a class attribute, magic happens. Example 55
56. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>> class Person: age = LoudNumber(30) In LoudNumber.__init__, n=30 >>> p = Person() Use it in a class attribute 56
57. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes 57
58. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>> p.age In LoudNumber.__get__, self.n=30 30 >>> Person.age In LoudNumber.__get__, self.n=30 30 Retrieve the value 58
59. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • I retrieve the attribute • I don’t use parentheses • Despite this, a method has run! Notice 59
60. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes class LoudNumber: def __init__(self, n): print(f'In LoudNumber.__init__, {n=}') self.n = n def __get__(self, obj, objtype): print(f'Also: {obj=}, {objtype=}') print(f'In LoudNumber.__get__, {self.n=}') return self.n What are __get__’s parameters? 60 From what instance is this attribute being retrieved? In what class is this attribute stored?
61. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>> class Person: age = LoudNumber(30) In LoudNumber.__init__, n=30 De f ine the class again 61
62. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>> p = Person() >>> p.age Also: obj=<__main__.Person object at 0x108f15240>, objtype=<class '__main__.Person'> In LoudNumber.__get__, self.n=30 30 • We retrieve via p, the instance — that’s obj • The class attribute is de f ined on Person — that’s objtype What happens now? 62
63. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>> Person.age Also: obj=None, objtype=<class '__main__.Person'> In LoudNumber.__get__, self.n=30 30 • We retrieve via Person, the the class — so obj is None! • The class attribute is de f ined on Person — that’s objtype And from the class? 63
64. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • When we retrieve a method via the class, Python returns the original function that we de f ined. • It can tell, because obj is set to None • Thus, we need to supply an instance as the f irst argument • When we retrieve a method via the instance, Python does its magic rewriting • It takes obj, the instance on which we ran the method • It then calls the original function that we de f ined • It sticks obj into the f irst location — what is assigned to self, and returns a partial function Methods are descriptors 64
65. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • A partial function is a function with some arguments “pre- loaded” into it from functools import partial def add(a, b): return a + b add5 = partial(add, 5) add5(10) Partial? 65
66. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes >>> Person.greet <function __main__.Person.greet(self)> >>> p1.greet <bound method Person.greet of <__main__.Person object at 0x10902c5e0>> Remember this? 66
67. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • We now understand that methods are class attributes • But where is our original function? • Stored on another class attribute, __dict__ • The method name (as a string) is the key • The original function is the value • Based on whether obj is None (i.e., we’re running things via the instance or the class), Python either returns the original function or the function with our f irst argument inserted there Where is our original function? 67
68. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • When we ask for a.b in Python, a lot is going on! • Python looks for our attribute via ICPO • When we f inally f ind an attribute, it might be a descriptor • We use descriptors every day without knowing it • They allow us to use methods via either the instance or the class — whichever we prefer. Whew! 68
69. Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il Understanding Attributes • E-mail me: reuven@lerner.co.il • Follow me on Twitter: @reuvenmlerner • Free, weekly newsletters about Python: • About Python: BetterDevelopersWeekly.com • Come see me at my booth, for T shirts and a raffle of my books! Questions or comments? 69