Luciano Ramalho
July 15, 2022
390

# typing.Protocol: type hints as Guido intented

Presented at EuroPython 2022, Dublin, Ireland

July 15, 2022

## Transcript

1. typing.Protocol:
type hints as
Guido intended

2. 2

3. Fluent Python,
Second Edition
● Covers 3.10, including pattern
matching
● 100+ pages about type hints,
with many examples
● New coverage of async/await,
with examples in asyncio, FastAPI
and Curio
● OReilly.com:
https://bit.ly/FluPy2e
Released May 2022
3

4. 4
What is a type?
1.
The four modes
of typing
2.
typing.Protocol
examples
3.
Conclusion
4.
4
Agenda

5. What is a type?
5

6. There are many
definitions of the
concept of type in the
literature. Here we
assume that type is
a set of values and a
set of functions that
one can apply to these
values.
6
Guido van Rossum, Ivan Levkivskyi in
PEP 483—The Theory of Type Hints
6

7. 7
>>> i = 10 ** 23
>>> i
100000000000000000000000
>>> f = float(i)
>>> f
1e+23
>>> i == f
False
>>> from decimal import Decimal
>>> Decimal(i)
Decimal('100000000000000000000000')
>>> Decimal(f)
Decimal('99999999999999991611392')
>>> i | 2
100000000000000000000002
>>> f | 2
Traceback (most recent call last):
File "", line 1, in
TypeError: unsupported operand type(s) for |: 'float' and 'int'

8. 8
>>> i = 10 ** 23
>>> i
100000000000000000000000
>>> f = float(i)
>>> f
1e+23
>>> i == f
False
>>> from decimal import Decimal
>>> Decimal(i)
Decimal('100000000000000000000000')
>>> Decimal(f)
Decimal('99999999999999991611392')
>>> i | 2
100000000000000000000002
>>> f | 2
Traceback (most recent call last):
File "", line 1, in
TypeError: unsupported operand type(s) for |: 'float' and 'int'

9. The “set of values” definition is not
useful: Python does not provide
practical ways to specify types as sets
of values, except for Enum.
We have very small sets (None, bool…)
or extremely large ones (int, str…).
9

10. We have no way to say Quantity is the
set of integers 0 < n ≤ 1000
or…
AirportCode is the set of all 17576
three-letter, ASCII uppercase strings.
10

11. In practice, it’s more useful to think
that int is a subtype of ﬂoat because
it implements the same interface,
—not because int is a subset of ﬂoat**
11
*not completely true, but a useful approximation
** which it definitely isn’t

12. 12
>>> i = 10 ** 23
>>> i
100000000000000000000000
>>> f = float(i)
>>> f
1e+23
>>> i == f
False
>>> from decimal import Decimal
>>> Decimal(i)
Decimal('100000000000000000000000')
>>> Decimal(f)
Decimal('99999999999999991611392')
>>> i | 2
100000000000000000000002
>>> f | 2
Traceback (most recent call last):
File "", line 1, in
TypeError: unsupported operand type(s) for |: 'float' and 'int'

13. Every Python value is of type object.
Every Python value is of type Any.
The object type implements a narrow
interface, but Any is assumed to
implement the widest possible
interface: all possible methods!
13

14. There are many
definitions of the
concept of type in the
literature. Here we
assume that type is
a set of values and a
set of functions that
one can apply to these
values.
14
Guido van Rossum, Ivan Levkivskyi in
PEP 483—The Theory of Type Hints
14

15. Every object in
Smalltalk, even a lowly
integer, has a set of
messages, a protocol,
that defines the explicit
communication to
which that object can
respond.
15
Dan Ingalls (Xerox PARC) in
Design Principles Behind Smalltalk
—BYTE Magazine, August 1981 15

16. To summarize:
● Types are defined by interfaces
● Protocol is a synonym for interface
16

17. Duck typing
17

18. Don’t check whether it
is-a duck:
check whether it
quacks-like-a duck,
walks-like-a duck, etc,
depending on exactly
what subset of
duck-like behavior you
need...
18
Alex Martelli in comp-lang-python, 2000-07-26
"polymorphism (was Re: Type checking in python?)"
18

19. 19
>>> def double(x):
... return x * 2
...
>>> double(3)
6
>>> double(3.5)
7.0
>>> double(3j+4)
(8+6j)
>>> from fractions import Fraction
>>> double(Fraction(1, 5))
Fraction(2, 5)
>>> double('Spam')
'SpamSpam'
>>> double([1, 2, 3])
[1, 2, 3, 1, 2, 3]

20. 20
>>> class Train:
... def __init__(self, length):
... self.length = length
... def __len__(self):
... return self.length
... def __getitem__(self, i):
... if i < self.length:
... return f'car #{i+1}'
... raise IndexError
...
>>> t = Train(3)
>>> len(t)
3
>>> t[0]
'car #1'
>>> for car in t: print(car)
...
car #1
car #2
car #3

21. 21
>>> class Train:
... def __init__(self, length):
... self.length = length
... def __len__(self):
... return self.length
... def __getitem__(self, i):
... if i < self.length:
... return f'car #{i+1}'
... raise IndexError
...
>>> t = Train(3)
>>> len(t)
3
>>> t[0]
'car #1'
>>> for car in t: print(car)
...
car #1
car #2
car #3

22. Example 1
22

23. typing.Protocol allows (static) duck typing
23
23

24. typing.Protocol allows (static) duck typing
24
24

25. typing.Protocol allows (static) duck typing
25
25

26. Preserve the flexibility of duck typing
Let your clients know what is the
minimal interface expected, regardless
of class hierarchies
Benefits of using
typing.Protocol
26
Support static analysis
IDEs and linters can verify that an
actual argument satisfies the protocol
in the formal parameter
Reduce coupling
Client classes don’t need to subclass
anything; just implement the protocol.
This also makes testing easier.

27. Preserve the flexibility of duck typing
Let your clients know what is the
minimal interface expected, regardless
of class hierarchies
Benefits of using
typing.Protocol
27
Support static analysis
IDEs and linters can verify that an
actual argument satisfies the protocol
in the formal parameter
Reduce coupling
Client classes don’t need to subclass
anything; just implement the protocol.
This also makes testing easier.

28. Preserve the flexibility of duck typing
Let your clients know what is the
minimal interface expected, regardless
of class hierarchies
Benefits of using
typing.Protocol
28
Support static analysis
IDEs and linters can verify that an
actual argument satisfies the protocol
in the formal parameter
Reduce coupling
Client classes don’t need to subclass
anything; just implement the protocol.
This also makes testing easier.

29. The four
modes of typing
29

30. Static v. Dynamic Typing
30
30
static
typing
dynamic
typing

31. Static v. Dynamic Typing
RUNTIME
CHECKING
A matter of when
31
31
STATIC
CHECKING
static
typing
dynamic
typing

32. Python will remain a
dynamically typed
language, and the
authors have no desire
to ever make type
hints mandatory,
even by convention.
32
Guido van Rossum, Jukka Lehtosalo, Łukasz Langa
in PEP 484—Type Hints
32

33. Typing Map
RUNTIME
CHECKING
33
33
STATIC
CHECKING
STRUCTURAL TYPES
NOMINAL TYPES
static
typing
duck
typing

34. Typing Map
RUNTIME
CHECKING
34
34
STATIC
CHECKING
STRUCTURAL TYPES
NOMINAL TYPES
“Pyt on
st le”
“Jav
st le”
static
typing
duck
typing

35. Typing Map
RUNTIME
CHECKING
35
35
STATIC
CHECKING
STRUCTURAL TYPES
NOMINAL TYPES
su porte
by AB
static
typing
duck
typing
goose
typing

36. 36

37. Typing Map
RUNTIME
CHECKING
37
37
static
typing
duck
typing
STATIC
CHECKING
STRUCTURAL TYPES
NOMINAL TYPES
goose
typing
static
duck typing
su porte by
typing.Proto l
su porte
by AB

38. Typing Map: languages
RUNTIME
CHECKING
38
38
STATIC
CHECKING
STRUCTURAL TYPES
NOMINAL TYPES
Python
TypeScript
JavaScript
Smalltalk
Python ≥ 2.6
TypeScript
Go
Python ≥ 3.8
TypeScript
Go
Python ≥ 3.5
TypeScript
Go
Java
static
typing
duck
typing
goose
typing
static
duck typing

39. More
Examples
39

40. 2.
double
40

41. 41
>>> def double(x):
... return x * 2
...
>>> double(3)
6
>>> double(3.5)
7.0
>>> double(3j+4)
(8+6j)
>>> from fractions import Fraction
>>> double(Fraction(1, 5))
Fraction(2, 5)
>>> double('Spam')
'SpamSpam'
>>> double([1, 2, 3])
[1, 2, 3, 1, 2, 3]

42. First take: object
42
42
Error: object
does not
implement
__mul__

43. Second take: Any
43
43
Useless: Any
defeats type
checking

44. Third take: Sequence[T]
44
44
Only works with
sequences, not
numbers

45. Fourth take: protocol misuse
45
45
Not OK: Type checker will assume
that result only supports
__mul__, but no other method.

46. Solution: type variable bounded by protocol
46
46

47. 3.
statistics.median_low
47

48. 48

49. 49

50. median_low: fixed code
50
50
etc.
🦆

51. Uses of the SupportsLessThan Protocol
Stub files for Python 3.9 standard library on typeshed
51
51
builtins: list.sort
max
min
sorted
statistics: median_low
median_high
functools: cmp_to_key
bisect: bisect_left
bisect_right
insort_left
insort_right
heapq: nlargest
nsmallest
os.path: commonprefix

52. 4.
52

53. The max() built-in function
53
Flexible and easy to use, but very hard to annotate

54. 54

55. 55
fa
neg t ve!

56. max: old type hints
56
56

57. max: fixed type hints
57
57

58. max
implemented in
Python, for testing
58
58

59. 59
59
26
Lines of code to implement all
the documented functionality,
with 2 constants, no imports
29
Lines of code for type hints:
7 imports, 4 definitions, and

60
Months after I contributed SupportsLessThan to typeshed, a few things happened:
● Edge cases were discovered where SupportsGreaterThan was needed
● SupportsLessThan was replaced with SupportsGreaterThan, but this exposed
symmetric bugs in cases that previously worked
● Both were superseded by SupportsDunderLT and SupportsDunderGT
● Almost all of their uses were replaced with a new type—the union of both of
them—named SupportsRichComparison
○ This means most functions that involve comparisons in the standard library now have type hints that
accept objects implementing either < or >. No need to implement both.
● For details, see:
https://github.com/python/typeshed/blob/master/stdlib/_typeshed/__init__.pyi

61. 5.
Some protocols in the
standard library
61

62. Protocols defined in the typing module
62

63. Example using SupportsIndex
63
63
🦆

64. Summary
64

65. Support duck typing with type hints
The essence of Python’s
Data Model and standard library
Use typing.Protocol
to build Pythonic APIs
65
Interface Segregation Principle
Client code should not be forced to
depend on methods it does not use
Prefer narrow protocols
Single method protocols should be the
most common. Sometimes, two
methods. Rarely more.

66. Support duck typing with type hints
The essence of Python’s
Data Model and standard library
Use typing.Protocol
to build Pythonic APIs
66
Interface Segregation Principle
Client code should not be forced to
depend on methods it does not use
Prefer narrow protocols
Single method protocols should be the
most common. Sometimes, two
methods. Rarely more.

67. Support duck typing with type hints
The essence of Python’s
Data Model and standard library
Use typing.Protocol
to build Pythonic APIs
67
Interface Segregation Principle
Client code should not be forced to
depend on methods it does not use
Prefer narrow protocols
Single method protocols should be the
most common. Sometimes, two
methods. Rarely more.

68. Closing words
68

69. Being optional is not a
bug or limitation of
Python type hints.
It’s the feature that
gives us the power
to cope with the
inherent complexities,
annoyances, ﬂaws,
and limitations of
static types.
69
69

70. Being optional is not a
bug or limitation of
Python type hints.
It’s the feature that
gives us the power
to cope with the
inherent complexities,
annoyances, ﬂaws,
and limitations of
static types.
70
70

71. Thank you!
Luciano Ramalho
Principal Consultant
[email protected]
71

72. Optional at All Levels
Type hints may be omitted and/or
type checking disabled on single lines,
functions and entire packages.
72
The Default type is Any
Any is consistent with all other types.
It is more general than object, and
is understood to implement all interfaces.
Does Not Catch Errors at Runtime
Type hints are not used at runtime,
only by code analysis tools such as
linters, IDEs, and dedicated type
checkers.
Does Not Increase Performance
Current Python implementations
do not use type hints to optimize
bytecode or machine code generation.

73. 73
TypeScript
by Microsoft
The most successful
JavaScript dialect to date.
Dart
The language of the
Flutter SDK.
Hack
PHP dialect supported by the
JIT-enabled HHVM runtime.

74. The best feature of
type hints are always
optional.
74
74

75. How type hints are used
75
Run time
Import time
CI pipeline
Linting
Writing code
IDE features
● Autocomplete
● Instant warnings (wiggly underlines)
● Refactoring assistance
Stand-alone type checkers
(same tools used for linting)
Explicit type checking and services
● explicit checks with
isinstance() or issubclass()
● Single dispatch functions
● Data validation (e.g. pydantic)
● API generation (e.g. FastAPI)
Stand-alone type checkers
● Mypy
● Pytype
● Pyre
● Pyright CLI
Data class builders
● @dataclasses.dataclass
● typing.NamedTuple metaclass

76. Companies with very large Python code bases
invested heavily in type hints, and report they
were also useful to migrate from Python 2.7.
76
Dropbox
Large scale users...
Employer of typing
PEP authors.
Released: Mypy.
Employer of typing
PEP authors.
Released Pyre.
Released Pytype.
Microsoft
Employer of typing
PEP authors.
Released Pyright.
JetBrains
Created proprietary
type checker
embedded in
PyCharm IDE.
...and IDE vendors

77. Python Developers
Survey, October 2020
77
Source: https://bit.ly/PySurvey2020
Produced by the
Python Software Foundation
and JetBrains

78. 78