typing.Protocol: type hints as Guido intented

Slide 1

Slide 1 text

typing.Protocol: type hints as Guido intended

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Fluent Python, Second Edition ● Covers 3.10, including pattern matching ● 100+ pages about type hints, with many examples ● New coverage of async/await, with examples in asyncio, FastAPI and Curio ● OReilly.com: https://bit.ly/FluPy2e Released May 2022 3

Slide 4

Slide 4 text

Slide 5

Slide 5 text

What is a type? 5

Slide 6

Slide 6 text

There are many definitions of the concept of type in the literature. Here we assume that type is a set of values and a set of functions that one can apply to these values. 6 Guido van Rossum, Ivan Levkivskyi in PEP 483—The Theory of Type Hints 6 © 2021 Thoughtworks

Slide 7

Slide 7 text

7 >>> i = 10 ** 23 >>> i 100000000000000000000000 >>> f = float(i) >>> f 1e+23 >>> i == f False >>> from decimal import Decimal >>> Decimal(i) Decimal('100000000000000000000000') >>> Decimal(f) Decimal('99999999999999991611392') >>> i | 2 100000000000000000000002 >>> f | 2 Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for |: 'float' and 'int'

Slide 8

Slide 8 text

8 >>> i = 10 ** 23 >>> i 100000000000000000000000 >>> f = float(i) >>> f 1e+23 >>> i == f False >>> from decimal import Decimal >>> Decimal(i) Decimal('100000000000000000000000') >>> Decimal(f) Decimal('99999999999999991611392') >>> i | 2 100000000000000000000002 >>> f | 2 Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for |: 'float' and 'int'

Slide 9

Slide 9 text

The “set of values” definition is not useful: Python does not provide practical ways to specify types as sets of values, except for Enum. We have very small sets (None, bool…) or extremely large ones (int, str…). 9

Slide 10

Slide 10 text

We have no way to say Quantity is the set of integers 0 < n ≤ 1000 or… AirportCode is the set of all 17576 three-letter, ASCII uppercase strings. 10

Slide 11

Slide 11 text

In practice, it’s more useful to think that int is a subtype of ﬂoat because it implements the same interface, in addition to extra methods* —not because int is a subset of ﬂoat** 11 *not completely true, but a useful approximation ** which it definitely isn’t

Slide 12

Slide 12 text

12 >>> i = 10 ** 23 >>> i 100000000000000000000000 >>> f = float(i) >>> f 1e+23 >>> i == f False >>> from decimal import Decimal >>> Decimal(i) Decimal('100000000000000000000000') >>> Decimal(f) Decimal('99999999999999991611392') >>> i | 2 100000000000000000000002 >>> f | 2 Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for |: 'float' and 'int'

Slide 13

Slide 13 text

Every Python value is of type object. Every Python value is of type Any. The object type implements a narrow interface, but Any is assumed to implement the widest possible interface: all possible methods! 13

Slide 14

Slide 14 text

There are many definitions of the concept of type in the literature. Here we assume that type is a set of values and a set of functions that one can apply to these values. 14 Guido van Rossum, Ivan Levkivskyi in PEP 483—The Theory of Type Hints 14 © 2021 Thoughtworks

Slide 15

Slide 15 text

Every object in Smalltalk, even a lowly integer, has a set of messages, a protocol, that defines the explicit communication to which that object can respond. 15 Dan Ingalls (Xerox PARC) in Design Principles Behind Smalltalk —BYTE Magazine, August 1981 15 © 2021 Thoughtworks

Slide 16

Slide 16 text

To summarize: ● Types are defined by interfaces ● Protocol is a synonym for interface 16

Slide 17

Slide 17 text

Duck typing 17

Slide 18

Slide 18 text

Don’t check whether it is-a duck: check whether it quacks-like-a duck, walks-like-a duck, etc, depending on exactly what subset of duck-like behavior you need... 18 Alex Martelli in comp-lang-python, 2000-07-26 "polymorphism (was Re: Type checking in python?)" 18 © 2021 Thoughtworks

Slide 19

Slide 19 text

19 >>> def double(x): ... return x * 2 ... >>> double(3) 6 >>> double(3.5) 7.0 >>> double(3j+4) (8+6j) >>> from fractions import Fraction >>> double(Fraction(1, 5)) Fraction(2, 5) >>> double('Spam') 'SpamSpam' >>> double([1, 2, 3]) [1, 2, 3, 1, 2, 3]

Slide 20

Slide 20 text

20 >>> class Train: ... def __init__(self, length): ... self.length = length ... def __len__(self): ... return self.length ... def __getitem__(self, i): ... if i < self.length: ... return f'car #{i+1}' ... raise IndexError ... >>> t = Train(3) >>> len(t) 3 >>> t[0] 'car #1' >>> for car in t: print(car) ... car #1 car #2 car #3

Slide 21

Slide 21 text

21 >>> class Train: ... def __init__(self, length): ... self.length = length ... def __len__(self): ... return self.length ... def __getitem__(self, i): ... if i < self.length: ... return f'car #{i+1}' ... raise IndexError ... >>> t = Train(3) >>> len(t) 3 >>> t[0] 'car #1' >>> for car in t: print(car) ... car #1 car #2 car #3

Slide 22

Slide 22 text

Example 1 A TextReader protocol 22

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Preserve the flexibility of duck typing Let your clients know what is the minimal interface expected, regardless of class hierarchies Benefits of using typing.Protocol 26 Support static analysis IDEs and linters can verify that an actual argument satisfies the protocol in the formal parameter Reduce coupling Client classes don’t need to subclass anything; just implement the protocol. This also makes testing easier.

Slide 27

Slide 27 text

Preserve the flexibility of duck typing Let your clients know what is the minimal interface expected, regardless of class hierarchies Benefits of using typing.Protocol 27 Support static analysis IDEs and linters can verify that an actual argument satisfies the protocol in the formal parameter Reduce coupling Client classes don’t need to subclass anything; just implement the protocol. This also makes testing easier.

Slide 28

Slide 28 text

Preserve the flexibility of duck typing Let your clients know what is the minimal interface expected, regardless of class hierarchies Benefits of using typing.Protocol 28 Support static analysis IDEs and linters can verify that an actual argument satisfies the protocol in the formal parameter Reduce coupling Client classes don’t need to subclass anything; just implement the protocol. This also makes testing easier.

Slide 29

Slide 29 text

The four modes of typing 29

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention. 32 Guido van Rossum, Jukka Lehtosalo, Łukasz Langa in PEP 484—Type Hints 32 © 2021 Thoughtworks

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Typing Map: languages RUNTIME CHECKING 38 38 © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES Python TypeScript JavaScript Smalltalk Python ≥ 2.6 TypeScript Go Python ≥ 3.8 TypeScript Go Python ≥ 3.5 TypeScript Go Java static typing duck typing goose typing static duck typing

Slide 39

Slide 39 text

More Examples 39

Slide 40

Slide 40 text

2. double 40

Slide 41

Slide 41 text

41 >>> def double(x): ... return x * 2 ... >>> double(3) 6 >>> double(3.5) 7.0 >>> double(3j+4) (8+6j) >>> from fractions import Fraction >>> double(Fraction(1, 5)) Fraction(2, 5) >>> double('Spam') 'SpamSpam' >>> double([1, 2, 3]) [1, 2, 3, 1, 2, 3]

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Slide 47

Slide 47 text

3. statistics.median_low 47

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Uses of the SupportsLessThan Protocol Stub files for Python 3.9 standard library on typeshed 51 51 builtins: list.sort max min sorted statistics: median_low median_high functools: cmp_to_key bisect: bisect_left bisect_right insort_left insort_right heapq: nlargest nsmallest os.path: commonprefix

Slide 52

Slide 52 text

4. max overload 52

Slide 53

Slide 53 text

The max() built-in function 53 Flexible and easy to use, but very hard to annotate

Slide 54

Slide 54 text

Slide 55

Slide 55 text

55 fa neg t ve!

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Slide 59

Slide 59 text

59 59 26 Lines of code to implement all the documented functionality, with 2 constants, no imports 29 Lines of code for type hints: 7 imports, 4 definitions, and 6 overloaded signatures

Slide 60

Slide 60 text

max overload: postscript 60 © 2021 Thoughtworks Months after I contributed SupportsLessThan to typeshed, a few things happened: ● Edge cases were discovered where SupportsGreaterThan was needed ● SupportsLessThan was replaced with SupportsGreaterThan, but this exposed symmetric bugs in cases that previously worked ● Both were superseded by SupportsDunderLT and SupportsDunderGT ● Almost all of their uses were replaced with a new type—the union of both of them—named SupportsRichComparison ○ This means most functions that involve comparisons in the standard library now have type hints that accept objects implementing either < or >. No need to implement both. ● For details, see: https://github.com/python/typeshed/blob/master/stdlib/_typeshed/__init__.pyi

Slide 61

Slide 61 text

5. Some protocols in the standard library 61

Slide 62

Slide 62 text

Protocols defined in the typing module 62

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Summary 64

Slide 65

Slide 65 text

Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 65 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.

Slide 66

Slide 66 text

Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 66 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.

Slide 67

Slide 67 text

Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 67 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.

Slide 68

Slide 68 text

Closing words 68

Slide 69

Slide 69 text

Being optional is not a bug or limitation of Python type hints. It’s the feature that gives us the power to cope with the inherent complexities, annoyances, ﬂaws, and limitations of static types. 69 69 © 2021 Thoughtworks

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Thank you! Luciano Ramalho Principal Consultant luciano.ramalho@thoughtworks.com Twitter: @ramalhoorg 71

Slide 72

Slide 72 text

Optional at All Levels Type hints may be omitted and/or type checking disabled on single lines, functions and entire packages. Python Adopted a Gradual Type System 72 The Default type is Any Any is consistent with all other types. It is more general than object, and is understood to implement all interfaces. Does Not Catch Errors at Runtime Type hints are not used at runtime, only by code analysis tools such as linters, IDEs, and dedicated type checkers. Does Not Increase Performance Current Python implementations do not use type hints to optimize bytecode or machine code generation.

Slide 73

Slide 73 text

73 TypeScript by Microsoft The most successful JavaScript dialect to date. Dart by Google The language of the Flutter SDK. Hack by Facebook PHP dialect supported by the JIT-enabled HHVM runtime. Other Languages with Gradual Typing

Slide 74

Slide 74 text

Slide 75

Slide 75 text

How type hints are used 75 Run time Import time CI pipeline Linting Writing code IDE features ● Autocomplete ● Instant warnings (wiggly underlines) ● Refactoring assistance Stand-alone type checkers (same tools used for linting) Explicit type checking and services ● explicit checks with isinstance() or issubclass() ● Single dispatch functions ● Data validation (e.g. pydantic) ● API generation (e.g. FastAPI) Stand-alone type checkers ● Mypy ● Pytype ● Pyre ● Pyright CLI Data class builders ● @dataclasses.dataclass ● typing.NamedTuple metaclass

Slide 76

Slide 76 text

Companies with very large Python code bases invested heavily in type hints, and report they were also useful to migrate from Python 2.7. 76 Dropbox Large scale users... Employer of typing PEP authors. Released: Mypy. Facebook Employer of typing PEP authors. Released Pyre. Google Released Pytype. Microsoft Employer of typing PEP authors. Released Pyright. JetBrains Created proprietary type checker embedded in PyCharm IDE. ...and IDE vendors

Slide 77

Slide 77 text

Python Developers Survey, October 2020 77 Source: https://bit.ly/PySurvey2020 Produced by the Python Software Foundation and JetBrains