Upgrade to Pro — share decks privately, control downloads, hide ads and more …

typing.Protocol: type hints as Guido intented

typing.Protocol: type hints as Guido intented

Presented at EuroPython 2022, Dublin, Ireland

Luciano Ramalho

July 15, 2022
Tweet

More Decks by Luciano Ramalho

Other Decks in Programming

Transcript

  1. typing.Protocol: type hints as Guido intended

  2. 2

  3. Fluent Python, Second Edition • Covers 3.10, including pattern matching

    • 100+ pages about type hints, with many examples • New coverage of async/await, with examples in asyncio, FastAPI and Curio • OReilly.com: https://bit.ly/FluPy2e Released May 2022 3
  4. 4 What is a type? 1. The four modes of

    typing 2. typing.Protocol examples 3. Conclusion 4. 4 © 2021 Thoughtworks Agenda
  5. What is a type? 5

  6. There are many definitions of the concept of type in

    the literature. Here we assume that type is a set of values and a set of functions that one can apply to these values. 6 Guido van Rossum, Ivan Levkivskyi in PEP 483—The Theory of Type Hints 6 © 2021 Thoughtworks
  7. 7 >>> i = 10 ** 23 >>> i 100000000000000000000000

    >>> f = float(i) >>> f 1e+23 >>> i == f False >>> from decimal import Decimal >>> Decimal(i) Decimal('100000000000000000000000') >>> Decimal(f) Decimal('99999999999999991611392') >>> i | 2 100000000000000000000002 >>> f | 2 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for |: 'float' and 'int'
  8. 8 >>> i = 10 ** 23 >>> i 100000000000000000000000

    >>> f = float(i) >>> f 1e+23 >>> i == f False >>> from decimal import Decimal >>> Decimal(i) Decimal('100000000000000000000000') >>> Decimal(f) Decimal('99999999999999991611392') >>> i | 2 100000000000000000000002 >>> f | 2 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for |: 'float' and 'int'
  9. The “set of values” definition is not useful: Python does

    not provide practical ways to specify types as sets of values, except for Enum. We have very small sets (None, bool…) or extremely large ones (int, str…). 9
  10. We have no way to say Quantity is the set

    of integers 0 < n ≤ 1000 or… AirportCode is the set of all 17576 three-letter, ASCII uppercase strings. 10
  11. In practice, it’s more useful to think that int is

    a subtype of float because it implements the same interface, in addition to extra methods* —not because int is a subset of float** 11 *not completely true, but a useful approximation ** which it definitely isn’t
  12. 12 >>> i = 10 ** 23 >>> i 100000000000000000000000

    >>> f = float(i) >>> f 1e+23 >>> i == f False >>> from decimal import Decimal >>> Decimal(i) Decimal('100000000000000000000000') >>> Decimal(f) Decimal('99999999999999991611392') >>> i | 2 100000000000000000000002 >>> f | 2 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for |: 'float' and 'int'
  13. Every Python value is of type object. Every Python value

    is of type Any. The object type implements a narrow interface, but Any is assumed to implement the widest possible interface: all possible methods! 13
  14. There are many definitions of the concept of type in

    the literature. Here we assume that type is a set of values and a set of functions that one can apply to these values. 14 Guido van Rossum, Ivan Levkivskyi in PEP 483—The Theory of Type Hints 14 © 2021 Thoughtworks
  15. Every object in Smalltalk, even a lowly integer, has a

    set of messages, a protocol, that defines the explicit communication to which that object can respond. 15 Dan Ingalls (Xerox PARC) in Design Principles Behind Smalltalk —BYTE Magazine, August 1981 15 © 2021 Thoughtworks
  16. To summarize: • Types are defined by interfaces • Protocol

    is a synonym for interface 16
  17. Duck typing 17

  18. Don’t check whether it is-a duck: check whether it quacks-like-a

    duck, walks-like-a duck, etc, depending on exactly what subset of duck-like behavior you need... 18 Alex Martelli in comp-lang-python, 2000-07-26 "polymorphism (was Re: Type checking in python?)" 18 © 2021 Thoughtworks
  19. 19 >>> def double(x): ... return x * 2 ...

    >>> double(3) 6 >>> double(3.5) 7.0 >>> double(3j+4) (8+6j) >>> from fractions import Fraction >>> double(Fraction(1, 5)) Fraction(2, 5) >>> double('Spam') 'SpamSpam' >>> double([1, 2, 3]) [1, 2, 3, 1, 2, 3]
  20. 20 >>> class Train: ... def __init__(self, length): ... self.length

    = length ... def __len__(self): ... return self.length ... def __getitem__(self, i): ... if i < self.length: ... return f'car #{i+1}' ... raise IndexError ... >>> t = Train(3) >>> len(t) 3 >>> t[0] 'car #1' >>> for car in t: print(car) ... car #1 car #2 car #3
  21. 21 >>> class Train: ... def __init__(self, length): ... self.length

    = length ... def __len__(self): ... return self.length ... def __getitem__(self, i): ... if i < self.length: ... return f'car #{i+1}' ... raise IndexError ... >>> t = Train(3) >>> len(t) 3 >>> t[0] 'car #1' >>> for car in t: print(car) ... car #1 car #2 car #3
  22. Example 1 A TextReader protocol 22

  23. typing.Protocol allows (static) duck typing 23 23 © 2021 Thoughtworks

  24. typing.Protocol allows (static) duck typing 24 24 © 2021 Thoughtworks

  25. typing.Protocol allows (static) duck typing 25 25 © 2021 Thoughtworks

  26. Preserve the flexibility of duck typing Let your clients know

    what is the minimal interface expected, regardless of class hierarchies Benefits of using typing.Protocol 26 Support static analysis IDEs and linters can verify that an actual argument satisfies the protocol in the formal parameter Reduce coupling Client classes don’t need to subclass anything; just implement the protocol. This also makes testing easier.
  27. Preserve the flexibility of duck typing Let your clients know

    what is the minimal interface expected, regardless of class hierarchies Benefits of using typing.Protocol 27 Support static analysis IDEs and linters can verify that an actual argument satisfies the protocol in the formal parameter Reduce coupling Client classes don’t need to subclass anything; just implement the protocol. This also makes testing easier.
  28. Preserve the flexibility of duck typing Let your clients know

    what is the minimal interface expected, regardless of class hierarchies Benefits of using typing.Protocol 28 Support static analysis IDEs and linters can verify that an actual argument satisfies the protocol in the formal parameter Reduce coupling Client classes don’t need to subclass anything; just implement the protocol. This also makes testing easier.
  29. The four modes of typing 29

  30. Static v. Dynamic Typing 30 30 © 2021 Thoughtworks static

    typing dynamic typing
  31. Static v. Dynamic Typing RUNTIME CHECKING A matter of when

    31 31 © 2021 Thoughtworks STATIC CHECKING static typing dynamic typing
  32. Python will remain a dynamically typed language, and the authors

    have no desire to ever make type hints mandatory, even by convention. 32 Guido van Rossum, Jukka Lehtosalo, Łukasz Langa in PEP 484—Type Hints 32 © 2021 Thoughtworks
  33. Typing Map RUNTIME CHECKING 33 33 © 2021 Thoughtworks STATIC

    CHECKING STRUCTURAL TYPES NOMINAL TYPES static typing duck typing
  34. Typing Map RUNTIME CHECKING 34 34 © 2021 Thoughtworks STATIC

    CHECKING STRUCTURAL TYPES NOMINAL TYPES “Pyt on st le” “Jav st le” static typing duck typing
  35. Typing Map RUNTIME CHECKING 35 35 © 2021 Thoughtworks STATIC

    CHECKING STRUCTURAL TYPES NOMINAL TYPES su porte by AB static typing duck typing goose typing
  36. 36

  37. Typing Map RUNTIME CHECKING 37 37 © 2021 Thoughtworks static

    typing duck typing STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES goose typing static duck typing su porte by typing.Proto l su porte by AB
  38. Typing Map: languages RUNTIME CHECKING 38 38 © 2021 Thoughtworks

    STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES Python TypeScript JavaScript Smalltalk Python ≥ 2.6 TypeScript Go Python ≥ 3.8 TypeScript Go Python ≥ 3.5 TypeScript Go Java static typing duck typing goose typing static duck typing
  39. More Examples 39

  40. 2. double 40

  41. 41 >>> def double(x): ... return x * 2 ...

    >>> double(3) 6 >>> double(3.5) 7.0 >>> double(3j+4) (8+6j) >>> from fractions import Fraction >>> double(Fraction(1, 5)) Fraction(2, 5) >>> double('Spam') 'SpamSpam' >>> double([1, 2, 3]) [1, 2, 3, 1, 2, 3]
  42. First take: object 42 42 © 2021 Thoughtworks Error: object

    does not implement __mul__
  43. Second take: Any 43 43 © 2021 Thoughtworks Useless: Any

    defeats type checking
  44. Third take: Sequence[T] 44 44 © 2021 Thoughtworks Only works

    with sequences, not numbers
  45. Fourth take: protocol misuse 45 45 © 2021 Thoughtworks Not

    OK: Type checker will assume that result only supports __mul__, but no other method.
  46. Solution: type variable bounded by protocol 46 46 © 2021

    Thoughtworks
  47. 3. statistics.median_low 47

  48. 48

  49. 49

  50. median_low: fixed code 50 50 © 2021 Thoughtworks etc. 🦆

  51. Uses of the SupportsLessThan Protocol Stub files for Python 3.9

    standard library on typeshed 51 51 builtins: list.sort max min sorted statistics: median_low median_high functools: cmp_to_key bisect: bisect_left bisect_right insort_left insort_right heapq: nlargest nsmallest os.path: commonprefix
  52. 4. max overload 52

  53. The max() built-in function 53 Flexible and easy to use,

    but very hard to annotate
  54. 54

  55. 55 fa neg t ve!

  56. max: old type hints 56 56 © 2021 Thoughtworks

  57. max: fixed type hints 57 57 © 2021 Thoughtworks

  58. max implemented in Python, for testing 58 58 © 2021

    Thoughtworks
  59. 59 59 26 Lines of code to implement all the

    documented functionality, with 2 constants, no imports 29 Lines of code for type hints: 7 imports, 4 definitions, and 6 overloaded signatures
  60. max overload: postscript 60 © 2021 Thoughtworks Months after I

    contributed SupportsLessThan to typeshed, a few things happened: • Edge cases were discovered where SupportsGreaterThan was needed • SupportsLessThan was replaced with SupportsGreaterThan, but this exposed symmetric bugs in cases that previously worked • Both were superseded by SupportsDunderLT and SupportsDunderGT • Almost all of their uses were replaced with a new type—the union of both of them—named SupportsRichComparison ◦ This means most functions that involve comparisons in the standard library now have type hints that accept objects implementing either < or >. No need to implement both. • For details, see: https://github.com/python/typeshed/blob/master/stdlib/_typeshed/__init__.pyi
  61. 5. Some protocols in the standard library 61

  62. Protocols defined in the typing module 62

  63. Example using SupportsIndex 63 63 © 2021 Thoughtworks 🦆

  64. Summary 64

  65. Support duck typing with type hints The essence of Python’s

    Data Model and standard library Use typing.Protocol to build Pythonic APIs 65 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.
  66. Support duck typing with type hints The essence of Python’s

    Data Model and standard library Use typing.Protocol to build Pythonic APIs 66 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.
  67. Support duck typing with type hints The essence of Python’s

    Data Model and standard library Use typing.Protocol to build Pythonic APIs 67 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.
  68. Closing words 68

  69. Being optional is not a bug or limitation of Python

    type hints. It’s the feature that gives us the power to cope with the inherent complexities, annoyances, flaws, and limitations of static types. 69 69 © 2021 Thoughtworks
  70. Being optional is not a bug or limitation of Python

    type hints. It’s the feature that gives us the power to cope with the inherent complexities, annoyances, flaws, and limitations of static types. 70 70 © 2021 Thoughtworks
  71. Thank you! Luciano Ramalho Principal Consultant luciano.ramalho@thoughtworks.com Twitter: @ramalhoorg 71

  72. Optional at All Levels Type hints may be omitted and/or

    type checking disabled on single lines, functions and entire packages. Python Adopted a Gradual Type System 72 The Default type is Any Any is consistent with all other types. It is more general than object, and is understood to implement all interfaces. Does Not Catch Errors at Runtime Type hints are not used at runtime, only by code analysis tools such as linters, IDEs, and dedicated type checkers. Does Not Increase Performance Current Python implementations do not use type hints to optimize bytecode or machine code generation.
  73. 73 TypeScript by Microsoft The most successful JavaScript dialect to

    date. Dart by Google The language of the Flutter SDK. Hack by Facebook PHP dialect supported by the JIT-enabled HHVM runtime. Other Languages with Gradual Typing
  74. The best feature of gradual typing: type hints are always

    optional. 74 74 © 2021 Thoughtworks
  75. How type hints are used 75 Run time Import time

    CI pipeline Linting Writing code IDE features • Autocomplete • Instant warnings (wiggly underlines) • Refactoring assistance Stand-alone type checkers (same tools used for linting) Explicit type checking and services • explicit checks with isinstance() or issubclass() • Single dispatch functions • Data validation (e.g. pydantic) • API generation (e.g. FastAPI) Stand-alone type checkers • Mypy • Pytype • Pyre • Pyright CLI Data class builders • @dataclasses.dataclass • typing.NamedTuple metaclass
  76. Companies with very large Python code bases invested heavily in

    type hints, and report they were also useful to migrate from Python 2.7. 76 Dropbox Large scale users... Employer of typing PEP authors. Released: Mypy. Facebook Employer of typing PEP authors. Released Pyre. Google Released Pytype. Microsoft Employer of typing PEP authors. Released Pyright. JetBrains Created proprietary type checker embedded in PyCharm IDE. ...and IDE vendors
  77. Python Developers Survey, October 2020 77 Source: https://bit.ly/PySurvey2020 Produced by

    the Python Software Foundation and JetBrains
  78. 78