$30 off During Our Annual Pro Sale. View Details »

typing.Protocol: type hints as Guido intented

typing.Protocol: type hints as Guido intented

Presented at EuroPython 2022, Dublin, Ireland

Luciano Ramalho

July 15, 2022
Tweet

More Decks by Luciano Ramalho

Other Decks in Programming

Transcript

  1. typing.Protocol:
    type hints as
    Guido intended

    View Slide

  2. 2

    View Slide

  3. Fluent Python,
    Second Edition
    ● Covers 3.10, including pattern
    matching
    ● 100+ pages about type hints,
    with many examples
    ● New coverage of async/await,
    with examples in asyncio, FastAPI
    and Curio
    ● OReilly.com:
    https://bit.ly/FluPy2e
    Released May 2022
    3

    View Slide

  4. 4
    What is a type?
    1.
    The four modes
    of typing
    2.
    typing.Protocol
    examples
    3.
    Conclusion
    4.
    4
    © 2021 Thoughtworks
    Agenda

    View Slide

  5. What is a type?
    5

    View Slide

  6. There are many
    definitions of the
    concept of type in the
    literature. Here we
    assume that type is
    a set of values and a
    set of functions that
    one can apply to these
    values.
    6
    Guido van Rossum, Ivan Levkivskyi in
    PEP 483—The Theory of Type Hints
    6
    © 2021 Thoughtworks

    View Slide

  7. 7
    >>> i = 10 ** 23
    >>> i
    100000000000000000000000
    >>> f = float(i)
    >>> f
    1e+23
    >>> i == f
    False
    >>> from decimal import Decimal
    >>> Decimal(i)
    Decimal('100000000000000000000000')
    >>> Decimal(f)
    Decimal('99999999999999991611392')
    >>> i | 2
    100000000000000000000002
    >>> f | 2
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: unsupported operand type(s) for |: 'float' and 'int'

    View Slide

  8. 8
    >>> i = 10 ** 23
    >>> i
    100000000000000000000000
    >>> f = float(i)
    >>> f
    1e+23
    >>> i == f
    False
    >>> from decimal import Decimal
    >>> Decimal(i)
    Decimal('100000000000000000000000')
    >>> Decimal(f)
    Decimal('99999999999999991611392')
    >>> i | 2
    100000000000000000000002
    >>> f | 2
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: unsupported operand type(s) for |: 'float' and 'int'

    View Slide

  9. The “set of values” definition is not
    useful: Python does not provide
    practical ways to specify types as sets
    of values, except for Enum.
    We have very small sets (None, bool…)
    or extremely large ones (int, str…).
    9

    View Slide

  10. We have no way to say Quantity is the
    set of integers 0 < n ≤ 1000
    or…
    AirportCode is the set of all 17576
    three-letter, ASCII uppercase strings.
    10

    View Slide

  11. In practice, it’s more useful to think
    that int is a subtype of float because
    it implements the same interface,
    in addition to extra methods*
    —not because int is a subset of float**
    11
    *not completely true, but a useful approximation
    ** which it definitely isn’t

    View Slide

  12. 12
    >>> i = 10 ** 23
    >>> i
    100000000000000000000000
    >>> f = float(i)
    >>> f
    1e+23
    >>> i == f
    False
    >>> from decimal import Decimal
    >>> Decimal(i)
    Decimal('100000000000000000000000')
    >>> Decimal(f)
    Decimal('99999999999999991611392')
    >>> i | 2
    100000000000000000000002
    >>> f | 2
    Traceback (most recent call last):
    File "", line 1, in
    TypeError: unsupported operand type(s) for |: 'float' and 'int'

    View Slide

  13. Every Python value is of type object.
    Every Python value is of type Any.
    The object type implements a narrow
    interface, but Any is assumed to
    implement the widest possible
    interface: all possible methods!
    13

    View Slide

  14. There are many
    definitions of the
    concept of type in the
    literature. Here we
    assume that type is
    a set of values and a
    set of functions that
    one can apply to these
    values.
    14
    Guido van Rossum, Ivan Levkivskyi in
    PEP 483—The Theory of Type Hints
    14
    © 2021 Thoughtworks

    View Slide

  15. Every object in
    Smalltalk, even a lowly
    integer, has a set of
    messages, a protocol,
    that defines the explicit
    communication to
    which that object can
    respond.
    15
    Dan Ingalls (Xerox PARC) in
    Design Principles Behind Smalltalk
    —BYTE Magazine, August 1981 15
    © 2021 Thoughtworks

    View Slide

  16. To summarize:
    ● Types are defined by interfaces
    ● Protocol is a synonym for interface
    16

    View Slide

  17. Duck typing
    17

    View Slide

  18. Don’t check whether it
    is-a duck:
    check whether it
    quacks-like-a duck,
    walks-like-a duck, etc,
    depending on exactly
    what subset of
    duck-like behavior you
    need...
    18
    Alex Martelli in comp-lang-python, 2000-07-26
    "polymorphism (was Re: Type checking in python?)"
    18
    © 2021 Thoughtworks

    View Slide

  19. 19
    >>> def double(x):
    ... return x * 2
    ...
    >>> double(3)
    6
    >>> double(3.5)
    7.0
    >>> double(3j+4)
    (8+6j)
    >>> from fractions import Fraction
    >>> double(Fraction(1, 5))
    Fraction(2, 5)
    >>> double('Spam')
    'SpamSpam'
    >>> double([1, 2, 3])
    [1, 2, 3, 1, 2, 3]

    View Slide

  20. 20
    >>> class Train:
    ... def __init__(self, length):
    ... self.length = length
    ... def __len__(self):
    ... return self.length
    ... def __getitem__(self, i):
    ... if i < self.length:
    ... return f'car #{i+1}'
    ... raise IndexError
    ...
    >>> t = Train(3)
    >>> len(t)
    3
    >>> t[0]
    'car #1'
    >>> for car in t: print(car)
    ...
    car #1
    car #2
    car #3

    View Slide

  21. 21
    >>> class Train:
    ... def __init__(self, length):
    ... self.length = length
    ... def __len__(self):
    ... return self.length
    ... def __getitem__(self, i):
    ... if i < self.length:
    ... return f'car #{i+1}'
    ... raise IndexError
    ...
    >>> t = Train(3)
    >>> len(t)
    3
    >>> t[0]
    'car #1'
    >>> for car in t: print(car)
    ...
    car #1
    car #2
    car #3

    View Slide

  22. Example 1
    A TextReader protocol
    22

    View Slide

  23. typing.Protocol allows (static) duck typing
    23
    23
    © 2021 Thoughtworks

    View Slide

  24. typing.Protocol allows (static) duck typing
    24
    24
    © 2021 Thoughtworks

    View Slide

  25. typing.Protocol allows (static) duck typing
    25
    25
    © 2021 Thoughtworks

    View Slide

  26. Preserve the flexibility of duck typing
    Let your clients know what is the
    minimal interface expected, regardless
    of class hierarchies
    Benefits of using
    typing.Protocol
    26
    Support static analysis
    IDEs and linters can verify that an
    actual argument satisfies the protocol
    in the formal parameter
    Reduce coupling
    Client classes don’t need to subclass
    anything; just implement the protocol.
    This also makes testing easier.

    View Slide

  27. Preserve the flexibility of duck typing
    Let your clients know what is the
    minimal interface expected, regardless
    of class hierarchies
    Benefits of using
    typing.Protocol
    27
    Support static analysis
    IDEs and linters can verify that an
    actual argument satisfies the protocol
    in the formal parameter
    Reduce coupling
    Client classes don’t need to subclass
    anything; just implement the protocol.
    This also makes testing easier.

    View Slide

  28. Preserve the flexibility of duck typing
    Let your clients know what is the
    minimal interface expected, regardless
    of class hierarchies
    Benefits of using
    typing.Protocol
    28
    Support static analysis
    IDEs and linters can verify that an
    actual argument satisfies the protocol
    in the formal parameter
    Reduce coupling
    Client classes don’t need to subclass
    anything; just implement the protocol.
    This also makes testing easier.

    View Slide

  29. The four
    modes of typing
    29

    View Slide

  30. Static v. Dynamic Typing
    30
    30
    © 2021 Thoughtworks
    static
    typing
    dynamic
    typing

    View Slide

  31. Static v. Dynamic Typing
    RUNTIME
    CHECKING
    A matter of when
    31
    31
    © 2021 Thoughtworks
    STATIC
    CHECKING
    static
    typing
    dynamic
    typing

    View Slide

  32. Python will remain a
    dynamically typed
    language, and the
    authors have no desire
    to ever make type
    hints mandatory,
    even by convention.
    32
    Guido van Rossum, Jukka Lehtosalo, Łukasz Langa
    in PEP 484—Type Hints
    32
    © 2021 Thoughtworks

    View Slide

  33. Typing Map
    RUNTIME
    CHECKING
    33
    33
    © 2021 Thoughtworks
    STATIC
    CHECKING
    STRUCTURAL TYPES
    NOMINAL TYPES
    static
    typing
    duck
    typing

    View Slide

  34. Typing Map
    RUNTIME
    CHECKING
    34
    34
    © 2021 Thoughtworks
    STATIC
    CHECKING
    STRUCTURAL TYPES
    NOMINAL TYPES
    “Pyt on
    st le”
    “Jav
    st le”
    static
    typing
    duck
    typing

    View Slide

  35. Typing Map
    RUNTIME
    CHECKING
    35
    35
    © 2021 Thoughtworks
    STATIC
    CHECKING
    STRUCTURAL TYPES
    NOMINAL TYPES
    su porte
    by AB
    static
    typing
    duck
    typing
    goose
    typing

    View Slide

  36. 36

    View Slide

  37. Typing Map
    RUNTIME
    CHECKING
    37
    37
    © 2021 Thoughtworks
    static
    typing
    duck
    typing
    STATIC
    CHECKING
    STRUCTURAL TYPES
    NOMINAL TYPES
    goose
    typing
    static
    duck typing
    su porte by
    typing.Proto l
    su porte
    by AB

    View Slide

  38. Typing Map: languages
    RUNTIME
    CHECKING
    38
    38
    © 2021 Thoughtworks
    STATIC
    CHECKING
    STRUCTURAL TYPES
    NOMINAL TYPES
    Python
    TypeScript
    JavaScript
    Smalltalk
    Python ≥ 2.6
    TypeScript
    Go
    Python ≥ 3.8
    TypeScript
    Go
    Python ≥ 3.5
    TypeScript
    Go
    Java
    static
    typing
    duck
    typing
    goose
    typing
    static
    duck typing

    View Slide

  39. More
    Examples
    39

    View Slide

  40. 2.
    double
    40

    View Slide

  41. 41
    >>> def double(x):
    ... return x * 2
    ...
    >>> double(3)
    6
    >>> double(3.5)
    7.0
    >>> double(3j+4)
    (8+6j)
    >>> from fractions import Fraction
    >>> double(Fraction(1, 5))
    Fraction(2, 5)
    >>> double('Spam')
    'SpamSpam'
    >>> double([1, 2, 3])
    [1, 2, 3, 1, 2, 3]

    View Slide

  42. First take: object
    42
    42
    © 2021 Thoughtworks
    Error: object
    does not
    implement
    __mul__

    View Slide

  43. Second take: Any
    43
    43
    © 2021 Thoughtworks
    Useless: Any
    defeats type
    checking

    View Slide

  44. Third take: Sequence[T]
    44
    44
    © 2021 Thoughtworks
    Only works with
    sequences, not
    numbers

    View Slide

  45. Fourth take: protocol misuse
    45
    45
    © 2021 Thoughtworks
    Not OK: Type checker will assume
    that result only supports
    __mul__, but no other method.

    View Slide

  46. Solution: type variable bounded by protocol
    46
    46
    © 2021 Thoughtworks

    View Slide

  47. 3.
    statistics.median_low
    47

    View Slide

  48. 48

    View Slide

  49. 49

    View Slide

  50. median_low: fixed code
    50
    50
    © 2021 Thoughtworks
    etc.
    🦆

    View Slide

  51. Uses of the SupportsLessThan Protocol
    Stub files for Python 3.9 standard library on typeshed
    51
    51
    builtins: list.sort
    max
    min
    sorted
    statistics: median_low
    median_high
    functools: cmp_to_key
    bisect: bisect_left
    bisect_right
    insort_left
    insort_right
    heapq: nlargest
    nsmallest
    os.path: commonprefix

    View Slide

  52. 4.
    max overload
    52

    View Slide

  53. The max() built-in function
    53
    Flexible and easy to use, but very hard to annotate

    View Slide

  54. 54

    View Slide

  55. 55
    fa
    neg t ve!

    View Slide

  56. max: old type hints
    56
    56
    © 2021 Thoughtworks

    View Slide

  57. max: fixed type hints
    57
    57
    © 2021 Thoughtworks

    View Slide

  58. max
    implemented in
    Python, for testing
    58
    58
    © 2021 Thoughtworks

    View Slide

  59. 59
    59
    26
    Lines of code to implement all
    the documented functionality,
    with 2 constants, no imports
    29
    Lines of code for type hints:
    7 imports, 4 definitions, and
    6 overloaded signatures

    View Slide

  60. max overload: postscript
    60
    © 2021 Thoughtworks
    Months after I contributed SupportsLessThan to typeshed, a few things happened:
    ● Edge cases were discovered where SupportsGreaterThan was needed
    ● SupportsLessThan was replaced with SupportsGreaterThan, but this exposed
    symmetric bugs in cases that previously worked
    ● Both were superseded by SupportsDunderLT and SupportsDunderGT
    ● Almost all of their uses were replaced with a new type—the union of both of
    them—named SupportsRichComparison
    ○ This means most functions that involve comparisons in the standard library now have type hints that
    accept objects implementing either < or >. No need to implement both.
    ● For details, see:
    https://github.com/python/typeshed/blob/master/stdlib/_typeshed/__init__.pyi

    View Slide

  61. 5.
    Some protocols in the
    standard library
    61

    View Slide

  62. Protocols defined in the typing module
    62

    View Slide

  63. Example using SupportsIndex
    63
    63
    © 2021 Thoughtworks
    🦆

    View Slide

  64. Summary
    64

    View Slide

  65. Support duck typing with type hints
    The essence of Python’s
    Data Model and standard library
    Use typing.Protocol
    to build Pythonic APIs
    65
    Follow the
    Interface Segregation Principle
    Client code should not be forced to
    depend on methods it does not use
    Prefer narrow protocols
    Single method protocols should be the
    most common. Sometimes, two
    methods. Rarely more.

    View Slide

  66. Support duck typing with type hints
    The essence of Python’s
    Data Model and standard library
    Use typing.Protocol
    to build Pythonic APIs
    66
    Follow the
    Interface Segregation Principle
    Client code should not be forced to
    depend on methods it does not use
    Prefer narrow protocols
    Single method protocols should be the
    most common. Sometimes, two
    methods. Rarely more.

    View Slide

  67. Support duck typing with type hints
    The essence of Python’s
    Data Model and standard library
    Use typing.Protocol
    to build Pythonic APIs
    67
    Follow the
    Interface Segregation Principle
    Client code should not be forced to
    depend on methods it does not use
    Prefer narrow protocols
    Single method protocols should be the
    most common. Sometimes, two
    methods. Rarely more.

    View Slide

  68. Closing words
    68

    View Slide

  69. Being optional is not a
    bug or limitation of
    Python type hints.
    It’s the feature that
    gives us the power
    to cope with the
    inherent complexities,
    annoyances, flaws,
    and limitations of
    static types.
    69
    69
    © 2021 Thoughtworks

    View Slide

  70. Being optional is not a
    bug or limitation of
    Python type hints.
    It’s the feature that
    gives us the power
    to cope with the
    inherent complexities,
    annoyances, flaws,
    and limitations of
    static types.
    70
    70
    © 2021 Thoughtworks

    View Slide

  71. Thank you!
    Luciano Ramalho
    Principal Consultant
    [email protected]
    Twitter: @ramalhoorg
    71

    View Slide

  72. Optional at All Levels
    Type hints may be omitted and/or
    type checking disabled on single lines,
    functions and entire packages.
    Python Adopted a
    Gradual Type System
    72
    The Default type is Any
    Any is consistent with all other types.
    It is more general than object, and
    is understood to implement all interfaces.
    Does Not Catch Errors at Runtime
    Type hints are not used at runtime,
    only by code analysis tools such as
    linters, IDEs, and dedicated type
    checkers.
    Does Not Increase Performance
    Current Python implementations
    do not use type hints to optimize
    bytecode or machine code generation.

    View Slide

  73. 73
    TypeScript
    by Microsoft
    The most successful
    JavaScript dialect to date.
    Dart
    by Google
    The language of the
    Flutter SDK.
    Hack
    by Facebook
    PHP dialect supported by the
    JIT-enabled HHVM runtime.
    Other Languages with Gradual Typing

    View Slide

  74. The best feature of
    gradual typing:
    type hints are always
    optional.
    74
    74
    © 2021 Thoughtworks

    View Slide

  75. How type hints are used
    75
    Run time
    Import time
    CI pipeline
    Linting
    Writing code
    IDE features
    ● Autocomplete
    ● Instant warnings (wiggly underlines)
    ● Refactoring assistance
    Stand-alone type checkers
    (same tools used for linting)
    Explicit type checking and services
    ● explicit checks with
    isinstance() or issubclass()
    ● Single dispatch functions
    ● Data validation (e.g. pydantic)
    ● API generation (e.g. FastAPI)
    Stand-alone type checkers
    ● Mypy
    ● Pytype
    ● Pyre
    ● Pyright CLI
    Data class builders
    ● @dataclasses.dataclass
    ● typing.NamedTuple metaclass

    View Slide

  76. Companies with very large Python code bases
    invested heavily in type hints, and report they
    were also useful to migrate from Python 2.7.
    76
    Dropbox
    Large scale users...
    Employer of typing
    PEP authors.
    Released: Mypy.
    Facebook
    Employer of typing
    PEP authors.
    Released Pyre.
    Google
    Released Pytype.
    Microsoft
    Employer of typing
    PEP authors.
    Released Pyright.
    JetBrains
    Created proprietary
    type checker
    embedded in
    PyCharm IDE.
    ...and IDE vendors

    View Slide

  77. Python Developers
    Survey, October 2020
    77
    Source: https://bit.ly/PySurvey2020
    Produced by the
    Python Software Foundation
    and JetBrains

    View Slide

  78. 78

    View Slide