Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Type hints, protocols, and good sense

Luciano Ramalho
September 19, 2021

Type hints, protocols, and good sense

PyCon India keynote, presented September 19, 2021.

Main sections:

1. The state of static typing in Python
2. The four modes of typing
3. The central role of typing.Protocol
4. Typing limits and how to address them

Luciano Ramalho

September 19, 2021
Tweet

More Decks by Luciano Ramalho

Other Decks in Programming

Transcript

  1. © 2021 Thoughtworks
    Type hints,
    protocols,
    and good sense

    View Slide

  2. © 2021 Thoughtworks 2

    View Slide

  3. © 2021 Thoughtworks
    Python’s static type hints are ready for use,
    but they are not perfect.
    Understand the caveats and workarounds.
    Avoiding them is the best solution
    in some cases.
    3

    View Slide

  4. © 2021 Thoughtworks
    Fluent Python,
    First Edition
    Published in 9 languages:
    1. English
    2. Portuguese
    3. Japanese
    4. Korean
    5. Simplified Chinese
    6. Traditional Chinese
    7. French
    8. Polish
    9. Russian
    4

    View Slide

  5. © 2021 Thoughtworks
    Fluent Python,
    Second Edition
    ● Covers Python 3.9 and 3.10,
    including pattern matching
    ● Extensive coverage of type hints,
    with many examples
    ● Updated coverage of async/await,
    with examples in asyncio, FastAPI
    and Curio
    ● Draft available now at
    OReilly.com:
    https://bit.ly/FluPy2e
    2021 Q4, est.)
    5

    View Slide

  6. © 2021 Thoughtworks 6
    Who am I to talk
    about this?
    Using Python professionally since
    1998, at first I considered static typing
    a bad fit for the language.
    typing.Protocol addressed my
    main concern.
    I am now a happy user,
    with moderation.

    View Slide

  7. © 2021 Thoughtworks 7
    Who am I to talk
    about this?
    Using Python professionally since
    1998, at first I considered static typing
    a bad fit for the language.
    typing.Protocol addressed my
    main concern.
    I am now a happy user,
    with moderation.

    View Slide

  8. © 2021 Thoughtworks 8
    Who am I to talk
    about this?
    Using Python professionally since
    1998, at first I considered static typing
    a bad fit for the language.
    typing.Protocol addressed my
    main concern.
    I am now a happy user,
    with moderation.

    View Slide

  9. © 2021 Thoughtworks 9
    Researched type hints
    to write about them
    Who am I to talk
    about type hints?
    Read lots of PEPs, messages, issue
    reports; interacted with some key
    proponents of type hints in Python;
    studied similar features in other
    languages—to write than 100 pages
    and dozens of examples in Fluent
    Python, Second Edition.
    Contributed bug fixes
    improving type hints on
    typeshed
    typeshed is the official repository
    of type hints used by all type
    checking tools, covering the
    standard library and other
    packages.
    Reorganized the
    documentation of Python’s
    typing module
    Proposed and implemented new
    overall organization to typing
    module docs in the Python 3.9
    standard library, in collaboration
    with Guido van Rossum.
    Using Python professionally since
    1998, at first I considered static typing
    a bad fit for the language.
    typing.Protocol addressed my
    main concern.
    I am now a happy user,
    with moderation.

    View Slide

  10. © 2021 Thoughtworks 10
    The State of
    Static Typing
    in Python
    1.
    The four modes
    of typing
    2.
    The central role of
    typing.Protocol
    3.
    Typing limits
    and how to
    address them
    4.
    10
    © 2021 Thoughtworks
    Agenda

    View Slide

  11. © 2021 Thoughtworks
    1.
    The State of Static Typing
    in Python
    11

    View Slide

  12. © 2021 Thoughtworks
    A few type declarations...
    12
    12
    © 2021 Thoughtworks

    View Slide

  13. © 2021 Thoughtworks
    ...and functions with type hints
    13
    13
    © 2021 Thoughtworks

    View Slide

  14. © 2021 Thoughtworks
    Static v. Dynamic Typing
    14
    14
    © 2021 Thoughtworks
    static
    typing
    dynamic
    typing

    View Slide

  15. © 2021 Thoughtworks
    Static v. Dynamic Typing
    RUNTIME
    CHECKING
    A matter of when
    15
    15
    © 2021 Thoughtworks
    STATIC
    CHECKING
    static
    typing
    dynamic
    typing

    View Slide

  16. © 2021 Thoughtworks
    Gradual typing
    Not a binary choice, but a continuum
    16
    16
    © 2021 Thoughtworks
    static
    typing
    dynamic
    typing
    gradual
    typing

    View Slide

  17. © 2021 Thoughtworks
    Optional at All Levels
    Type hints may be omitted and/or
    type checking disabled on single lines,
    functions and entire packages.
    Python Adopted a
    Gradual Type System
    17
    The Default type is Any
    Any is consistent with all other types.
    It is more general than object, and
    is understood to implement all interfaces.
    Does Not Catch Errors at Runtime
    Type hints are not used at runtime,
    only by code analysis tools such as
    linters, IDEs, and dedicated type
    checkers.
    Does Not Increase Performance
    Current Python implementations
    do not use type hints to optimize
    bytecode or machine code generation.

    View Slide

  18. © 2021 Thoughtworks 18
    TypeScript
    by Microsoft
    The most successful
    JavaScript dialect to date.
    Dart
    by Google
    The language of the
    Flutter SDK.
    Hack
    by Facebook
    PHP dialect supported by the
    JIT-enabled HHVM runtime.
    Other Languages with Gradual Typing

    View Slide

  19. © 2021 Thoughtworks
    Python will remain a
    dynamically typed
    language, and the
    authors have no desire
    to ever make type
    hints mandatory,
    even by convention.
    19
    Guido van Rossum, Jukka Lehtosalo, Łukasz Langa
    in PEP 484Type Hints
    19
    © 2021 Thoughtworks

    View Slide

  20. © 2021 Thoughtworks
    The best feature of
    gradual typing:
    type hints are always
    optional.
    20
    20
    © 2021 Thoughtworks

    View Slide

  21. © 2021 Thoughtworks
    How type hints are used
    21
    Run time
    Import time
    CI pipeline
    Linting
    Writing code
    IDE features
    ● Autocomplete
    ● Instant warnings (wiggly underlines)
    ● Refactoring assistance
    Stand-alone type checkers
    (same tools used for linting)
    Explicit type checking and services
    ● explicit checks with
    isinstance() or issubclass()
    ● Single dispatch functions
    ● Data validation (e.g. pydantic)
    ● API generation (e.g. FastAPI
    Stand-alone type checkers
    ● Mypy
    ● Pytype
    ● Pyre
    ● Pyright CLI
    Data class builders
    ● @dataclasses.dataclass
    ● typing.NamedTuple metaclass

    View Slide

  22. © 2021 Thoughtworks
    Companies with very large Python code bases
    invested heavily in type hints, and report they
    were also useful to migrate from Python 2.7.
    22
    Dropbox
    Large scale users...
    Employer of typing
    PEP authors.
    Released: Mypy.
    Facebook
    Employer of typing
    PEP authors.
    Released Pyre.
    Google
    Released Pytype.
    Microsoft
    Employer of typing
    PEP authors.
    Released Pyright.
    JetBrains
    Created proprietary
    type checker
    embedded in
    PyCharm IDE.
    ...and IDE vendors

    View Slide

  23. © 2021 Thoughtworks
    Python Developers
    Survey, October 2020
    23
    Source: https://bit.ly/PySurvey2020
    Produced by the
    Python Software Foundation
    and JetBrains

    View Slide

  24. © 2021 Thoughtworks 24
    “At least sometimes”
    79% use code linters
    78% write tests
    71% use type hints

    View Slide

  25. © 2021 Thoughtworks
    2.
    The four modes
    of typing
    25

    View Slide

  26. © 2021 Thoughtworks
    Duck typing
    26

    View Slide

  27. © 2021 Thoughtworks
    Don’t check whether it
    is-a duck:
    check whether it
    quacks-like-a duck,
    walks-like-a duck, etc,
    depending on exactly
    what subsetof
    duck-like behavior you
    need...
    27
    Alex Martelli in comp-lang-python, 20000726
    "polymorphism (was Re: Type checking in python?"
    27
    © 2021 Thoughtworks

    View Slide

  28. © 2021 Thoughtworks
    Typing Map
    RUNTIME
    CHECKING
    28
    28
    © 2021 Thoughtworks
    STATIC
    CHECKING
    STRUCTURAL TYPES
    NOMINAL TYPES
    static
    typing
    duck
    typing

    View Slide

  29. © 2021 Thoughtworks
    Typing Map
    RUNTIME
    CHECKING
    29
    29
    © 2021 Thoughtworks
    STATIC
    CHECKING
    STRUCTURAL TYPES
    NOMINAL TYPES
    “Pyt on
    st le”
    “Jav
    st le”
    static
    typing
    duck
    typing

    View Slide

  30. © 2021 Thoughtworks
    Typing Map
    RUNTIME
    CHECKING
    30
    30
    © 2021 Thoughtworks
    STATIC
    CHECKING
    STRUCTURAL TYPES
    NOMINAL TYPES
    su ported
    by AB
    static
    typing
    duck
    typing
    goose
    typing

    View Slide

  31. © 2021 Thoughtworks 31

    View Slide

  32. © 2021 Thoughtworks
    Typing Map
    RUNTIME
    CHECKING
    32
    32
    © 2021 Thoughtworks
    static
    typing
    duck
    typing
    STATIC
    CHECKING
    STRUCTURAL TYPES
    NOMINAL TYPES
    goose
    typing
    static
    duck typing
    su ported by
    typing.Proto l
    su ported
    by AB

    View Slide

  33. © 2021 Thoughtworks
    Typing Map: languages
    RUNTIME
    CHECKING
    33
    33
    © 2021 Thoughtworks
    STATIC
    CHECKING
    STRUCTURAL TYPES
    NOMINAL TYPES
    Python
    TypeScript
    JavaScript
    Smalltalk
    Python ≥ 2.6
    TypeScript
    Go
    Python ≥ 3.8
    TypeScript
    Go
    Python ≥ 3.5
    TypeScript
    Go
    Java
    static
    typing
    duck
    typing
    goose
    typing
    static
    duck typing

    View Slide

  34. © 2021 Thoughtworks
    3.
    The central role of
    typing.Protocol
    34

    View Slide

  35. © 2021 Thoughtworks
    typing.Protocol allows (static) duck typing
    35
    35
    © 2021 Thoughtworks

    View Slide

  36. © 2021 Thoughtworks
    typing.Protocol allows (static) duck typing
    36
    36
    © 2021 Thoughtworks

    View Slide

  37. © 2021 Thoughtworks
    typing.Protocol allows (static) duck typing
    37
    37
    © 2021 Thoughtworks

    View Slide

  38. © 2021 Thoughtworks 38

    View Slide

  39. © 2021 Thoughtworks 39

    View Slide

  40. © 2021 Thoughtworks
    median_low: fixed code
    40
    40
    © 2021 Thoughtworks
    etc.

    View Slide

  41. © 2021 Thoughtworks
    Uses of the SupportsLessThan Protocol
    Stub files for Python 3.9 standard library on typeshed
    41
    41
    builtins: list.sort
    max
    min
    sorted
    statistics: median_low
    median_high
    functools: cmp_to_key
    bisect: bisect_left
    bisect_right
    insort_left
    insort_right
    heapq: nlargest
    nsmallest
    os.path: commonprefix

    View Slide

  42. © 2021 Thoughtworks
    Support duck typing with type hints
    The essence of Python’s
    Data Model and standard library
    Use typing.Protocol
    to build Pythonic APIs
    42
    Follow the
    Interface Segregation Principle
    Client code should not be forced to
    depend on methods it does not use
    Prefer narrow protocols
    Single method protocols should be the
    most common. Sometimes, two
    methods. Rarely more.

    View Slide

  43. © 2021 Thoughtworks
    Support duck typing with type hints
    The essence of Python’s
    Data Model and standard library
    Use typing.Protocol
    to build Pythonic APIs
    43
    Follow the
    Interface Segregation Principle
    Client code should not be forced to
    depend on methods it does not use
    Prefer narrow protocols
    Single method protocols should be the
    most common. Sometimes, two
    methods. Rarely more.

    View Slide

  44. © 2021 Thoughtworks
    Support duck typing with type hints
    The essence of Python’s
    Data Model and standard library
    Use typing.Protocol
    to build Pythonic APIs
    44
    Follow the
    Interface Segregation Principle
    Client code should not be forced to
    depend on methods it does not use
    Prefer narrow protocols
    Single method protocols should be the
    most common. Sometimes, two
    methods. Rarely more.

    View Slide

  45. © 2021 Thoughtworks
    4.
    Typing limits and
    how to address them
    45
    45
    © 2021 Thoughtworks

    View Slide

  46. © 2021 Thoughtworks
    Type hints are only
    helpful if they are used
    to enhance thought,
    not replace it.
    46
    Variation on a phrase by Brian Marick, in
    How to Misuse Code Coverage
    https://bit.ly/miscodecover 46
    © 2021 Thoughtworks

    View Slide

  47. © 2021 Thoughtworks
    Imagine a programming environment
    that refuses to build a program
    unless there is 100% test coverage.
    47

    View Slide

  48. © 2021 Thoughtworks
    False Positives and False Negatives
    The static type checking
    ecosystem—tools and annotations—
    are still immature and beneath the
    quality of Python itself.
    Static Typing is
    No Silver Bullet
    48
    No support for common, useful data
    constraints—even simple ones
    “Quantity must be int > 0”
    “Airport code must be str of length 3”
    “Email address must not be empty str”
    Type hints are generally unsuitable to
    check business rules.
    We need testing and runtime checking
    to verify business rules and algorithms.

    View Slide

  49. © 2021 Thoughtworks 49

    View Slide

  50. © 2021 Thoughtworks 50

    View Slide

  51. © 2021 Thoughtworks 51

    View Slide

  52. © 2021 Thoughtworks
    Complexity
    52
    52
    © 2021 Thoughtworks

    View Slide

  53. © 2021 Thoughtworks
    A parade of typing-related PEPs
    53
    PEP  Python Enhancement Proposal
    22 and counting...
    No other aspect of
    Python has more PEPs.

    View Slide

  54. © 2021 Thoughtworks
    The max() built-in function
    54
    Flexible and easy to use, but very hard to annotate

    View Slide

  55. © 2021 Thoughtworks 55

    View Slide

  56. © 2021 Thoughtworks 56
    fa
    neg t ve!

    View Slide

  57. © 2021 Thoughtworks
    max: old type hints
    57
    57
    © 2021 Thoughtworks

    View Slide

  58. © 2021 Thoughtworks
    max: fixed type hints
    58
    58
    © 2021 Thoughtworks

    View Slide

  59. © 2021 Thoughtworks
    max
    implemented in
    Python, for testing
    59
    59
    © 2021 Thoughtworks

    View Slide

  60. © 2021 Thoughtworks 60
    60
    26
    Lines of code to implement all
    the documented functionality,
    with 2 constants, no imports
    29
    Lines of code for type hints:
    7 imports, 4 definitions, and
    6 overloaded signatures

    View Slide

  61. © 2021 Thoughtworks
    No support for convenient
    coding idioms, like unpacking
    config(**settings)
    Static type checkers
    can’t handle the
    expressive power
    of Python
    61
    Limited support for advanced
    features and abstractions
    Properties, descriptors,
    metaclasses, metaprogramming
    techniques in general
    Type checkers lag behind
    False reports and crashes when
    checking new features, sometimes
    more than a year after a Python
    release

    View Slide

  62. © 2021 Thoughtworks
    Good sense
    62
    62
    © 2021 Thoughtworks

    View Slide

  63. © 2021 Thoughtworks
    Any code you can write
    in Python, you can test
    in Python.
    63
    63
    © 2021 Thoughtworks

    View Slide

  64. © 2021 Thoughtworks
    If a Python program
    has adequate unit
    tests, it can be as
    robust as a C++, Java,
    or C# program with
    adequate unit tests
    (although the tests in
    Python will be faster
    to write).
    64
    Bruce Eckel, author of Thinking in C, On Java 8,
    Atomic Scala, and Atomic Kotlin in The Best Of
    Software Writing anthology by Joel Spolsky (ed.) 64
    © 2021 Thoughtworks

    View Slide

  65. © 2021 Thoughtworks
    Compared with unit
    tests, type hints help
    detect a different and
    smaller set of bugs.
    Static type checking
    complements but
    cannot replace
    automated testing.
    65
    65
    © 2021 Thoughtworks

    View Slide

  66. © 2021 Thoughtworks
    Python is a dynamic
    language with
    powerful abstractions
    and advanced
    metaprogramming
    features.
    That’s why we like it.
    66
    66
    © 2021 Thoughtworks

    View Slide

  67. © 2021 Thoughtworks
    Teams can only avoid
    the downsides of type
    hints if they are not
    mandatory everywhere.
    67
    67
    © 2021 Thoughtworks

    View Slide

  68. © 2021 Thoughtworks
    Embrace Gradual
    Typing with Mypy
    Some options help with gradual typing.
    Others are unhelpful:
    they force static typing everywhere.
    68
    Helpful for gradual typing:
    --disallow-incomplete-defs
    --check-untyped-defs
    --warn-unused-ignores
    Unhelpful for gradual typing:
    --strict
    --disallow-any-* 3 options)
    --disallow-untyped-* 2 options)

    View Slide

  69. © 2021 Thoughtworks
    Static typing is valuable in certain contexts
    For professional software development teams,
    the benefits of static typing surpass the cost.
    But remember that Python is widely used in
    many other contexts.
    Main takeaways
    69

    View Slide

  70. © 2021 Thoughtworks
    Static typing is valuable in certain contexts
    For professional software development teams,
    the benefits of static typing surpass the cost.
    But remember that Python is widely used in
    many other contexts.
    Main takeaways
    70
    Use Protocols to enable Pythonic APIs
    Without the support for static duck typing
    provided by typing.Protocol, widespread
    use of type hints may lead to inflexible and
    verbose codebases.
    Define and use narrow protocols as needed.

    View Slide

  71. © 2021 Thoughtworks
    Static typing is valuable in certain contexts
    For professional software development teams,
    the benefits of static typing surpass its
    significant cost. But remember that Python is
    widely used in many other contexts.
    Main takeaways
    71
    Use Protocols to enable Pythonic APIs
    Without the support for static duck typing
    provided by typing.Protocol, widespread
    use of type hints may lead to inflexible and
    verbose codebases.
    Define and use narrow protocols as needed.
    Don’t make type hints mandatory everywhere
    Developers should be free to leave portions of
    the code without type hints, to avoid the
    downsides and limitations of the static type
    system and the tools that enforce it.

    View Slide

  72. © 2021 Thoughtworks
    Being optional is not a
    bug or limitation of
    Python type hints.
    It’s the feature that
    gives us the power
    to cope with the
    inherent complexities,
    annoyances, flaws,
    and limitations of
    static types.
    72
    72
    © 2021 Thoughtworks

    View Slide

  73. © 2021 Thoughtworks
    Being optional is not a
    bug or limitation of
    Python type hints.
    It’s the feature that
    gives us the power
    to cope with the
    inherent complexities,
    annoyances, flaws,
    and limitations of
    static types.
    73
    73
    © 2021 Thoughtworks

    View Slide

  74. © 2021 Thoughtworks
    Thank you!
    Luciano Ramalho
    Principal Consultant
    [email protected]
    Twitter: @ramalhoorg
    74

    View Slide