Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Type hints, protocols, and good sense

Type hints, protocols, and good sense

PyCon India keynote, presented September 19, 2021.

Main sections:

1. The state of static typing in Python
2. The four modes of typing
3. The central role of typing.Protocol
4. Typing limits and how to address them

27c093d0834208f4712faaaec38c2c5c?s=128

Luciano Ramalho

September 19, 2021
Tweet

Transcript

  1. © 2021 Thoughtworks Type hints, protocols, and good sense

  2. © 2021 Thoughtworks 2

  3. © 2021 Thoughtworks Python’s static type hints are ready for

    use, but they are not perfect. Understand the caveats and workarounds. Avoiding them is the best solution in some cases. 3
  4. © 2021 Thoughtworks Fluent Python, First Edition Published in 9

    languages: 1. English 2. Portuguese 3. Japanese 4. Korean 5. Simplified Chinese 6. Traditional Chinese 7. French 8. Polish 9. Russian 4
  5. © 2021 Thoughtworks Fluent Python, Second Edition • Covers Python

    3.9 and 3.10, including pattern matching • Extensive coverage of type hints, with many examples • Updated coverage of async/await, with examples in asyncio, FastAPI and Curio • Draft available now at OReilly.com: https://bit.ly/FluPy2e 2021 Q4, est.) 5
  6. © 2021 Thoughtworks 6 Who am I to talk about

    this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.
  7. © 2021 Thoughtworks 7 Who am I to talk about

    this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.
  8. © 2021 Thoughtworks 8 Who am I to talk about

    this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.
  9. © 2021 Thoughtworks 9 Researched type hints to write about

    them Who am I to talk about type hints? Read lots of PEPs, messages, issue reports; interacted with some key proponents of type hints in Python; studied similar features in other languages—to write than 100 pages and dozens of examples in Fluent Python, Second Edition. Contributed bug fixes improving type hints on typeshed typeshed is the official repository of type hints used by all type checking tools, covering the standard library and other packages. Reorganized the documentation of Python’s typing module Proposed and implemented new overall organization to typing module docs in the Python 3.9 standard library, in collaboration with Guido van Rossum. Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.
  10. © 2021 Thoughtworks 10 The State of Static Typing in

    Python 1. The four modes of typing 2. The central role of typing.Protocol 3. Typing limits and how to address them 4. 10 © 2021 Thoughtworks Agenda
  11. © 2021 Thoughtworks 1. The State of Static Typing in

    Python 11
  12. © 2021 Thoughtworks A few type declarations... 12 12 ©

    2021 Thoughtworks
  13. © 2021 Thoughtworks ...and functions with type hints 13 13

    © 2021 Thoughtworks
  14. © 2021 Thoughtworks Static v. Dynamic Typing 14 14 ©

    2021 Thoughtworks static typing dynamic typing
  15. © 2021 Thoughtworks Static v. Dynamic Typing RUNTIME CHECKING A

    matter of when 15 15 © 2021 Thoughtworks STATIC CHECKING static typing dynamic typing
  16. © 2021 Thoughtworks Gradual typing Not a binary choice, but

    a continuum 16 16 © 2021 Thoughtworks static typing dynamic typing gradual typing
  17. © 2021 Thoughtworks Optional at All Levels Type hints may

    be omitted and/or type checking disabled on single lines, functions and entire packages. Python Adopted a Gradual Type System 17 The Default type is Any Any is consistent with all other types. It is more general than object, and is understood to implement all interfaces. Does Not Catch Errors at Runtime Type hints are not used at runtime, only by code analysis tools such as linters, IDEs, and dedicated type checkers. Does Not Increase Performance Current Python implementations do not use type hints to optimize bytecode or machine code generation.
  18. © 2021 Thoughtworks 18 TypeScript by Microsoft The most successful

    JavaScript dialect to date. Dart by Google The language of the Flutter SDK. Hack by Facebook PHP dialect supported by the JIT-enabled HHVM runtime. Other Languages with Gradual Typing
  19. © 2021 Thoughtworks Python will remain a dynamically typed language,

    and the authors have no desire to ever make type hints mandatory, even by convention. 19 Guido van Rossum, Jukka Lehtosalo, Łukasz Langa in PEP 484Type Hints 19 © 2021 Thoughtworks
  20. © 2021 Thoughtworks The best feature of gradual typing: type

    hints are always optional. 20 20 © 2021 Thoughtworks
  21. © 2021 Thoughtworks How type hints are used 21 Run

    time Import time CI pipeline Linting Writing code IDE features • Autocomplete • Instant warnings (wiggly underlines) • Refactoring assistance Stand-alone type checkers (same tools used for linting) Explicit type checking and services • explicit checks with isinstance() or issubclass() • Single dispatch functions • Data validation (e.g. pydantic) • API generation (e.g. FastAPI Stand-alone type checkers • Mypy • Pytype • Pyre • Pyright CLI Data class builders • @dataclasses.dataclass • typing.NamedTuple metaclass
  22. © 2021 Thoughtworks Companies with very large Python code bases

    invested heavily in type hints, and report they were also useful to migrate from Python 2.7. 22 Dropbox Large scale users... Employer of typing PEP authors. Released: Mypy. Facebook Employer of typing PEP authors. Released Pyre. Google Released Pytype. Microsoft Employer of typing PEP authors. Released Pyright. JetBrains Created proprietary type checker embedded in PyCharm IDE. ...and IDE vendors
  23. © 2021 Thoughtworks Python Developers Survey, October 2020 23 Source:

    https://bit.ly/PySurvey2020 Produced by the Python Software Foundation and JetBrains
  24. © 2021 Thoughtworks 24 “At least sometimes” 79% use code

    linters 78% write tests 71% use type hints
  25. © 2021 Thoughtworks 2. The four modes of typing 25

  26. © 2021 Thoughtworks Duck typing 26

  27. © 2021 Thoughtworks Don’t check whether it is-a duck: check

    whether it quacks-like-a duck, walks-like-a duck, etc, depending on exactly what subsetof duck-like behavior you need... 27 Alex Martelli in comp-lang-python, 20000726 "polymorphism (was Re: Type checking in python?" 27 © 2021 Thoughtworks
  28. © 2021 Thoughtworks Typing Map RUNTIME CHECKING 28 28 ©

    2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES static typing duck typing
  29. © 2021 Thoughtworks Typing Map RUNTIME CHECKING 29 29 ©

    2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES “Pyt on st le” “Jav st le” static typing duck typing
  30. © 2021 Thoughtworks Typing Map RUNTIME CHECKING 30 30 ©

    2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES su ported by AB static typing duck typing goose typing
  31. © 2021 Thoughtworks 31

  32. © 2021 Thoughtworks Typing Map RUNTIME CHECKING 32 32 ©

    2021 Thoughtworks static typing duck typing STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES goose typing static duck typing su ported by typing.Proto l su ported by AB
  33. © 2021 Thoughtworks Typing Map: languages RUNTIME CHECKING 33 33

    © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES Python TypeScript JavaScript Smalltalk Python ≥ 2.6 TypeScript Go Python ≥ 3.8 TypeScript Go Python ≥ 3.5 TypeScript Go Java static typing duck typing goose typing static duck typing
  34. © 2021 Thoughtworks 3. The central role of typing.Protocol 34

  35. © 2021 Thoughtworks typing.Protocol allows (static) duck typing 35 35

    © 2021 Thoughtworks
  36. © 2021 Thoughtworks typing.Protocol allows (static) duck typing 36 36

    © 2021 Thoughtworks
  37. © 2021 Thoughtworks typing.Protocol allows (static) duck typing 37 37

    © 2021 Thoughtworks
  38. © 2021 Thoughtworks 38

  39. © 2021 Thoughtworks 39

  40. © 2021 Thoughtworks median_low: fixed code 40 40 © 2021

    Thoughtworks etc.
  41. © 2021 Thoughtworks Uses of the SupportsLessThan Protocol Stub files

    for Python 3.9 standard library on typeshed 41 41 builtins: list.sort max min sorted statistics: median_low median_high functools: cmp_to_key bisect: bisect_left bisect_right insort_left insort_right heapq: nlargest nsmallest os.path: commonprefix
  42. © 2021 Thoughtworks Support duck typing with type hints The

    essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 42 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.
  43. © 2021 Thoughtworks Support duck typing with type hints The

    essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 43 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.
  44. © 2021 Thoughtworks Support duck typing with type hints The

    essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 44 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.
  45. © 2021 Thoughtworks 4. Typing limits and how to address

    them 45 45 © 2021 Thoughtworks
  46. © 2021 Thoughtworks Type hints are only helpful if they

    are used to enhance thought, not replace it. 46 Variation on a phrase by Brian Marick, in How to Misuse Code Coverage https://bit.ly/miscodecover 46 © 2021 Thoughtworks
  47. © 2021 Thoughtworks Imagine a programming environment that refuses to

    build a program unless there is 100% test coverage. 47
  48. © 2021 Thoughtworks False Positives and False Negatives The static

    type checking ecosystem—tools and annotations— are still immature and beneath the quality of Python itself. Static Typing is No Silver Bullet 48 No support for common, useful data constraints—even simple ones “Quantity must be int > 0” “Airport code must be str of length 3” “Email address must not be empty str” Type hints are generally unsuitable to check business rules. We need testing and runtime checking to verify business rules and algorithms.
  49. © 2021 Thoughtworks 49

  50. © 2021 Thoughtworks 50

  51. © 2021 Thoughtworks 51

  52. © 2021 Thoughtworks Complexity 52 52 © 2021 Thoughtworks

  53. © 2021 Thoughtworks A parade of typing-related PEPs 53 PEP

     Python Enhancement Proposal 22 and counting... No other aspect of Python has more PEPs.
  54. © 2021 Thoughtworks The max() built-in function 54 Flexible and

    easy to use, but very hard to annotate
  55. © 2021 Thoughtworks 55

  56. © 2021 Thoughtworks 56 fa neg t ve!

  57. © 2021 Thoughtworks max: old type hints 57 57 ©

    2021 Thoughtworks
  58. © 2021 Thoughtworks max: fixed type hints 58 58 ©

    2021 Thoughtworks
  59. © 2021 Thoughtworks max implemented in Python, for testing 59

    59 © 2021 Thoughtworks
  60. © 2021 Thoughtworks 60 60 26 Lines of code to

    implement all the documented functionality, with 2 constants, no imports 29 Lines of code for type hints: 7 imports, 4 definitions, and 6 overloaded signatures
  61. © 2021 Thoughtworks No support for convenient coding idioms, like

    unpacking config(**settings) Static type checkers can’t handle the expressive power of Python 61 Limited support for advanced features and abstractions Properties, descriptors, metaclasses, metaprogramming techniques in general Type checkers lag behind False reports and crashes when checking new features, sometimes more than a year after a Python release
  62. © 2021 Thoughtworks Good sense 62 62 © 2021 Thoughtworks

  63. © 2021 Thoughtworks Any code you can write in Python,

    you can test in Python. 63 63 © 2021 Thoughtworks
  64. © 2021 Thoughtworks If a Python program has adequate unit

    tests, it can be as robust as a C++, Java, or C# program with adequate unit tests (although the tests in Python will be faster to write). 64 Bruce Eckel, author of Thinking in C, On Java 8, Atomic Scala, and Atomic Kotlin in The Best Of Software Writing anthology by Joel Spolsky (ed.) 64 © 2021 Thoughtworks
  65. © 2021 Thoughtworks Compared with unit tests, type hints help

    detect a different and smaller set of bugs. Static type checking complements but cannot replace automated testing. 65 65 © 2021 Thoughtworks
  66. © 2021 Thoughtworks Python is a dynamic language with powerful

    abstractions and advanced metaprogramming features. That’s why we like it. 66 66 © 2021 Thoughtworks
  67. © 2021 Thoughtworks Teams can only avoid the downsides of

    type hints if they are not mandatory everywhere. 67 67 © 2021 Thoughtworks
  68. © 2021 Thoughtworks Embrace Gradual Typing with Mypy Some options

    help with gradual typing. Others are unhelpful: they force static typing everywhere. 68 Helpful for gradual typing: --disallow-incomplete-defs --check-untyped-defs --warn-unused-ignores Unhelpful for gradual typing: --strict --disallow-any-* 3 options) --disallow-untyped-* 2 options)
  69. © 2021 Thoughtworks Static typing is valuable in certain contexts

    For professional software development teams, the benefits of static typing surpass the cost. But remember that Python is widely used in many other contexts. Main takeaways 69
  70. © 2021 Thoughtworks Static typing is valuable in certain contexts

    For professional software development teams, the benefits of static typing surpass the cost. But remember that Python is widely used in many other contexts. Main takeaways 70 Use Protocols to enable Pythonic APIs Without the support for static duck typing provided by typing.Protocol, widespread use of type hints may lead to inflexible and verbose codebases. Define and use narrow protocols as needed.
  71. © 2021 Thoughtworks Static typing is valuable in certain contexts

    For professional software development teams, the benefits of static typing surpass its significant cost. But remember that Python is widely used in many other contexts. Main takeaways 71 Use Protocols to enable Pythonic APIs Without the support for static duck typing provided by typing.Protocol, widespread use of type hints may lead to inflexible and verbose codebases. Define and use narrow protocols as needed. Don’t make type hints mandatory everywhere Developers should be free to leave portions of the code without type hints, to avoid the downsides and limitations of the static type system and the tools that enforce it.
  72. © 2021 Thoughtworks Being optional is not a bug or

    limitation of Python type hints. It’s the feature that gives us the power to cope with the inherent complexities, annoyances, flaws, and limitations of static types. 72 72 © 2021 Thoughtworks
  73. © 2021 Thoughtworks Being optional is not a bug or

    limitation of Python type hints. It’s the feature that gives us the power to cope with the inherent complexities, annoyances, flaws, and limitations of static types. 73 73 © 2021 Thoughtworks
  74. © 2021 Thoughtworks Thank you! Luciano Ramalho Principal Consultant luciano.ramalho@thoughtworks.com

    Twitter: @ramalhoorg 74