Type hints, protocols, and good sense

PyCon India keynote, presented September 19, 2021.

Main sections:

1. The state of static typing in Python
2. The four modes of typing
3. The central role of typing.Protocol
4. Typing limits and how to address them

1. © 2021 Thoughtworks Type hints, protocols, and good sense
2. © 2021 Thoughtworks 2
3. © 2021 Thoughtworks Python’s static type hints are ready for use, but they are not perfect. Understand the caveats and workarounds. Avoiding them is the best solution in some cases. 3
4. © 2021 Thoughtworks Fluent Python, First Edition Published in 9 languages: 1. English 2. Portuguese 3. Japanese 4. Korean 5. Simplified Chinese 6. Traditional Chinese 7. French 8. Polish 9. Russian 4
5. © 2021 Thoughtworks Fluent Python, Second Edition • Covers Python 3.9 and 3.10, including pattern matching • Extensive coverage of type hints, with many examples • Updated coverage of async/await, with examples in asyncio, FastAPI and Curio • Draft available now at OReilly.com: https://bit.ly/FluPy2e 2021 Q4, est.) 5
6. © 2021 Thoughtworks 6 Who am I to talk about this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.
7. © 2021 Thoughtworks 7 Who am I to talk about this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.
8. © 2021 Thoughtworks 8 Who am I to talk about this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.
9. © 2021 Thoughtworks 9 Researched type hints to write about them Who am I to talk about type hints? Read lots of PEPs, messages, issue reports; interacted with some key proponents of type hints in Python; studied similar features in other languages—to write than 100 pages and dozens of examples in Fluent Python, Second Edition. Contributed bug fixes improving type hints on typeshed typeshed is the official repository of type hints used by all type checking tools, covering the standard library and other packages. Reorganized the documentation of Python’s typing module Proposed and implemented new overall organization to typing module docs in the Python 3.9 standard library, in collaboration with Guido van Rossum. Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.
10. © 2021 Thoughtworks 10 The State of Static Typing in Python 1. The four modes of typing 2. The central role of typing.Protocol 3. Typing limits and how to address them 4. 10 © 2021 Thoughtworks Agenda
11. © 2021 Thoughtworks 1. The State of Static Typing in Python 11
12. © 2021 Thoughtworks A few type declarations... 12 12 © 2021 Thoughtworks
13. © 2021 Thoughtworks ...and functions with type hints 13 13 © 2021 Thoughtworks
14. © 2021 Thoughtworks Static v. Dynamic Typing 14 14 © 2021 Thoughtworks static typing dynamic typing
15. © 2021 Thoughtworks Static v. Dynamic Typing RUNTIME CHECKING A matter of when 15 15 © 2021 Thoughtworks STATIC CHECKING static typing dynamic typing
16. © 2021 Thoughtworks Gradual typing Not a binary choice, but a continuum 16 16 © 2021 Thoughtworks static typing dynamic typing gradual typing
17. © 2021 Thoughtworks Optional at All Levels Type hints may be omitted and/or type checking disabled on single lines, functions and entire packages. Python Adopted a Gradual Type System 17 The Default type is Any Any is consistent with all other types. It is more general than object, and is understood to implement all interfaces. Does Not Catch Errors at Runtime Type hints are not used at runtime, only by code analysis tools such as linters, IDEs, and dedicated type checkers. Does Not Increase Performance Current Python implementations do not use type hints to optimize bytecode or machine code generation.
18. © 2021 Thoughtworks 18 TypeScript by Microsoft The most successful JavaScript dialect to date. Dart by Google The language of the Flutter SDK. Hack by Facebook PHP dialect supported by the JIT-enabled HHVM runtime. Other Languages with Gradual Typing
19. © 2021 Thoughtworks Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention. 19 Guido van Rossum, Jukka Lehtosalo, Łukasz Langa in PEP 484Type Hints 19 © 2021 Thoughtworks
20. © 2021 Thoughtworks The best feature of gradual typing: type hints are always optional. 20 20 © 2021 Thoughtworks
21. © 2021 Thoughtworks How type hints are used 21 Run time Import time CI pipeline Linting Writing code IDE features • Autocomplete • Instant warnings (wiggly underlines) • Refactoring assistance Stand-alone type checkers (same tools used for linting) Explicit type checking and services • explicit checks with isinstance() or issubclass() • Single dispatch functions • Data validation (e.g. pydantic) • API generation (e.g. FastAPI Stand-alone type checkers • Mypy • Pytype • Pyre • Pyright CLI Data class builders • @dataclasses.dataclass • typing.NamedTuple metaclass
22. © 2021 Thoughtworks Companies with very large Python code bases invested heavily in type hints, and report they were also useful to migrate from Python 2.7. 22 Dropbox Large scale users... Employer of typing PEP authors. Released: Mypy. Facebook Employer of typing PEP authors. Released Pyre. Google Released Pytype. Microsoft Employer of typing PEP authors. Released Pyright. JetBrains Created proprietary type checker embedded in PyCharm IDE. ...and IDE vendors
23. © 2021 Thoughtworks Python Developers Survey, October 2020 23 Source: https://bit.ly/PySurvey2020 Produced by the Python Software Foundation and JetBrains
24. © 2021 Thoughtworks 24 “At least sometimes” 79% use code linters 78% write tests 71% use type hints
25. © 2021 Thoughtworks 2. The four modes of typing 25
26. © 2021 Thoughtworks Duck typing 26
27. © 2021 Thoughtworks Don’t check whether it is-a duck: check whether it quacks-like-a duck, walks-like-a duck, etc, depending on exactly what subsetof duck-like behavior you need... 27 Alex Martelli in comp-lang-python, 20000726 "polymorphism (was Re: Type checking in python?" 27 © 2021 Thoughtworks
28. © 2021 Thoughtworks Typing Map RUNTIME CHECKING 28 28 © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES static typing duck typing
29. © 2021 Thoughtworks Typing Map RUNTIME CHECKING 29 29 © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES “Pyt on st le” “Jav st le” static typing duck typing
30. © 2021 Thoughtworks Typing Map RUNTIME CHECKING 30 30 © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES su ported by AB static typing duck typing goose typing
31. © 2021 Thoughtworks 31
32. © 2021 Thoughtworks Typing Map RUNTIME CHECKING 32 32 © 2021 Thoughtworks static typing duck typing STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES goose typing static duck typing su ported by typing.Proto l su ported by AB
33. © 2021 Thoughtworks Typing Map: languages RUNTIME CHECKING 33 33 © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES Python TypeScript JavaScript Smalltalk Python ≥ 2.6 TypeScript Go Python ≥ 3.8 TypeScript Go Python ≥ 3.5 TypeScript Go Java static typing duck typing goose typing static duck typing
34. © 2021 Thoughtworks 3. The central role of typing.Protocol 34
35. © 2021 Thoughtworks typing.Protocol allows (static) duck typing 35 35 © 2021 Thoughtworks
36. © 2021 Thoughtworks typing.Protocol allows (static) duck typing 36 36 © 2021 Thoughtworks
37. © 2021 Thoughtworks typing.Protocol allows (static) duck typing 37 37 © 2021 Thoughtworks
38. © 2021 Thoughtworks 38
39. © 2021 Thoughtworks 39
40. © 2021 Thoughtworks median_low: fixed code 40 40 © 2021 Thoughtworks etc.
41. © 2021 Thoughtworks Uses of the SupportsLessThan Protocol Stub files for Python 3.9 standard library on typeshed 41 41 builtins: list.sort max min sorted statistics: median_low median_high functools: cmp_to_key bisect: bisect_left bisect_right insort_left insort_right heapq: nlargest nsmallest os.path: commonprefix
42. © 2021 Thoughtworks Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 42 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.
43. © 2021 Thoughtworks Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 43 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.
44. © 2021 Thoughtworks Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 44 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.
45. © 2021 Thoughtworks 4. Typing limits and how to address them 45 45 © 2021 Thoughtworks
46. © 2021 Thoughtworks Type hints are only helpful if they are used to enhance thought, not replace it. 46 Variation on a phrase by Brian Marick, in How to Misuse Code Coverage https://bit.ly/miscodecover 46 © 2021 Thoughtworks
47. © 2021 Thoughtworks Imagine a programming environment that refuses to build a program unless there is 100% test coverage. 47
48. © 2021 Thoughtworks False Positives and False Negatives The static type checking ecosystem—tools and annotations— are still immature and beneath the quality of Python itself. Static Typing is No Silver Bullet 48 No support for common, useful data constraints—even simple ones “Quantity must be int > 0” “Airport code must be str of length 3” “Email address must not be empty str” Type hints are generally unsuitable to check business rules. We need testing and runtime checking to verify business rules and algorithms.
49. © 2021 Thoughtworks 49
50. © 2021 Thoughtworks 50
51. © 2021 Thoughtworks 51
52. © 2021 Thoughtworks Complexity 52 52 © 2021 Thoughtworks
53. © 2021 Thoughtworks A parade of typing-related PEPs 53 PEP  Python Enhancement Proposal 22 and counting... No other aspect of Python has more PEPs.
54. © 2021 Thoughtworks The max() built-in function 54 Flexible and easy to use, but very hard to annotate
55. © 2021 Thoughtworks 55
56. © 2021 Thoughtworks 56 fa neg t ve!
57. © 2021 Thoughtworks max: old type hints 57 57 © 2021 Thoughtworks
58. © 2021 Thoughtworks max: fixed type hints 58 58 © 2021 Thoughtworks
59. © 2021 Thoughtworks max implemented in Python, for testing 59 59 © 2021 Thoughtworks
60. © 2021 Thoughtworks 60 60 26 Lines of code to implement all the documented functionality, with 2 constants, no imports 29 Lines of code for type hints: 7 imports, 4 definitions, and 6 overloaded signatures
61. © 2021 Thoughtworks No support for convenient coding idioms, like unpacking config(**settings) Static type checkers can’t handle the expressive power of Python 61 Limited support for advanced features and abstractions Properties, descriptors, metaclasses, metaprogramming techniques in general Type checkers lag behind False reports and crashes when checking new features, sometimes more than a year after a Python release
62. © 2021 Thoughtworks Good sense 62 62 © 2021 Thoughtworks
63. © 2021 Thoughtworks Any code you can write in Python, you can test in Python. 63 63 © 2021 Thoughtworks
64. © 2021 Thoughtworks If a Python program has adequate unit tests, it can be as robust as a C++, Java, or C# program with adequate unit tests (although the tests in Python will be faster to write). 64 Bruce Eckel, author of Thinking in C, On Java 8, Atomic Scala, and Atomic Kotlin in The Best Of Software Writing anthology by Joel Spolsky (ed.) 64 © 2021 Thoughtworks
65. © 2021 Thoughtworks Compared with unit tests, type hints help detect a different and smaller set of bugs. Static type checking complements but cannot replace automated testing. 65 65 © 2021 Thoughtworks
66. © 2021 Thoughtworks Python is a dynamic language with powerful abstractions and advanced metaprogramming features. That’s why we like it. 66 66 © 2021 Thoughtworks
67. © 2021 Thoughtworks Teams can only avoid the downsides of type hints if they are not mandatory everywhere. 67 67 © 2021 Thoughtworks
68. © 2021 Thoughtworks Embrace Gradual Typing with Mypy Some options help with gradual typing. Others are unhelpful: they force static typing everywhere. 68 Helpful for gradual typing: --disallow-incomplete-defs --check-untyped-defs --warn-unused-ignores Unhelpful for gradual typing: --strict --disallow-any-* 3 options) --disallow-untyped-* 2 options)
69. © 2021 Thoughtworks Static typing is valuable in certain contexts For professional software development teams, the benefits of static typing surpass the cost. But remember that Python is widely used in many other contexts. Main takeaways 69
70. © 2021 Thoughtworks Static typing is valuable in certain contexts For professional software development teams, the benefits of static typing surpass the cost. But remember that Python is widely used in many other contexts. Main takeaways 70 Use Protocols to enable Pythonic APIs Without the support for static duck typing provided by typing.Protocol, widespread use of type hints may lead to inflexible and verbose codebases. Define and use narrow protocols as needed.
71. © 2021 Thoughtworks Static typing is valuable in certain contexts For professional software development teams, the benefits of static typing surpass its significant cost. But remember that Python is widely used in many other contexts. Main takeaways 71 Use Protocols to enable Pythonic APIs Without the support for static duck typing provided by typing.Protocol, widespread use of type hints may lead to inflexible and verbose codebases. Define and use narrow protocols as needed. Don’t make type hints mandatory everywhere Developers should be free to leave portions of the code without type hints, to avoid the downsides and limitations of the static type system and the tools that enforce it.
72. © 2021 Thoughtworks Being optional is not a bug or limitation of Python type hints. It’s the feature that gives us the power to cope with the inherent complexities, annoyances, flaws, and limitations of static types. 72 72 © 2021 Thoughtworks
73. © 2021 Thoughtworks Being optional is not a bug or limitation of Python type hints. It’s the feature that gives us the power to cope with the inherent complexities, annoyances, flaws, and limitations of static types. 73 73 © 2021 Thoughtworks
74. © 2021 Thoughtworks Thank you! Luciano Ramalho Principal Consultant luciano.ramalho@thoughtworks.com Twitter: @ramalhoorg 74