Type hints, protocols, and good sense

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

© 2021 Thoughtworks Fluent Python, Second Edition ● Covers Python 3.9 and 3.10, including pattern matching ● Extensive coverage of type hints, with many examples ● Updated coverage of async/await, with examples in asyncio, FastAPI and Curio ● Draft available now at OReilly.com: https://bit.ly/FluPy2e 2021 Q4, est.) 5

Slide 6

Slide 6 text

© 2021 Thoughtworks 6 Who am I to talk about this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.

Slide 7

Slide 7 text

© 2021 Thoughtworks 7 Who am I to talk about this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.

Slide 8

Slide 8 text

© 2021 Thoughtworks 8 Who am I to talk about this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.

Slide 9

Slide 9 text

© 2021 Thoughtworks 9 Researched type hints to write about them Who am I to talk about type hints? Read lots of PEPs, messages, issue reports; interacted with some key proponents of type hints in Python; studied similar features in other languages—to write than 100 pages and dozens of examples in Fluent Python, Second Edition. Contributed bug fixes improving type hints on typeshed typeshed is the official repository of type hints used by all type checking tools, covering the standard library and other packages. Reorganized the documentation of Python’s typing module Proposed and implemented new overall organization to typing module docs in the Python 3.9 standard library, in collaboration with Guido van Rossum. Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

© 2021 Thoughtworks Optional at All Levels Type hints may be omitted and/or type checking disabled on single lines, functions and entire packages. Python Adopted a Gradual Type System 17 The Default type is Any Any is consistent with all other types. It is more general than object, and is understood to implement all interfaces. Does Not Catch Errors at Runtime Type hints are not used at runtime, only by code analysis tools such as linters, IDEs, and dedicated type checkers. Does Not Increase Performance Current Python implementations do not use type hints to optimize bytecode or machine code generation.

Slide 18

Slide 18 text

© 2021 Thoughtworks 18 TypeScript by Microsoft The most successful JavaScript dialect to date. Dart by Google The language of the Flutter SDK. Hack by Facebook PHP dialect supported by the JIT-enabled HHVM runtime. Other Languages with Gradual Typing

Slide 19

Slide 19 text

© 2021 Thoughtworks Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention. 19 Guido van Rossum, Jukka Lehtosalo, Łukasz Langa in PEP 484Type Hints 19 © 2021 Thoughtworks

Slide 20

Slide 20 text

Slide 21

Slide 21 text

© 2021 Thoughtworks How type hints are used 21 Run time Import time CI pipeline Linting Writing code IDE features ● Autocomplete ● Instant warnings (wiggly underlines) ● Refactoring assistance Stand-alone type checkers (same tools used for linting) Explicit type checking and services ● explicit checks with isinstance() or issubclass() ● Single dispatch functions ● Data validation (e.g. pydantic) ● API generation (e.g. FastAPI Stand-alone type checkers ● Mypy ● Pytype ● Pyre ● Pyright CLI Data class builders ● @dataclasses.dataclass ● typing.NamedTuple metaclass

Slide 22

Slide 22 text

© 2021 Thoughtworks Companies with very large Python code bases invested heavily in type hints, and report they were also useful to migrate from Python 2.7. 22 Dropbox Large scale users... Employer of typing PEP authors. Released: Mypy. Facebook Employer of typing PEP authors. Released Pyre. Google Released Pytype. Microsoft Employer of typing PEP authors. Released Pyright. JetBrains Created proprietary type checker embedded in PyCharm IDE. ...and IDE vendors

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

© 2021 Thoughtworks Don’t check whether it is-a duck: check whether it quacks-like-a duck, walks-like-a duck, etc, depending on exactly what subsetof duck-like behavior you need... 27 Alex Martelli in comp-lang-python, 20000726 "polymorphism (was Re: Type checking in python?" 27 © 2021 Thoughtworks

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

© 2021 Thoughtworks Typing Map RUNTIME CHECKING 32 32 © 2021 Thoughtworks static typing duck typing STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES goose typing static duck typing su ported by typing.Proto l su ported by AB

Slide 33

Slide 33 text

© 2021 Thoughtworks Typing Map: languages RUNTIME CHECKING 33 33 © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES Python TypeScript JavaScript Smalltalk Python ≥ 2.6 TypeScript Go Python ≥ 3.8 TypeScript Go Python ≥ 3.5 TypeScript Go Java static typing duck typing goose typing static duck typing

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

© 2021 Thoughtworks Uses of the SupportsLessThan Protocol Stub files for Python 3.9 standard library on typeshed 41 41 builtins: list.sort max min sorted statistics: median_low median_high functools: cmp_to_key bisect: bisect_left bisect_right insort_left insort_right heapq: nlargest nsmallest os.path: commonprefix

Slide 42

Slide 42 text

© 2021 Thoughtworks Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 42 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.

Slide 43

Slide 43 text

© 2021 Thoughtworks Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 43 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.

Slide 44

Slide 44 text

© 2021 Thoughtworks Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 44 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.

Slide 45

Slide 45 text

Slide 46

Slide 46 text

© 2021 Thoughtworks Type hints are only helpful if they are used to enhance thought, not replace it. 46 Variation on a phrase by Brian Marick, in How to Misuse Code Coverage https://bit.ly/miscodecover 46 © 2021 Thoughtworks

Slide 47

Slide 47 text

Slide 48

Slide 48 text

© 2021 Thoughtworks False Positives and False Negatives The static type checking ecosystem—tools and annotations— are still immature and beneath the quality of Python itself. Static Typing is No Silver Bullet 48 No support for common, useful data constraints—even simple ones “Quantity must be int > 0” “Airport code must be str of length 3” “Email address must not be empty str” Type hints are generally unsuitable to check business rules. We need testing and runtime checking to verify business rules and algorithms.

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Slide 60

Slide 60 text

© 2021 Thoughtworks 60 60 26 Lines of code to implement all the documented functionality, with 2 constants, no imports 29 Lines of code for type hints: 7 imports, 4 definitions, and 6 overloaded signatures

Slide 61

Slide 61 text

© 2021 Thoughtworks No support for convenient coding idioms, like unpacking config(**settings) Static type checkers can’t handle the expressive power of Python 61 Limited support for advanced features and abstractions Properties, descriptors, metaclasses, metaprogramming techniques in general Type checkers lag behind False reports and crashes when checking new features, sometimes more than a year after a Python release

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Slide 64

Slide 64 text

© 2021 Thoughtworks If a Python program has adequate unit tests, it can be as robust as a C++, Java, or C# program with adequate unit tests (although the tests in Python will be faster to write). 64 Bruce Eckel, author of Thinking in C, On Java 8, Atomic Scala, and Atomic Kotlin in The Best Of Software Writing anthology by Joel Spolsky (ed.) 64 © 2021 Thoughtworks

Slide 65

Slide 65 text

© 2021 Thoughtworks Compared with unit tests, type hints help detect a different and smaller set of bugs. Static type checking complements but cannot replace automated testing. 65 65 © 2021 Thoughtworks

Slide 66

Slide 66 text

Slide 67

Slide 67 text

Slide 68

Slide 68 text

© 2021 Thoughtworks Embrace Gradual Typing with Mypy Some options help with gradual typing. Others are unhelpful: they force static typing everywhere. 68 Helpful for gradual typing: --disallow-incomplete-defs --check-untyped-defs --warn-unused-ignores Unhelpful for gradual typing: --strict --disallow-any-* 3 options) --disallow-untyped-* 2 options)

Slide 69

Slide 69 text

Slide 70

Slide 70 text

© 2021 Thoughtworks Static typing is valuable in certain contexts For professional software development teams, the benefits of static typing surpass the cost. But remember that Python is widely used in many other contexts. Main takeaways 70 Use Protocols to enable Pythonic APIs Without the support for static duck typing provided by typing.Protocol, widespread use of type hints may lead to inflexible and verbose codebases. Define and use narrow protocols as needed.

Slide 71

Slide 71 text

© 2021 Thoughtworks Static typing is valuable in certain contexts For professional software development teams, the benefits of static typing surpass its significant cost. But remember that Python is widely used in many other contexts. Main takeaways 71 Use Protocols to enable Pythonic APIs Without the support for static duck typing provided by typing.Protocol, widespread use of type hints may lead to inflexible and verbose codebases. Define and use narrow protocols as needed. Don’t make type hints mandatory everywhere Developers should be free to leave portions of the code without type hints, to avoid the downsides and limitations of the static type system and the tools that enforce it.

Slide 72

Slide 72 text

© 2021 Thoughtworks Being optional is not a bug or limitation of Python type hints. It’s the feature that gives us the power to cope with the inherent complexities, annoyances, ﬂaws, and limitations of static types. 72 72 © 2021 Thoughtworks

Slide 73

Slide 73 text

Slide 74

Slide 74 text