Slide 1

Slide 1 text

© 2021 Thoughtworks Type hints, protocols, and good sense

Slide 2

Slide 2 text

© 2021 Thoughtworks 2

Slide 3

Slide 3 text

© 2021 Thoughtworks Python’s static type hints are ready for use, but they are not perfect. Understand the caveats and workarounds. Avoiding them is the best solution in some cases. 3

Slide 4

Slide 4 text

© 2021 Thoughtworks Fluent Python, First Edition Published in 9 languages: 1. English 2. Portuguese 3. Japanese 4. Korean 5. Simplified Chinese 6. Traditional Chinese 7. French 8. Polish 9. Russian 4

Slide 5

Slide 5 text

© 2021 Thoughtworks Fluent Python, Second Edition ● Covers Python 3.9 and 3.10, including pattern matching ● Extensive coverage of type hints, with many examples ● Updated coverage of async/await, with examples in asyncio, FastAPI and Curio ● Draft available now at OReilly.com: https://bit.ly/FluPy2e 2021 Q4, est.) 5

Slide 6

Slide 6 text

© 2021 Thoughtworks 6 Who am I to talk about this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.

Slide 7

Slide 7 text

© 2021 Thoughtworks 7 Who am I to talk about this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.

Slide 8

Slide 8 text

© 2021 Thoughtworks 8 Who am I to talk about this? Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.

Slide 9

Slide 9 text

© 2021 Thoughtworks 9 Researched type hints to write about them Who am I to talk about type hints? Read lots of PEPs, messages, issue reports; interacted with some key proponents of type hints in Python; studied similar features in other languages—to write than 100 pages and dozens of examples in Fluent Python, Second Edition. Contributed bug fixes improving type hints on typeshed typeshed is the official repository of type hints used by all type checking tools, covering the standard library and other packages. Reorganized the documentation of Python’s typing module Proposed and implemented new overall organization to typing module docs in the Python 3.9 standard library, in collaboration with Guido van Rossum. Using Python professionally since 1998, at first I considered static typing a bad fit for the language. typing.Protocol addressed my main concern. I am now a happy user, with moderation.

Slide 10

Slide 10 text

© 2021 Thoughtworks 10 The State of Static Typing in Python 1. The four modes of typing 2. The central role of typing.Protocol 3. Typing limits and how to address them 4. 10 © 2021 Thoughtworks Agenda

Slide 11

Slide 11 text

© 2021 Thoughtworks 1. The State of Static Typing in Python 11

Slide 12

Slide 12 text

© 2021 Thoughtworks A few type declarations... 12 12 © 2021 Thoughtworks

Slide 13

Slide 13 text

© 2021 Thoughtworks ...and functions with type hints 13 13 © 2021 Thoughtworks

Slide 14

Slide 14 text

© 2021 Thoughtworks Static v. Dynamic Typing 14 14 © 2021 Thoughtworks static typing dynamic typing

Slide 15

Slide 15 text

© 2021 Thoughtworks Static v. Dynamic Typing RUNTIME CHECKING A matter of when 15 15 © 2021 Thoughtworks STATIC CHECKING static typing dynamic typing

Slide 16

Slide 16 text

© 2021 Thoughtworks Gradual typing Not a binary choice, but a continuum 16 16 © 2021 Thoughtworks static typing dynamic typing gradual typing

Slide 17

Slide 17 text

© 2021 Thoughtworks Optional at All Levels Type hints may be omitted and/or type checking disabled on single lines, functions and entire packages. Python Adopted a Gradual Type System 17 The Default type is Any Any is consistent with all other types. It is more general than object, and is understood to implement all interfaces. Does Not Catch Errors at Runtime Type hints are not used at runtime, only by code analysis tools such as linters, IDEs, and dedicated type checkers. Does Not Increase Performance Current Python implementations do not use type hints to optimize bytecode or machine code generation.

Slide 18

Slide 18 text

© 2021 Thoughtworks 18 TypeScript by Microsoft The most successful JavaScript dialect to date. Dart by Google The language of the Flutter SDK. Hack by Facebook PHP dialect supported by the JIT-enabled HHVM runtime. Other Languages with Gradual Typing

Slide 19

Slide 19 text

© 2021 Thoughtworks Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention. 19 Guido van Rossum, Jukka Lehtosalo, Łukasz Langa in PEP 484Type Hints 19 © 2021 Thoughtworks

Slide 20

Slide 20 text

© 2021 Thoughtworks The best feature of gradual typing: type hints are always optional. 20 20 © 2021 Thoughtworks

Slide 21

Slide 21 text

© 2021 Thoughtworks How type hints are used 21 Run time Import time CI pipeline Linting Writing code IDE features ● Autocomplete ● Instant warnings (wiggly underlines) ● Refactoring assistance Stand-alone type checkers (same tools used for linting) Explicit type checking and services ● explicit checks with isinstance() or issubclass() ● Single dispatch functions ● Data validation (e.g. pydantic) ● API generation (e.g. FastAPI Stand-alone type checkers ● Mypy ● Pytype ● Pyre ● Pyright CLI Data class builders ● @dataclasses.dataclass ● typing.NamedTuple metaclass

Slide 22

Slide 22 text

© 2021 Thoughtworks Companies with very large Python code bases invested heavily in type hints, and report they were also useful to migrate from Python 2.7. 22 Dropbox Large scale users... Employer of typing PEP authors. Released: Mypy. Facebook Employer of typing PEP authors. Released Pyre. Google Released Pytype. Microsoft Employer of typing PEP authors. Released Pyright. JetBrains Created proprietary type checker embedded in PyCharm IDE. ...and IDE vendors

Slide 23

Slide 23 text

© 2021 Thoughtworks Python Developers Survey, October 2020 23 Source: https://bit.ly/PySurvey2020 Produced by the Python Software Foundation and JetBrains

Slide 24

Slide 24 text

© 2021 Thoughtworks 24 “At least sometimes” 79% use code linters 78% write tests 71% use type hints

Slide 25

Slide 25 text

© 2021 Thoughtworks 2. The four modes of typing 25

Slide 26

Slide 26 text

© 2021 Thoughtworks Duck typing 26

Slide 27

Slide 27 text

© 2021 Thoughtworks Don’t check whether it is-a duck: check whether it quacks-like-a duck, walks-like-a duck, etc, depending on exactly what subsetof duck-like behavior you need... 27 Alex Martelli in comp-lang-python, 20000726 "polymorphism (was Re: Type checking in python?" 27 © 2021 Thoughtworks

Slide 28

Slide 28 text

© 2021 Thoughtworks Typing Map RUNTIME CHECKING 28 28 © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES static typing duck typing

Slide 29

Slide 29 text

© 2021 Thoughtworks Typing Map RUNTIME CHECKING 29 29 © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES “Pyt on st le” “Jav st le” static typing duck typing

Slide 30

Slide 30 text

© 2021 Thoughtworks Typing Map RUNTIME CHECKING 30 30 © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES su ported by AB static typing duck typing goose typing

Slide 31

Slide 31 text

© 2021 Thoughtworks 31

Slide 32

Slide 32 text

© 2021 Thoughtworks Typing Map RUNTIME CHECKING 32 32 © 2021 Thoughtworks static typing duck typing STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES goose typing static duck typing su ported by typing.Proto l su ported by AB

Slide 33

Slide 33 text

© 2021 Thoughtworks Typing Map: languages RUNTIME CHECKING 33 33 © 2021 Thoughtworks STATIC CHECKING STRUCTURAL TYPES NOMINAL TYPES Python TypeScript JavaScript Smalltalk Python ≥ 2.6 TypeScript Go Python ≥ 3.8 TypeScript Go Python ≥ 3.5 TypeScript Go Java static typing duck typing goose typing static duck typing

Slide 34

Slide 34 text

© 2021 Thoughtworks 3. The central role of typing.Protocol 34

Slide 35

Slide 35 text

© 2021 Thoughtworks typing.Protocol allows (static) duck typing 35 35 © 2021 Thoughtworks

Slide 36

Slide 36 text

© 2021 Thoughtworks typing.Protocol allows (static) duck typing 36 36 © 2021 Thoughtworks

Slide 37

Slide 37 text

© 2021 Thoughtworks typing.Protocol allows (static) duck typing 37 37 © 2021 Thoughtworks

Slide 38

Slide 38 text

© 2021 Thoughtworks 38

Slide 39

Slide 39 text

© 2021 Thoughtworks 39

Slide 40

Slide 40 text

© 2021 Thoughtworks median_low: fixed code 40 40 © 2021 Thoughtworks etc.

Slide 41

Slide 41 text

© 2021 Thoughtworks Uses of the SupportsLessThan Protocol Stub files for Python 3.9 standard library on typeshed 41 41 builtins: list.sort max min sorted statistics: median_low median_high functools: cmp_to_key bisect: bisect_left bisect_right insort_left insort_right heapq: nlargest nsmallest os.path: commonprefix

Slide 42

Slide 42 text

© 2021 Thoughtworks Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 42 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.

Slide 43

Slide 43 text

© 2021 Thoughtworks Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 43 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.

Slide 44

Slide 44 text

© 2021 Thoughtworks Support duck typing with type hints The essence of Python’s Data Model and standard library Use typing.Protocol to build Pythonic APIs 44 Follow the Interface Segregation Principle Client code should not be forced to depend on methods it does not use Prefer narrow protocols Single method protocols should be the most common. Sometimes, two methods. Rarely more.

Slide 45

Slide 45 text

© 2021 Thoughtworks 4. Typing limits and how to address them 45 45 © 2021 Thoughtworks

Slide 46

Slide 46 text

© 2021 Thoughtworks Type hints are only helpful if they are used to enhance thought, not replace it. 46 Variation on a phrase by Brian Marick, in How to Misuse Code Coverage https://bit.ly/miscodecover 46 © 2021 Thoughtworks

Slide 47

Slide 47 text

© 2021 Thoughtworks Imagine a programming environment that refuses to build a program unless there is 100% test coverage. 47

Slide 48

Slide 48 text

© 2021 Thoughtworks False Positives and False Negatives The static type checking ecosystem—tools and annotations— are still immature and beneath the quality of Python itself. Static Typing is No Silver Bullet 48 No support for common, useful data constraints—even simple ones “Quantity must be int > 0” “Airport code must be str of length 3” “Email address must not be empty str” Type hints are generally unsuitable to check business rules. We need testing and runtime checking to verify business rules and algorithms.

Slide 49

Slide 49 text

© 2021 Thoughtworks 49

Slide 50

Slide 50 text

© 2021 Thoughtworks 50

Slide 51

Slide 51 text

© 2021 Thoughtworks 51

Slide 52

Slide 52 text

© 2021 Thoughtworks Complexity 52 52 © 2021 Thoughtworks

Slide 53

Slide 53 text

© 2021 Thoughtworks A parade of typing-related PEPs 53 PEP  Python Enhancement Proposal 22 and counting... No other aspect of Python has more PEPs.

Slide 54

Slide 54 text

© 2021 Thoughtworks The max() built-in function 54 Flexible and easy to use, but very hard to annotate

Slide 55

Slide 55 text

© 2021 Thoughtworks 55

Slide 56

Slide 56 text

© 2021 Thoughtworks 56 fa neg t ve!

Slide 57

Slide 57 text

© 2021 Thoughtworks max: old type hints 57 57 © 2021 Thoughtworks

Slide 58

Slide 58 text

© 2021 Thoughtworks max: fixed type hints 58 58 © 2021 Thoughtworks

Slide 59

Slide 59 text

© 2021 Thoughtworks max implemented in Python, for testing 59 59 © 2021 Thoughtworks

Slide 60

Slide 60 text

© 2021 Thoughtworks 60 60 26 Lines of code to implement all the documented functionality, with 2 constants, no imports 29 Lines of code for type hints: 7 imports, 4 definitions, and 6 overloaded signatures

Slide 61

Slide 61 text

© 2021 Thoughtworks No support for convenient coding idioms, like unpacking config(**settings) Static type checkers can’t handle the expressive power of Python 61 Limited support for advanced features and abstractions Properties, descriptors, metaclasses, metaprogramming techniques in general Type checkers lag behind False reports and crashes when checking new features, sometimes more than a year after a Python release

Slide 62

Slide 62 text

© 2021 Thoughtworks Good sense 62 62 © 2021 Thoughtworks

Slide 63

Slide 63 text

© 2021 Thoughtworks Any code you can write in Python, you can test in Python. 63 63 © 2021 Thoughtworks

Slide 64

Slide 64 text

© 2021 Thoughtworks If a Python program has adequate unit tests, it can be as robust as a C++, Java, or C# program with adequate unit tests (although the tests in Python will be faster to write). 64 Bruce Eckel, author of Thinking in C, On Java 8, Atomic Scala, and Atomic Kotlin in The Best Of Software Writing anthology by Joel Spolsky (ed.) 64 © 2021 Thoughtworks

Slide 65

Slide 65 text

© 2021 Thoughtworks Compared with unit tests, type hints help detect a different and smaller set of bugs. Static type checking complements but cannot replace automated testing. 65 65 © 2021 Thoughtworks

Slide 66

Slide 66 text

© 2021 Thoughtworks Python is a dynamic language with powerful abstractions and advanced metaprogramming features. That’s why we like it. 66 66 © 2021 Thoughtworks

Slide 67

Slide 67 text

© 2021 Thoughtworks Teams can only avoid the downsides of type hints if they are not mandatory everywhere. 67 67 © 2021 Thoughtworks

Slide 68

Slide 68 text

© 2021 Thoughtworks Embrace Gradual Typing with Mypy Some options help with gradual typing. Others are unhelpful: they force static typing everywhere. 68 Helpful for gradual typing: --disallow-incomplete-defs --check-untyped-defs --warn-unused-ignores Unhelpful for gradual typing: --strict --disallow-any-* 3 options) --disallow-untyped-* 2 options)

Slide 69

Slide 69 text

© 2021 Thoughtworks Static typing is valuable in certain contexts For professional software development teams, the benefits of static typing surpass the cost. But remember that Python is widely used in many other contexts. Main takeaways 69

Slide 70

Slide 70 text

© 2021 Thoughtworks Static typing is valuable in certain contexts For professional software development teams, the benefits of static typing surpass the cost. But remember that Python is widely used in many other contexts. Main takeaways 70 Use Protocols to enable Pythonic APIs Without the support for static duck typing provided by typing.Protocol, widespread use of type hints may lead to inflexible and verbose codebases. Define and use narrow protocols as needed.

Slide 71

Slide 71 text

© 2021 Thoughtworks Static typing is valuable in certain contexts For professional software development teams, the benefits of static typing surpass its significant cost. But remember that Python is widely used in many other contexts. Main takeaways 71 Use Protocols to enable Pythonic APIs Without the support for static duck typing provided by typing.Protocol, widespread use of type hints may lead to inflexible and verbose codebases. Define and use narrow protocols as needed. Don’t make type hints mandatory everywhere Developers should be free to leave portions of the code without type hints, to avoid the downsides and limitations of the static type system and the tools that enforce it.

Slide 72

Slide 72 text

© 2021 Thoughtworks Being optional is not a bug or limitation of Python type hints. It’s the feature that gives us the power to cope with the inherent complexities, annoyances, flaws, and limitations of static types. 72 72 © 2021 Thoughtworks

Slide 73

Slide 73 text

© 2021 Thoughtworks Being optional is not a bug or limitation of Python type hints. It’s the feature that gives us the power to cope with the inherent complexities, annoyances, flaws, and limitations of static types. 73 73 © 2021 Thoughtworks

Slide 74

Slide 74 text

© 2021 Thoughtworks Thank you! Luciano Ramalho Principal Consultant [email protected] Twitter: @ramalhoorg 74