Type hints, protocols, and good sense

© 2021 Thoughtworks
Type hints,
protocols,
and good sense

View Slide

© 2021 Thoughtworks 2

View Slide

Python’s static type hints are ready for use,
but they are not perfect.
Understand the caveats and workarounds.
Avoiding them is the best solution
in some cases.
3

View Slide

Fluent Python,
First Edition
Published in 9 languages:
1. English
2. Portuguese
3. Japanese
4. Korean
5. Simpliﬁed Chinese
6. Traditional Chinese
7. French
8. Polish
9. Russian
4

View Slide

Fluent Python,
Second Edition
● Covers Python 3.9 and 3.10,
including pattern matching
● Extensive coverage of type hints,
with many examples
● Updated coverage of async/await,
with examples in asyncio, FastAPI
and Curio
● Draft available now at
OReilly.com:
https://bit.ly/FluPy2e
2021 Q4, est.)
5

View Slide

Who am I to talk
about this?
Using Python professionally since
1998, at first I considered static typing
a bad fit for the language.
typing.Protocol addressed my
main concern.
I am now a happy user,
with moderation.

View Slide

Who am I to talk
about this?
main concern.
with moderation.

View Slide

Researched type hints
to write about them
Who am I to talk
about type hints?
Read lots of PEPs, messages, issue
reports; interacted with some key
proponents of type hints in Python;
studied similar features in other
languages—to write than 100 pages
and dozens of examples in Fluent
Python, Second Edition.
Contributed bug fixes
improving type hints on
typeshed
typeshed is the official repository
of type hints used by all type
checking tools, covering the
standard library and other
packages.
Reorganized the
documentation of Python’s
typing module
Proposed and implemented new
overall organization to typing
module docs in the Python 3.9
standard library, in collaboration
with Guido van Rossum.
main concern.
with moderation.

View Slide

The State of
Static Typing
in Python
1.
The four modes
of typing
2.
The central role of
typing.Protocol
3.
Typing limits
and how to
address them
4.
10
Agenda

View Slide

1.
The State of Static Typing
in Python
11

View Slide

A few type declarations...
12
12

View Slide

...and functions with type hints
13
13

View Slide

Static v. Dynamic Typing
14
14
static
typing
dynamic
typing

View Slide

Static v. Dynamic Typing
RUNTIME
CHECKING
A matter of when
15
15
STATIC
CHECKING
static
typing
dynamic
typing

View Slide

Gradual typing
Not a binary choice, but a continuum
16
16
static
typing
dynamic
typing
gradual
typing

View Slide

Optional at All Levels
Type hints may be omitted and/or
type checking disabled on single lines,
functions and entire packages.
Python Adopted a
Gradual Type System
17
The Default type is Any
Any is consistent with all other types.
It is more general than object, and
is understood to implement all interfaces.
Does Not Catch Errors at Runtime
Type hints are not used at runtime,
only by code analysis tools such as
linters, IDEs, and dedicated type
checkers.
Does Not Increase Performance
Current Python implementations
do not use type hints to optimize
bytecode or machine code generation.

View Slide

TypeScript
by Microsoft
The most successful
JavaScript dialect to date.
Dart
by Google
The language of the
Flutter SDK.
Hack
by Facebook
PHP dialect supported by the
JIT-enabled HHVM runtime.
Other Languages with Gradual Typing

View Slide

Python will remain a
dynamically typed
language, and the
authors have no desire
to ever make type
hints mandatory,
even by convention.
19
Guido van Rossum, Jukka Lehtosalo, Łukasz Langa
in PEP 484Type Hints
19

View Slide

The best feature of
gradual typing:
type hints are always
optional.
20
20

View Slide

How type hints are used
21
Run time
Import time
CI pipeline
Linting
Writing code
IDE features
● Autocomplete
● Instant warnings (wiggly underlines)
● Refactoring assistance
Stand-alone type checkers
(same tools used for linting)
Explicit type checking and services
● explicit checks with
isinstance() or issubclass()
● Single dispatch functions
● Data validation (e.g. pydantic)
● API generation (e.g. FastAPI
Stand-alone type checkers
● Mypy
● Pytype
● Pyre
● Pyright CLI
Data class builders
● @dataclasses.dataclass
● typing.NamedTuple metaclass

View Slide

Companies with very large Python code bases
invested heavily in type hints, and report they
were also useful to migrate from Python 2.7.
22
Dropbox
Large scale users...
Employer of typing
PEP authors.
Released: Mypy.
Facebook
Employer of typing
PEP authors.
Released Pyre.
Google
Released Pytype.
Microsoft
Employer of typing
PEP authors.
Released Pyright.
JetBrains
Created proprietary
type checker
embedded in
PyCharm IDE.
...and IDE vendors

View Slide

Python Developers
Survey, October 2020
23
Source: https://bit.ly/PySurvey2020
Produced by the
Python Software Foundation
and JetBrains

View Slide

“At least sometimes”
79% use code linters
78% write tests
71% use type hints

View Slide

2.
The four modes
of typing
25

View Slide

Duck typing
26

View Slide

Don’t check whether it
is-a duck:
check whether it
quacks-like-a duck,
walks-like-a duck, etc,
depending on exactly
what subsetof
duck-like behavior you
need...
27
Alex Martelli in comp-lang-python, 20000726
"polymorphism (was Re: Type checking in python?"
27

View Slide

Typing Map
RUNTIME
CHECKING
28
28
STATIC
CHECKING
STRUCTURAL TYPES
NOMINAL TYPES
static
typing
duck
typing

View Slide

Typing Map
RUNTIME
CHECKING
29
29
STATIC
CHECKING
STRUCTURAL TYPES
NOMINAL TYPES
“Pyt on
st le”
“Jav
st le”
static
typing
duck
typing

View Slide

Typing Map
RUNTIME
CHECKING
30
30
STATIC
CHECKING
STRUCTURAL TYPES
NOMINAL TYPES
su ported
by AB
static
typing
duck
typing
goose
typing

View Slide


View Slide

Typing Map
RUNTIME
CHECKING
32
32
static
typing
duck
typing
STATIC
CHECKING
STRUCTURAL TYPES
NOMINAL TYPES
goose
typing
static
duck typing
su ported by
typing.Proto l
su ported
by AB

View Slide

Typing Map: languages
RUNTIME
CHECKING
33
33
STATIC
CHECKING
STRUCTURAL TYPES
NOMINAL TYPES
Python
TypeScript
JavaScript
Smalltalk
Python ≥ 2.6
TypeScript
Go
Python ≥ 3.8
TypeScript
Go
Python ≥ 3.5
TypeScript
Go
Java
static
typing
duck
typing
goose
typing
static
duck typing

View Slide

3.
The central role of
typing.Protocol
34

View Slide

typing.Protocol allows (static) duck typing
35
35

View Slide

36
36

View Slide

37
37

View Slide


View Slide

median_low: fixed code
40
40
etc.

View Slide

Uses of the SupportsLessThan Protocol
Stub files for Python 3.9 standard library on typeshed
41
41
builtins: list.sort
max
min
sorted
statistics: median_low
median_high
functools: cmp_to_key
bisect: bisect_left
bisect_right
insort_left
insort_right
heapq: nlargest
nsmallest
os.path: commonprefix

View Slide

Support duck typing with type hints
The essence of Python’s
Data Model and standard library
Use typing.Protocol
to build Pythonic APIs
42
Follow the
Interface Segregation Principle
Client code should not be forced to
depend on methods it does not use
Prefer narrow protocols
Single method protocols should be the
most common. Sometimes, two
methods. Rarely more.

View Slide

Use typing.Protocol
43
Follow the

View Slide

Use typing.Protocol
44
Follow the

View Slide

4.
Typing limits and
how to address them
45
45

View Slide

Type hints are only
helpful if they are used
to enhance thought,
not replace it.
46
Variation on a phrase by Brian Marick, in
How to Misuse Code Coverage
https://bit.ly/miscodecover 46

View Slide

Imagine a programming environment
that refuses to build a program
unless there is 100% test coverage.
47

View Slide

False Positives and False Negatives
The static type checking
ecosystem—tools and annotations—
are still immature and beneath the
quality of Python itself.
Static Typing is
No Silver Bullet
48
No support for common, useful data
constraints—even simple ones
“Quantity must be int > 0”
“Airport code must be str of length 3”
“Email address must not be empty str”
Type hints are generally unsuitable to
check business rules.
We need testing and runtime checking
to verify business rules and algorithms.

View Slide


View Slide

Complexity
52
52

View Slide

A parade of typing-related PEPs
53
PEP  Python Enhancement Proposal
22 and counting...
No other aspect of
Python has more PEPs.

View Slide

The max() built-in function
54
Flexible and easy to use, but very hard to annotate

View Slide


View Slide

fa
neg t ve!

View Slide

max: old type hints
57
57

View Slide

max: fixed type hints
58
58

View Slide

max
implemented in
Python, for testing
59
59

View Slide

60
26
Lines of code to implement all
the documented functionality,
with 2 constants, no imports
29
Lines of code for type hints:
7 imports, 4 definitions, and
6 overloaded signatures

View Slide

No support for convenient
coding idioms, like unpacking
config(**settings)
Static type checkers
can’t handle the
expressive power
of Python
61
Limited support for advanced
features and abstractions
Properties, descriptors,
metaclasses, metaprogramming
techniques in general
Type checkers lag behind
False reports and crashes when
checking new features, sometimes
more than a year after a Python
release

View Slide

Good sense
62
62

View Slide

Any code you can write
in Python, you can test
in Python.
63
63

View Slide

If a Python program
has adequate unit
tests, it can be as
robust as a C++, Java,
or C# program with
adequate unit tests
(although the tests in
Python will be faster
to write).
64
Bruce Eckel, author of Thinking in C, On Java 8,
Atomic Scala, and Atomic Kotlin in The Best Of
Software Writing anthology by Joel Spolsky (ed.) 64

View Slide

Compared with unit
tests, type hints help
detect a different and
smaller set of bugs.
Static type checking
complements but
cannot replace
automated testing.
65
65

View Slide

Python is a dynamic
language with
powerful abstractions
and advanced
metaprogramming
features.
That’s why we like it.
66
66

View Slide

Teams can only avoid
the downsides of type
hints if they are not
mandatory everywhere.
67
67

View Slide

Embrace Gradual
Typing with Mypy
Some options help with gradual typing.
Others are unhelpful:
they force static typing everywhere.
68
Helpful for gradual typing:
--disallow-incomplete-defs
--check-untyped-defs
--warn-unused-ignores
Unhelpful for gradual typing:
--strict
--disallow-any-* 3 options)
--disallow-untyped-* 2 options)

View Slide

Static typing is valuable in certain contexts
For professional software development teams,
the benefits of static typing surpass the cost.
But remember that Python is widely used in
many other contexts.
Main takeaways
69

View Slide

the benefits of static typing surpass the cost.
But remember that Python is widely used in
many other contexts.
Main takeaways
70
Use Protocols to enable Pythonic APIs
Without the support for static duck typing
provided by typing.Protocol, widespread
use of type hints may lead to inflexible and
verbose codebases.
Define and use narrow protocols as needed.

View Slide

the benefits of static typing surpass its
significant cost. But remember that Python is
widely used in many other contexts.
Main takeaways
71
Use Protocols to enable Pythonic APIs
Without the support for static duck typing
provided by typing.Protocol, widespread
use of type hints may lead to inflexible and
verbose codebases.
Define and use narrow protocols as needed.
Don’t make type hints mandatory everywhere
Developers should be free to leave portions of
the code without type hints, to avoid the
downsides and limitations of the static type
system and the tools that enforce it.

View Slide

Being optional is not a
bug or limitation of
Python type hints.
It’s the feature that
gives us the power
to cope with the
inherent complexities,
annoyances, ﬂaws,
and limitations of
static types.
72
72

View Slide

Being optional is not a
bug or limitation of
Python type hints.
It’s the feature that
gives us the power
to cope with the
inherent complexities,
annoyances, ﬂaws,
and limitations of
static types.
73
73

View Slide

Thank you!
Luciano Ramalho
Principal Consultant
[email protected]
Twitter: @ramalhoorg
74

View Slide

Type hints, protocols, and good sense

Type hints, protocols, and good sense

Luciano Ramalho

More Decks by Luciano Ramalho

Other Decks in Programming

Featured

Transcript