Carl Meyer - Type-checked Python in the real world

You've heard about Python type annotations, but wondered if they're useful in the real world? Worried you've got too much code and can't afford to annotate it? Type-checked Python is here, it's for real, and it can help you catch bugs and make your code easier to understand. Come learn from our experience gradually typing a million-LOC production Python application!

Type checking solves real world problems in production Python systems. We'll cover the benefits, how type checking in Python works, how to introduce it gradually and sustainably in a production Python application, and how to measure success and avoid common pitfalls. We'll even demonstrate how modern Python typechecking goes hand-in-hand with duck-typing! Join us for a deep dive into type-checked Python in the real world.

https://us.pycon.org/2018/schedule/presentation/102/

Transcript

1. Type-Checked Python INSTAGRAM Carl Meyer
3. @carljm
4. @carljm
5. @carljm
6. The Plan WHY type 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4
7. add annotations to your Python? WHY
8. "Items"? def process(self, items): for item in items: self.append(item.value.id)
9. "Items"? def process(self, items): for item in items: self.append(item.value.id)
10. "Items"? def process(self, items): for item in items: self.append(item.value.id)
11. "Items"? def process(self, items): for item in items: self.append(item.value.id)
12. "Items"? def process(self, items): for item in items: self.append(item.value.id)
13. "Items"? from typing import Sequence from .models import Item def process(self, items: Sequence[Item]) -> None: for item in items: self.append(item.value.id)
14. "Items"? from typing import Sequence from .models import Item def process(self, items: Sequence[Item]) -> None: for item in items: self.append(item.value.id)
15. "That's cool, but I don't need it; 
 I'd catch that with a test!" — PYTHONISTA
16. One Test Case
17. More Test Cases
18. Parametrized Test Case
19. Property-based Test
20. Strings
21. Lists
22. Dictionaries
24. Why type The Plan 1 How to even type 2 Gradual typing of existing code 3 Issues you may run into 4 WHY type The Plan 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4
25. HOW to even type
26. square.py def square(x: int) -> int: return x**2
27. square.py def square(x: int) -> int: return x**2
28. square.py def square(x: int) -> int: return x**2 square(3) square('foo') square(4) + 'foo'
29. Running Mypy $ pip install mypy $ mypy square.py square.py:5: error: Argument 1 to "square" has incompatible type "str"; expected "int" square.py:6: error: Unsupported operand types for + ("int" and "str")
30. square.py def square(x: int) -> int: return x**2 square(3) square('foo') square(4) + 'foo' Argument 1 to "square" has incompatible type "str"; expected "int"
31. square.py def square(x: int) -> int: return x**2 square(3) square('foo') square(4) + 'foo' Unsupported operand types for + ("int" and "str")
32. Type Inference from typing import Tuple class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height def get_dimensions(self) -> Tuple[str, str]: return (self.width, self.height)
33. Type Inference Incompatible return value type (got "Tuple[int, int]", expected "Tuple[str, str]") from typing import Tuple class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height def get_dimensions(self) -> Tuple[str, str]: return (self.width, self.height)
34. Type Inference photos = [ Photo(640, 480), Photo(1024, 768), ] photos.append('foo')
35. Type Inference photos = [ Photo(640, 480), Photo(1024, 768), ] photos.append('foo') Argument 1 to "append" of "list" has incompatible type "str"; expected "Photo"
36. Variable Annotation class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height self.tags = []
37. Variable Annotation class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height self.tags = [] Need type annotation for variable
38. Variable Annotation from typing import List class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height self.tags: List[str] = []
39. Review
40. Annotate Function Signatures ARGUMENTS AND RETURN VALUES Annotate Variables ONLY IF YOU HAVE TO
41. Union from typing import Union def get_foo_or_bar(id: int) -> Union[Foo, Bar]: ... def get_foo_or_none(id: int) -> Union[Foo, None]: ...
42. Union from typing import Union, Optional def get_foo_or_bar(id: int) -> Union[Foo, Bar]: ... def get_foo_or_none(id: int) -> Union[Foo, None]: ... def get_foo_or_none(id: int) -> Optional[Foo]: ...
43. Optional from typing import Optional def get_foo(foo_id: Optional[int]) -> Optional[Foo]: if foo_id is None: return None return Foo(foo_id) my_foo = get_foo(3) my_foo.id # error: NoneType has no attribute 'id'
44. Function Overloads from typing import Optional, overload @overload def get_foo(foo_id: None) -> None: pass @overload def get_foo(foo_id: int) -> Foo: pass def get_foo(foo_id: Optional[int]) -> Optional[Foo]: if foo_id is None: return None return Foo(foo_id) reveal_type(get_foo(None)) # None reveal_type(get_foo(1)) # Foo
45. Function Overloads from typing import Optional, overload @overload def get_foo(foo_id: None) -> None: pass @overload def get_foo(foo_id: int) -> Foo: pass def get_foo(foo_id: Optional[int]) -> Optional[Foo]: if foo_id is None: return None return Foo(foo_id) reveal_type(get_foo(None)) # None reveal_type(get_foo(1)) # Foo
46. Generics numbers: List[int] = [] books_by_id: Dict[int, Book] = {} int_tree: TreeNode[int] = TreeNode(3) str_tree: TreeNode[str] = TreeNode('foo') book_tree: TreeNode[Book] = TreeNode(mybook)
47. Generics from typing import Generic, TypeVar TNodeValue = TypeVar('TNodeValue') class TreeNode(Generic[TNodeValue]): def __init__(self, value: TNodeValue) -> None: self.value = value node = TreeNode(3) reveal_type(node) # TreeNode[int] reveal_type(node.value) # int
48. Generics from typing import Generic, TypeVar TNodeValue = TypeVar('TNodeValue') class TreeNode(Generic[TNodeValue]): def __init__(self, value: TNodeValue) -> None: self.value = value node = TreeNode(3) reveal_type(node) # TreeNode[int] reveal_type(node.value) # int
49. Generic Functions from typing import TypeVar AnyStr = TypeVar('AnyStr', str, bytes) def concat(a: AnyStr, b: AnyStr) -> AnyStr: return a + b concat('foo', b'bar') # typecheck error! concat(3, 6) # typecheck error! reveal_type(concat('foo', 'bar')) # str reveal_type(concat(b'foo', b'bar')) # bytes
50. Generic Functions from typing import AnyStr def concat(a: AnyStr, b: AnyStr) -> AnyStr: return a + b concat('foo', b'bar') # typecheck error! concat(3, 6) # typecheck error! reveal_type(concat('foo', 'bar')) # str reveal_type(concat(b'foo', b'bar')) # bytes
51. Review
52. Annotate Function Signatures ARGUMENTS AND RETURN VALUES Annotate Variables ONLY IF YOU HAVE TO Unions and Optionals USE SPARINGLY Overloads and Generics TEACH THE TYPE CHECKER TO BE SMARTER
53. Where's my duck?
54. def render(obj): return obj.render() Where's my duck?
55. def render(obj: object) -> str: return obj.render() "object" has no attribute "render" Where's my duck?
56. from typing import Any def render(obj: Any) -> str: return obj.render() render(3) # typechecks, but fails at runtime Where's my duck?
57. from typing_extensions import Protocol class Renderable(Protocol): def render(self) -> str: ... def render(obj: Renderable) -> str: return obj.render() class Foo: def render(self) -> str: return "Foo!" render(Foo()) # clean! render(3) # error: expected Renderable
58. VS Structural
 Sub-typing Nominal
 Sub-typing
59. Escape Hatches
60. Escape Hatch #1: Any from typing import Any def __getattr__(self, name: str) -> Any: return getattr(self.wrapped, name) my_config: Dict[str, Union[str, int, List[Union[str, int], Dict[str, Union[str, float]]]]] my_config: Dict[str, Any] # (could also use TypedDict)
61. Escape Hatch #2: cast from typing import cast my_config = get_config_var('my_config') reveal_type(my_config) # Any my_config = cast( Dict[str, int], get_config_var('my_config'), ) reveal_type(my_config) # Dict[str, int]
62. Escape Hatch #3: ignore triggers_a_mypy_bug('foo') # type: ignore # github.com/python/mypy/issues/1362 @property # type: ignore @timer def is_visible(self) -> bool: ...
63. Escape Hatch #4: stub (pyi) files $ ls fastmath.so fastmath.pyi
64. Escape Hatch #4: stub (pyi) files # fastmath.pyi def square(x: int) -> int: ... class Complex: def __init__(real: int, imag: int) -> None: self.real = real self.imag = imag
65. Review
66. Protocols STATICALLY CHECKED DUCK-TYPING! Escape Hatches ANY, CAST, IGNORE, STUB FILES Annotate Function Signatures ARGUMENTS AND RETURN VALUES Annotate Variables ONLY IF YOU HAVE TO Unions and Optionals USE SPARINGLY Overloads and Generics TEACH THE TYPE CHECKER TO BE SMARTER
67. Why type The Plan 1 How to even type 2 Gradual typing of existing code 3 Issues you may run into 4 WHY type The Plan 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4
68. GRADUAL typing
71. • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT Gradual Typing
72. • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT Gradual Typing
73. • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT Gradual Typing
74. Gradual Typing • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT • START WITH MOST-USED
75. • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT • START WITH MOST-USED • USE CI TO DEFEND PROGRESS Gradual Typing
76. MonkeyType
77. Using Monkeytype $ pip install monkeytype $ monkeytype run mytests.py $ monkeytype stub some.module def myfunc(x: int) -> int: ... class Duck: def __init__(self, name: str) -> None: ... def quack(self, volume: int) -> str: ... $ monkeytype apply some.module
78. Why type The Plan 1 How to even type 2 Gradual typing of existing code 3 Issues you may run into 4 WHY type The Plan 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4
79. ISSUES
 you may run into
80. • Why does mypy complain about X here and not there? • Code in non-annotated functions is not checked! • Possible fix: visualize type-checked vs not-type-checked? The Apparent Inconsistency 3 4 5 2 1 1
81. class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self): ... class Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2 2
82. class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self): ... class Bar: def __init__(self, foo: Foo): self.foo = foo NameError: name "Bar" is not defined The Forward Reference 3 4 5 1 2
83. class Foo: def get_bars(self) -> List['Bar']: return get_bars_for_foo(self): ... class Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2
84. from typing import TYPE_CHECKING if TYPE_CHECKING: from .models import Bar def get_bar() -> 'Bar': return bar_from_somewhere() # noqa The Forward Reference 3 4 5 1 2
85. # fix coming in Python 3.7! from __future__ import annotations class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self) class Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2
86. # fix coming in Python 3.7! from __future__ import annotations class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self) class Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2
87. def process(nums: List[Optional[int]]): ... def foo(nums: List[int]): process(nums) The "Mutable Containers Are Invariant" 3 4 5 1 3 2
88. def process(nums: List[Optional[int]]): ... def foo(nums: List[int]): process(nums) # incompatible type "List[int]"; # expected "List[Optional[int]]" The "Mutable Containers Are Invariant" 4 5 1 3 2
89. def process(nums: Sequence[Optional[int]]): ... def foo(nums: List[int]): process(nums) The "Mutable Containers Are Invariant" 4 5 1 3 2
90. def process(objs: List[Base]): ... def foo(objs: List[SubBase]): process(objs) The "Mutable Containers Are Invariant" 4 5 1 3 2
91. def process(objs: List[Base]): ... def foo(objs: List[SubBase]): process(objs) # incompatible type "List[SubBase]"; # expected "List[Base]" The "Mutable Containers Are Invariant" 4 5 1 3 2
92. def process(objs: Sequence[Base]): ... def foo(objs: List[Base]): process(objs) The "Mutable Containers Are Invariant" 4 5 1 3 2
93. • The Python standard library has no type annotations. • But stdlib annotations would be really useful! • Instead: github.com/python/typeshed • When mypy says "no such attribute" on a stdlib type, but it clearly exists: typeshed bug! The Type Shed 3 4 5 1 4 2
94. # you _always_ want this option: --no-implicit-optional # if you're starting a greenfield project, # you might want this: --strict The Legacy Config Defaults 3 4 5 1 5 2
95. The Future...
96. Python 3.7: no more ugly string forward references. Fewer imports from typing module: dict not typing.Dict PEP 561: bundling type stubs with third-party packages Using type annotations to improve performance? Runtime type enforcement?
97. Type-checked Python is here and it works! It catches bugs, and developers like it. With MonkeyType, you can annotate large legacy codebases. Early days, far from perfect, but Good Enough. It'll get better!
98. THANK YOU!
99. MYPY DOCUMENTATION: MYPY.READTHEDOCS.IO THE REFERENCE STANDARD: PYTHON.ORG/DEV/PEPS/PEP-0484/ REALTIME SUPPORT: GITTER.IM/PYTHON/TYPING PEP 484 ISSUES: GITHUB.COM/PYTHON/TYPING TYPE CHECKER ISSUES: GITHUB.COM/PYTHON/MYPY STDLIB ANNOTATION ISSUES: GITHUB.COM/PYTHON/TYPESHED MONKEYTYPE ISSUES: GITHUB.COM/INSTAGRAM/MONKEYTYPE
100. •CARLJM ON IRC (#PYTHON) •@CARLJM •CARLJM@INSTAGRAM.COM •INSTAGR.AM/CARL.J.MEYER SEE YOU ON THE INTERNETS!