Carl Meyer - Type-checked Python in the real world

Slide 1

Slide 1 text

Type-Checked Python INSTAGRAM Carl Meyer

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

@carljm

Slide 4

Slide 4 text

@carljm

Slide 5

Slide 5 text

@carljm

Slide 6

Slide 6 text

The Plan WHY type 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4

Slide 7

Slide 7 text

add annotations to your Python? WHY

Slide 8

Slide 8 text

"Items"? def process(self, items): for item in items: self.append(item.value.id)

Slide 9

Slide 9 text

"Items"? def process(self, items): for item in items: self.append(item.value.id)

Slide 10

Slide 10 text

"Items"? def process(self, items): for item in items: self.append(item.value.id)

Slide 11

Slide 11 text

"Items"? def process(self, items): for item in items: self.append(item.value.id)

Slide 12

Slide 12 text

"Items"? def process(self, items): for item in items: self.append(item.value.id)

Slide 13

Slide 13 text

"Items"? from typing import Sequence from .models import Item def process(self, items: Sequence[Item]) -> None: for item in items: self.append(item.value.id)

Slide 14

Slide 14 text

"Items"? from typing import Sequence from .models import Item def process(self, items: Sequence[Item]) -> None: for item in items: self.append(item.value.id)

Slide 15

Slide 15 text

"That's cool, but I don't need it;   I'd catch that with a test!" — PYTHONISTA

Slide 16

Slide 16 text

One Test Case

Slide 17

Slide 17 text

More Test Cases

Slide 18

Slide 18 text

Parametrized Test Case

Slide 19

Slide 19 text

Property-based Test

Slide 20

Slide 20 text

Strings

Slide 21

Slide 21 text

Lists

Slide 22

Slide 22 text

Dictionaries

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Why type The Plan 1 How to even type 2 Gradual typing of existing code 3 Issues you may run into 4 WHY type The Plan 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4

Slide 25

Slide 25 text

HOW to even type

Slide 26

Slide 26 text

square.py def square(x: int) -> int: return x**2

Slide 27

Slide 27 text

square.py def square(x: int) -> int: return x**2

Slide 28

Slide 28 text

square.py def square(x: int) -> int: return x**2 square(3) square('foo') square(4) + 'foo'

Slide 29

Slide 29 text

Running Mypy $ pip install mypy $ mypy square.py square.py:5: error: Argument 1 to "square" has incompatible type "str"; expected "int" square.py:6: error: Unsupported operand types for + ("int" and "str")

Slide 30

Slide 30 text

square.py def square(x: int) -> int: return x**2 square(3) square('foo') square(4) + 'foo' Argument 1 to "square" has incompatible type "str"; expected "int"

Slide 31

Slide 31 text

square.py def square(x: int) -> int: return x**2 square(3) square('foo') square(4) + 'foo' Unsupported operand types for + ("int" and "str")

Slide 32

Slide 32 text

Type Inference from typing import Tuple class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height def get_dimensions(self) -> Tuple[str, str]: return (self.width, self.height)

Slide 33

Slide 33 text

Type Inference Incompatible return value type (got "Tuple[int, int]", expected "Tuple[str, str]") from typing import Tuple class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height def get_dimensions(self) -> Tuple[str, str]: return (self.width, self.height)

Slide 34

Slide 34 text

Type Inference photos = [ Photo(640, 480), Photo(1024, 768), ] photos.append('foo')

Slide 35

Slide 35 text

Type Inference photos = [ Photo(640, 480), Photo(1024, 768), ] photos.append('foo') Argument 1 to "append" of "list" has incompatible type "str"; expected "Photo"

Slide 36

Slide 36 text

Variable Annotation class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height self.tags = []

Slide 37

Slide 37 text

Variable Annotation class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height self.tags = [] Need type annotation for variable

Slide 38

Slide 38 text

Variable Annotation from typing import List class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height self.tags: List[str] = []

Slide 39

Slide 39 text

Review

Slide 40

Slide 40 text

Annotate Function Signatures ARGUMENTS AND RETURN VALUES Annotate Variables ONLY IF YOU HAVE TO

Slide 41

Slide 41 text

Union from typing import Union def get_foo_or_bar(id: int) -> Union[Foo, Bar]: ... def get_foo_or_none(id: int) -> Union[Foo, None]: ...

Slide 42

Slide 42 text

Union from typing import Union, Optional def get_foo_or_bar(id: int) -> Union[Foo, Bar]: ... def get_foo_or_none(id: int) -> Union[Foo, None]: ... def get_foo_or_none(id: int) -> Optional[Foo]: ...

Slide 43

Slide 43 text

Optional from typing import Optional def get_foo(foo_id: Optional[int]) -> Optional[Foo]: if foo_id is None: return None return Foo(foo_id) my_foo = get_foo(3) my_foo.id # error: NoneType has no attribute 'id'

Slide 44

Slide 44 text

Function Overloads from typing import Optional, overload @overload def get_foo(foo_id: None) -> None: pass @overload def get_foo(foo_id: int) -> Foo: pass def get_foo(foo_id: Optional[int]) -> Optional[Foo]: if foo_id is None: return None return Foo(foo_id) reveal_type(get_foo(None)) # None reveal_type(get_foo(1)) # Foo

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Generics numbers: List[int] = [] books_by_id: Dict[int, Book] = {} int_tree: TreeNode[int] = TreeNode(3) str_tree: TreeNode[str] = TreeNode('foo') book_tree: TreeNode[Book] = TreeNode(mybook)

Slide 47

Slide 47 text

Generics from typing import Generic, TypeVar TNodeValue = TypeVar('TNodeValue') class TreeNode(Generic[TNodeValue]): def __init__(self, value: TNodeValue) -> None: self.value = value node = TreeNode(3) reveal_type(node) # TreeNode[int] reveal_type(node.value) # int

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Generic Functions from typing import TypeVar AnyStr = TypeVar('AnyStr', str, bytes) def concat(a: AnyStr, b: AnyStr) -> AnyStr: return a + b concat('foo', b'bar') # typecheck error! concat(3, 6) # typecheck error! reveal_type(concat('foo', 'bar')) # str reveal_type(concat(b'foo', b'bar')) # bytes

Slide 50

Slide 50 text

Generic Functions from typing import AnyStr def concat(a: AnyStr, b: AnyStr) -> AnyStr: return a + b concat('foo', b'bar') # typecheck error! concat(3, 6) # typecheck error! reveal_type(concat('foo', 'bar')) # str reveal_type(concat(b'foo', b'bar')) # bytes

Slide 51

Slide 51 text

Review

Slide 52

Slide 52 text

Annotate Function Signatures ARGUMENTS AND RETURN VALUES Annotate Variables ONLY IF YOU HAVE TO Unions and Optionals USE SPARINGLY Overloads and Generics TEACH THE TYPE CHECKER TO BE SMARTER

Slide 53

Slide 53 text

Where's my duck?

Slide 54

Slide 54 text

def render(obj): return obj.render() Where's my duck?

Slide 55

Slide 55 text

def render(obj: object) -> str: return obj.render() "object" has no attribute "render" Where's my duck?

Slide 56

Slide 56 text

from typing import Any def render(obj: Any) -> str: return obj.render() render(3) # typechecks, but fails at runtime Where's my duck?

Slide 57

Slide 57 text

from typing_extensions import Protocol class Renderable(Protocol): def render(self) -> str: ... def render(obj: Renderable) -> str: return obj.render() class Foo: def render(self) -> str: return "Foo!" render(Foo()) # clean! render(3) # error: expected Renderable

Slide 58

Slide 58 text

VS Structural  Sub-typing Nominal  Sub-typing

Slide 59

Slide 59 text

Escape Hatches

Slide 60

Slide 60 text

Escape Hatch #1: Any from typing import Any def __getattr__(self, name: str) -> Any: return getattr(self.wrapped, name) my_config: Dict[str, Union[str, int, List[Union[str, int], Dict[str, Union[str, float]]]]] my_config: Dict[str, Any] # (could also use TypedDict)

Slide 61

Slide 61 text

Escape Hatch #2: cast from typing import cast my_config = get_config_var('my_config') reveal_type(my_config) # Any my_config = cast( Dict[str, int], get_config_var('my_config'), ) reveal_type(my_config) # Dict[str, int]

Slide 62

Slide 62 text

Escape Hatch #3: ignore triggers_a_mypy_bug('foo') # type: ignore # github.com/python/mypy/issues/1362 @property # type: ignore @timer def is_visible(self) -> bool: ...

Slide 63

Slide 63 text

Escape Hatch #4: stub (pyi) files $ ls fastmath.so fastmath.pyi

Slide 64

Slide 64 text

Escape Hatch #4: stub (pyi) files # fastmath.pyi def square(x: int) -> int: ... class Complex: def __init__(real: int, imag: int) -> None: self.real = real self.imag = imag

Slide 65

Slide 65 text

Review

Slide 66

Slide 66 text

Protocols STATICALLY CHECKED DUCK-TYPING! Escape Hatches ANY, CAST, IGNORE, STUB FILES Annotate Function Signatures ARGUMENTS AND RETURN VALUES Annotate Variables ONLY IF YOU HAVE TO Unions and Optionals USE SPARINGLY Overloads and Generics TEACH THE TYPE CHECKER TO BE SMARTER

Slide 67

Slide 67 text

Why type The Plan 1 How to even type 2 Gradual typing of existing code 3 Issues you may run into 4 WHY type The Plan 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4

Slide 68

Slide 68 text

GRADUAL typing

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

• ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT Gradual Typing

Slide 72

Slide 72 text

• ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT Gradual Typing

Slide 73

Slide 73 text

• ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT Gradual Typing

Slide 74

Slide 74 text

Gradual Typing • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT • START WITH MOST-USED

Slide 75

Slide 75 text

• ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT • START WITH MOST-USED • USE CI TO DEFEND PROGRESS Gradual Typing

Slide 76

Slide 76 text

MonkeyType

Slide 77

Slide 77 text

Using Monkeytype $ pip install monkeytype $ monkeytype run mytests.py $ monkeytype stub some.module def myfunc(x: int) -> int: ... class Duck: def __init__(self, name: str) -> None: ... def quack(self, volume: int) -> str: ... $ monkeytype apply some.module

Slide 78

Slide 78 text

Why type The Plan 1 How to even type 2 Gradual typing of existing code 3 Issues you may run into 4 WHY type The Plan 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4

Slide 79

Slide 79 text

ISSUES  you may run into

Slide 80

Slide 80 text

• Why does mypy complain about X here and not there? • Code in non-annotated functions is not checked! • Possible fix: visualize type-checked vs not-type-checked? The Apparent Inconsistency 3 4 5 2 1 1

Slide 81

Slide 81 text

class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self): ... class Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2 2

Slide 82

Slide 82 text

class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self): ... class Bar: def __init__(self, foo: Foo): self.foo = foo NameError: name "Bar" is not deﬁned The Forward Reference 3 4 5 1 2

Slide 83

Slide 83 text

class Foo: def get_bars(self) -> List['Bar']: return get_bars_for_foo(self): ... class Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2

Slide 84

Slide 84 text

from typing import TYPE_CHECKING if TYPE_CHECKING: from .models import Bar def get_bar() -> 'Bar': return bar_from_somewhere() # noqa The Forward Reference 3 4 5 1 2

Slide 85

Slide 85 text

# ﬁx coming in Python 3.7! from __future__ import annotations class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self) class Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2

Slide 86

Slide 86 text

Slide 87

Slide 87 text

def process(nums: List[Optional[int]]): ... def foo(nums: List[int]): process(nums) The "Mutable Containers Are Invariant" 3 4 5 1 3 2

Slide 88

Slide 88 text

def process(nums: List[Optional[int]]): ... def foo(nums: List[int]): process(nums) # incompatible type "List[int]"; # expected "List[Optional[int]]" The "Mutable Containers Are Invariant" 4 5 1 3 2

Slide 89

Slide 89 text

def process(nums: Sequence[Optional[int]]): ... def foo(nums: List[int]): process(nums) The "Mutable Containers Are Invariant" 4 5 1 3 2

Slide 90

Slide 90 text

def process(objs: List[Base]): ... def foo(objs: List[SubBase]): process(objs) The "Mutable Containers Are Invariant" 4 5 1 3 2

Slide 91

Slide 91 text

def process(objs: List[Base]): ... def foo(objs: List[SubBase]): process(objs) # incompatible type "List[SubBase]"; # expected "List[Base]" The "Mutable Containers Are Invariant" 4 5 1 3 2

Slide 92

Slide 92 text

def process(objs: Sequence[Base]): ... def foo(objs: List[Base]): process(objs) The "Mutable Containers Are Invariant" 4 5 1 3 2

Slide 93

Slide 93 text

• The Python standard library has no type annotations. • But stdlib annotations would be really useful! • Instead: github.com/python/typeshed • When mypy says "no such attribute" on a stdlib type, but it clearly exists: typeshed bug! The Type Shed 3 4 5 1 4 2

Slide 94

Slide 94 text

# you _always_ want this option: --no-implicit-optional # if you're starting a greenﬁeld project, # you might want this: --strict The Legacy Config Defaults 3 4 5 1 5 2

Slide 95

Slide 95 text

The Future...

Slide 96

Slide 96 text

Python 3.7: no more ugly string forward references. Fewer imports from typing module: dict not typing.Dict PEP 561: bundling type stubs with third-party packages Using type annotations to improve performance? Runtime type enforcement?

Slide 97

Slide 97 text

Type-checked Python is here and it works! It catches bugs, and developers like it. With MonkeyType, you can annotate large legacy codebases. Early days, far from perfect, but Good Enough. It'll get better!

Slide 98

Slide 98 text

THANK YOU!

Slide 99

Slide 99 text

MYPY DOCUMENTATION: MYPY.READTHEDOCS.IO THE REFERENCE STANDARD: PYTHON.ORG/DEV/PEPS/PEP-0484/ REALTIME SUPPORT: GITTER.IM/PYTHON/TYPING PEP 484 ISSUES: GITHUB.COM/PYTHON/TYPING TYPE CHECKER ISSUES: GITHUB.COM/PYTHON/MYPY STDLIB ANNOTATION ISSUES: GITHUB.COM/PYTHON/TYPESHED MONKEYTYPE ISSUES: GITHUB.COM/INSTAGRAM/MONKEYTYPE

Slide 100

Slide 100 text

•CARLJM ON IRC (#PYTHON) •@CARLJM •CARLJM@INSTAGRAM.COM •INSTAGR.AM/CARL.J.MEYER SEE YOU ON THE INTERNETS!

Slide 101

Slide 101 text

No content