Carl Meyer - Type-checked Python in the real world

Carl Meyer - Type-checked Python in the real world

You've heard about Python type annotations, but wondered if they're useful in the real world? Worried you've got too much code and can't afford to annotate it? Type-checked Python is here, it's for real, and it can help you catch bugs and make your code easier to understand. Come learn from our experience gradually typing a million-LOC production Python application!

Type checking solves real world problems in production Python systems. We'll cover the benefits, how type checking in Python works, how to introduce it gradually and sustainably in a production Python application, and how to measure success and avoid common pitfalls. We'll even demonstrate how modern Python typechecking goes hand-in-hand with duck-typing! Join us for a deep dive into type-checked Python in the real world.

https://us.pycon.org/2018/schedule/presentation/102/

De174d82b2bbfe9e6f14a5a8c38b14be?s=128

PyCon 2018

May 11, 2018
Tweet

Transcript

  1. Type-Checked Python INSTAGRAM Carl Meyer

  2. None
  3. @carljm

  4. @carljm

  5. @carljm

  6. The Plan WHY type 1 HOW to even type 2

    GRADUAL typing 3 ISSUES you may run into 4
  7. add annotations to your Python? WHY

  8. "Items"? def process(self, items): for item in items: self.append(item.value.id)

  9. "Items"? def process(self, items): for item in items: self.append(item.value.id)

  10. "Items"? def process(self, items): for item in items: self.append(item.value.id)

  11. "Items"? def process(self, items): for item in items: self.append(item.value.id)

  12. "Items"? def process(self, items): for item in items: self.append(item.value.id)

  13. "Items"? from typing import Sequence from .models import Item def

    process(self, items: Sequence[Item]) -> None: for item in items: self.append(item.value.id)
  14. "Items"? from typing import Sequence from .models import Item def

    process(self, items: Sequence[Item]) -> None: for item in items: self.append(item.value.id)
  15. "That's cool, but I don't need it; 
 I'd catch

    that with a test!" — PYTHONISTA
  16. One Test Case

  17. More Test Cases

  18. Parametrized Test Case

  19. Property-based Test

  20. Strings

  21. Lists

  22. Dictionaries

  23. None
  24. Why type The Plan 1 How to even type 2

    Gradual typing of existing code 3 Issues you may run into 4 WHY type The Plan 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4
  25. HOW to even type

  26. square.py def square(x: int) -> int: return x**2

  27. square.py def square(x: int) -> int: return x**2

  28. square.py def square(x: int) -> int: return x**2 square(3) square('foo')

    square(4) + 'foo'
  29. Running Mypy $ pip install mypy $ mypy square.py square.py:5:

    error: Argument 1 to "square" has incompatible type "str"; expected "int" square.py:6: error: Unsupported operand types for + ("int" and "str")
  30. square.py def square(x: int) -> int: return x**2 square(3) square('foo')

    square(4) + 'foo' Argument 1 to "square" has incompatible type "str"; expected "int"
  31. square.py def square(x: int) -> int: return x**2 square(3) square('foo')

    square(4) + 'foo' Unsupported operand types for + ("int" and "str")
  32. Type Inference from typing import Tuple class Photo: def __init__(self,

    width: int, height: int) -> None: self.width = width self.height = height def get_dimensions(self) -> Tuple[str, str]: return (self.width, self.height)
  33. Type Inference Incompatible return value type (got "Tuple[int, int]", expected

    "Tuple[str, str]") from typing import Tuple class Photo: def __init__(self, width: int, height: int) -> None: self.width = width self.height = height def get_dimensions(self) -> Tuple[str, str]: return (self.width, self.height)
  34. Type Inference photos = [ Photo(640, 480), Photo(1024, 768), ]

    photos.append('foo')
  35. Type Inference photos = [ Photo(640, 480), Photo(1024, 768), ]

    photos.append('foo') Argument 1 to "append" of "list" has incompatible type "str"; expected "Photo"
  36. Variable Annotation class Photo: def __init__(self, width: int, height: int)

    -> None: self.width = width self.height = height self.tags = []
  37. Variable Annotation class Photo: def __init__(self, width: int, height: int)

    -> None: self.width = width self.height = height self.tags = [] Need type annotation for variable
  38. Variable Annotation from typing import List class Photo: def __init__(self,

    width: int, height: int) -> None: self.width = width self.height = height self.tags: List[str] = []
  39. Review

  40. Annotate Function Signatures ARGUMENTS AND RETURN VALUES Annotate Variables ONLY

    IF YOU HAVE TO
  41. Union from typing import Union def get_foo_or_bar(id: int) -> Union[Foo,

    Bar]: ... def get_foo_or_none(id: int) -> Union[Foo, None]: ...
  42. Union from typing import Union, Optional def get_foo_or_bar(id: int) ->

    Union[Foo, Bar]: ... def get_foo_or_none(id: int) -> Union[Foo, None]: ... def get_foo_or_none(id: int) -> Optional[Foo]: ...
  43. Optional from typing import Optional def get_foo(foo_id: Optional[int]) -> Optional[Foo]:

    if foo_id is None: return None return Foo(foo_id) my_foo = get_foo(3) my_foo.id # error: NoneType has no attribute 'id'
  44. Function Overloads from typing import Optional, overload @overload def get_foo(foo_id:

    None) -> None: pass @overload def get_foo(foo_id: int) -> Foo: pass def get_foo(foo_id: Optional[int]) -> Optional[Foo]: if foo_id is None: return None return Foo(foo_id) reveal_type(get_foo(None)) # None reveal_type(get_foo(1)) # Foo
  45. Function Overloads from typing import Optional, overload @overload def get_foo(foo_id:

    None) -> None: pass @overload def get_foo(foo_id: int) -> Foo: pass def get_foo(foo_id: Optional[int]) -> Optional[Foo]: if foo_id is None: return None return Foo(foo_id) reveal_type(get_foo(None)) # None reveal_type(get_foo(1)) # Foo
  46. Generics numbers: List[int] = [] books_by_id: Dict[int, Book] = {}

    int_tree: TreeNode[int] = TreeNode(3) str_tree: TreeNode[str] = TreeNode('foo') book_tree: TreeNode[Book] = TreeNode(mybook)
  47. Generics from typing import Generic, TypeVar TNodeValue = TypeVar('TNodeValue') class

    TreeNode(Generic[TNodeValue]): def __init__(self, value: TNodeValue) -> None: self.value = value node = TreeNode(3) reveal_type(node) # TreeNode[int] reveal_type(node.value) # int
  48. Generics from typing import Generic, TypeVar TNodeValue = TypeVar('TNodeValue') class

    TreeNode(Generic[TNodeValue]): def __init__(self, value: TNodeValue) -> None: self.value = value node = TreeNode(3) reveal_type(node) # TreeNode[int] reveal_type(node.value) # int
  49. Generic Functions from typing import TypeVar AnyStr = TypeVar('AnyStr', str,

    bytes) def concat(a: AnyStr, b: AnyStr) -> AnyStr: return a + b concat('foo', b'bar') # typecheck error! concat(3, 6) # typecheck error! reveal_type(concat('foo', 'bar')) # str reveal_type(concat(b'foo', b'bar')) # bytes
  50. Generic Functions from typing import AnyStr def concat(a: AnyStr, b:

    AnyStr) -> AnyStr: return a + b concat('foo', b'bar') # typecheck error! concat(3, 6) # typecheck error! reveal_type(concat('foo', 'bar')) # str reveal_type(concat(b'foo', b'bar')) # bytes
  51. Review

  52. Annotate Function Signatures ARGUMENTS AND RETURN VALUES Annotate Variables ONLY

    IF YOU HAVE TO Unions and Optionals USE SPARINGLY Overloads and Generics TEACH THE TYPE CHECKER TO BE SMARTER
  53. Where's my duck?

  54. def render(obj): return obj.render() Where's my duck?

  55. def render(obj: object) -> str: return obj.render() "object" has no

    attribute "render" Where's my duck?
  56. from typing import Any def render(obj: Any) -> str: return

    obj.render() render(3) # typechecks, but fails at runtime Where's my duck?
  57. from typing_extensions import Protocol class Renderable(Protocol): def render(self) -> str:

    ... def render(obj: Renderable) -> str: return obj.render() class Foo: def render(self) -> str: return "Foo!" render(Foo()) # clean! render(3) # error: expected Renderable
  58. VS Structural
 Sub-typing Nominal
 Sub-typing

  59. Escape Hatches

  60. Escape Hatch #1: Any from typing import Any def __getattr__(self,

    name: str) -> Any: return getattr(self.wrapped, name) my_config: Dict[str, Union[str, int, List[Union[str, int], Dict[str, Union[str, float]]]]] my_config: Dict[str, Any] # (could also use TypedDict)
  61. Escape Hatch #2: cast from typing import cast my_config =

    get_config_var('my_config') reveal_type(my_config) # Any my_config = cast( Dict[str, int], get_config_var('my_config'), ) reveal_type(my_config) # Dict[str, int]
  62. Escape Hatch #3: ignore triggers_a_mypy_bug('foo') # type: ignore # github.com/python/mypy/issues/1362

    @property # type: ignore @timer def is_visible(self) -> bool: ...
  63. Escape Hatch #4: stub (pyi) files $ ls fastmath.so fastmath.pyi

  64. Escape Hatch #4: stub (pyi) files # fastmath.pyi def square(x:

    int) -> int: ... class Complex: def __init__(real: int, imag: int) -> None: self.real = real self.imag = imag
  65. Review

  66. Protocols STATICALLY CHECKED DUCK-TYPING! Escape Hatches ANY, CAST, IGNORE, STUB

    FILES Annotate Function Signatures ARGUMENTS AND RETURN VALUES Annotate Variables ONLY IF YOU HAVE TO Unions and Optionals USE SPARINGLY Overloads and Generics TEACH THE TYPE CHECKER TO BE SMARTER
  67. Why type The Plan 1 How to even type 2

    Gradual typing of existing code 3 Issues you may run into 4 WHY type The Plan 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4
  68. GRADUAL typing

  69. None
  70. None
  71. • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT Gradual Typing

  72. • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT Gradual Typing

  73. • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT Gradual Typing

  74. Gradual Typing • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT •

    START WITH MOST-USED
  75. • ANNOTATED FUNCTIONS ONLY • NETWORK EFFECT • START WITH

    MOST-USED • USE CI TO DEFEND PROGRESS Gradual Typing
  76. MonkeyType

  77. Using Monkeytype $ pip install monkeytype $ monkeytype run mytests.py

    $ monkeytype stub some.module def myfunc(x: int) -> int: ... class Duck: def __init__(self, name: str) -> None: ... def quack(self, volume: int) -> str: ... $ monkeytype apply some.module
  78. Why type The Plan 1 How to even type 2

    Gradual typing of existing code 3 Issues you may run into 4 WHY type The Plan 1 HOW to even type 2 GRADUAL typing 3 ISSUES you may run into 4
  79. ISSUES
 you may run into

  80. • Why does mypy complain about X here and not

    there? • Code in non-annotated functions is not checked! • Possible fix: visualize type-checked vs not-type-checked? The Apparent Inconsistency 3 4 5 2 1 1
  81. class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self): ... class

    Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2 2
  82. class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self): ... class

    Bar: def __init__(self, foo: Foo): self.foo = foo NameError: name "Bar" is not defined The Forward Reference 3 4 5 1 2
  83. class Foo: def get_bars(self) -> List['Bar']: return get_bars_for_foo(self): ... class

    Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2
  84. from typing import TYPE_CHECKING if TYPE_CHECKING: from .models import Bar

    def get_bar() -> 'Bar': return bar_from_somewhere() # noqa The Forward Reference 3 4 5 1 2
  85. # fix coming in Python 3.7! from __future__ import annotations

    class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self) class Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2
  86. # fix coming in Python 3.7! from __future__ import annotations

    class Foo: def get_bars(self) -> List[Bar]: return get_bars_for_foo(self) class Bar: def __init__(self, foo: Foo): self.foo = foo The Forward Reference 3 4 5 1 2
  87. def process(nums: List[Optional[int]]): ... def foo(nums: List[int]): process(nums) The "Mutable

    Containers Are Invariant" 3 4 5 1 3 2
  88. def process(nums: List[Optional[int]]): ... def foo(nums: List[int]): process(nums) # incompatible

    type "List[int]"; # expected "List[Optional[int]]" The "Mutable Containers Are Invariant" 4 5 1 3 2
  89. def process(nums: Sequence[Optional[int]]): ... def foo(nums: List[int]): process(nums) The "Mutable

    Containers Are Invariant" 4 5 1 3 2
  90. def process(objs: List[Base]): ... def foo(objs: List[SubBase]): process(objs) The "Mutable

    Containers Are Invariant" 4 5 1 3 2
  91. def process(objs: List[Base]): ... def foo(objs: List[SubBase]): process(objs) # incompatible

    type "List[SubBase]"; # expected "List[Base]" The "Mutable Containers Are Invariant" 4 5 1 3 2
  92. def process(objs: Sequence[Base]): ... def foo(objs: List[Base]): process(objs) The "Mutable

    Containers Are Invariant" 4 5 1 3 2
  93. • The Python standard library has no type annotations. •

    But stdlib annotations would be really useful! • Instead: github.com/python/typeshed • When mypy says "no such attribute" on a stdlib type, but it clearly exists: typeshed bug! The Type Shed 3 4 5 1 4 2
  94. # you _always_ want this option: --no-implicit-optional # if you're

    starting a greenfield project, # you might want this: --strict The Legacy Config Defaults 3 4 5 1 5 2
  95. The Future...

  96. Python 3.7: no more ugly string forward references. Fewer imports

    from typing module: dict not typing.Dict PEP 561: bundling type stubs with third-party packages Using type annotations to improve performance? Runtime type enforcement?
  97. Type-checked Python is here and it works! It catches bugs,

    and developers like it. With MonkeyType, you can annotate large legacy codebases. Early days, far from perfect, but Good Enough. It'll get better!
  98. THANK YOU!

  99. MYPY DOCUMENTATION: MYPY.READTHEDOCS.IO THE REFERENCE STANDARD: PYTHON.ORG/DEV/PEPS/PEP-0484/ REALTIME SUPPORT: GITTER.IM/PYTHON/TYPING

    PEP 484 ISSUES: GITHUB.COM/PYTHON/TYPING TYPE CHECKER ISSUES: GITHUB.COM/PYTHON/MYPY STDLIB ANNOTATION ISSUES: GITHUB.COM/PYTHON/TYPESHED MONKEYTYPE ISSUES: GITHUB.COM/INSTAGRAM/MONKEYTYPE
  100. •CARLJM ON IRC (#PYTHON) •@CARLJM •CARLJM@INSTAGRAM.COM •INSTAGR.AM/CARL.J.MEYER SEE YOU ON

    THE INTERNETS!
  101. None