Oh my py. Type hints in Python

Oh my py. Type hints in Python

Slides for Dnepr.py#5 talk about type hints / type annotations in Python 3 & mypy. How to use them, when and what the common problems


Igor Davydenko

November 02, 2016


  1. Oh my py Oh my py ! ! Igor Davydenko

    2016, Dnepr.py#5
  2. Type hints Type hints in Python in Python And how

    to use them And how to use them Igor Davydenko 2016, Dnepr.py#5
  3. Brief History aka Brief History aka PEP-484 PEP-484 Designed on

    top of PEP-3107 (Function Annotations) Created on September 2014 Became part of standard library in Python 3.5 Endorsed by Guido van Rossum
  4. Basics Basics def hello(text: str) -> str: return 'Hello, {0}'.format(text)

    def sum(a: int, b: int) -> int: return a + b
  5. Basics Basics from decimal import Decimal, ROUND_UP PRECISION = Decimal('.01')

    def quantize_payment(hours: float, rate: Decimal) -> Decimal: return (Decimal(hours) * rate).quantize(PRECISION, ROUND_UP)
  6. str, bytes, Text, AnyStr str, bytes, Text, AnyStr from typing

    import AnyStr, Text def response(body: bytes=None, text: str=None) -> bytes: ... def response(content: AnyStr=None) -> bytes: ... def render_template(path: Text) -> str: ...
  7. list, List, Tuple, Set, Sequence list, List, Tuple, Set, Sequence

    from typing import List, Sequence, Set, Tuple def avg(data: list) -> float: return sum(data, 0.) / len(data) def avg(data: List[float]) -> float: return sum(data, 0.) / len(data) def split_name(value: str) -> Tuple[str, str]: return tuple(value.split(' ', 1)) def avg_tuple(data: Tuple[float, ...]) -> float: return sum(data, 0.) / len(data) def unique(data: List[str]) -> Set[str]: return set(data) def avg(data: Sequence[float]) -> float: return sum(data, 0.) / len(data)
  8. dict, Dict, Mapping dict, Dict, Mapping from typing import Dict,

    Mapping def read_data(slug: str) -> dict: ... def read_data(slug: str) -> Dict[str, str]: ... def process_data(data: Mapping[str, str]) -> None: for key, value in data.items(): ...
  9. Any, Union, Optional Any, Union, Optional from typing import Any,

    Optional, Sequence, Union def read_data(slug: str) -> Dict[str, Union[int, str]]: ... def read_data(slug: str) -> Dict[str, Any]: ... def avg(data: Sequence[float]) -> Optional[float]: if len(data): return sum(data, 0.) / len(data) return None
  10. Callable Callable from random import randint from typing import Callable

    def random_int_factory(min: int=0, max: int=100) -> Callable[[], int]: def random_int(): return randint(min, max) return random_int
  11. Aliases Aliases from aiohttp import web View = Callable[[web.Request], web.Response]

    def middleware() -> Callable[[web.Application, View], Awaitable[View]]: async def factory(app: web.Application, handler: View) -> View: async def middleware(request: web.Request) -> web.Response: ...
  12. Classes Classes class Schema(object): name = None # type: str

    def __init__(self, name: str=None) -> None: self.name = name or self.name def clone(self) -> 'Schema': return Schema(self.name) def load(self) -> bool: with open(SCHEMA_PATH / '{0}.json'.format(self.name)) as handler: return json.loads(handler.read()) def validate(self, data: Dict[str, Any]) -> Dict[str, Any]: json_schema = self.load() return fastjsonschema.compile(json_schema).validate(data)
  13. Types Types from typing import NewType UserID = NewType('UserID', int)

    def fetch_user(user_id: UserID) -> Any: ... fetch_user(UserID(1)) # OK fetch_user(1) # Type check will fail
  14. Inline Type Hints Inline Type Hints WEEKS = defaultdict(lambda: {

    False: list(range(8)), True: list(range(16)), }) # type: Dict[str, Dict[bool, List[int]]]
  15. Stubs Stubs package/module.py package/module.py def avg(data): if not len(data): return

    None return sum(data, 0.) / len(data) def avg_unique(data): return avg(unique(data)) def unique(data): return set(data)
  16. Stubs Stubs package/module.pyi package/module.pyi def avg(data: Sequence[float]) -> Optional[float]: ...

    def avg_unique(data: Sequence[float]) -> Optional[float]: ... def unique(data: Sequence[float]) -> Set[float]: ...
  17. Typeshed Typeshed python/typeshed Provide stubs for standard library And some

    widely-used shared libraries Have stubs for Python 2 & Python 3 Curated by PSF
  18. Python 2 Type Hints Python 2 Type Hints def hello(name):

    # type: (name: str) -> str return 'Hello, {0}'.format(name) def multi_line_annotations(address, # type: Union[str, List[str]] sender, # type: str subject, # type: str body # type: str ): # type: (...) -> bool ... More info at mypy docs.
  19. mypy mypy

  20. Introduction Introduction Static type checker for Python Static type checker

    for Python Works with Python 3 & Python 2 code Still experimental Still experimental Developed at Dropbox Again. Endorsed by Guido
  21. Usage Usage $ pip install mypy-lang typed-ast $ mypy ...

    $ mypy --fast-parser ...
  22. Configuration. Step 1 Configuration. Step 1 mypy.ini mypy.ini [mypy] fast_parser

    = True check_untyped_defs = True warn_redundant_casts = True
  23. Configuration. Step 2 Configuration. Step 2 mypy.ini mypy.ini silent_imports =

    True [mypy] fast_parser = True check_untyped_defs = True warn_redundant_casts = True
  24. Configuration. Step 3 Configuration. Step 3 mypy.ini mypy.ini disallow_untyped_defs =

    True [mypy] fast_parser = True silent_imports = True check_untyped_defs = True warn_redundant_casts = True
  25. Output Output env/bin/mypy project/ project/signals.py: note: In function "cache_api_urls": project/signals.py:25:

    error: Argument 2 to "api_url" has incompatible type "str"; expected "int"
  26. Problems Problems

  27. No Hype No Hype mypy is not widely used No

    viable benefits for users Hard to migrate large codebase to type hints
  28. Long Lines / Ugly Code Long Lines / Ugly Code

    Before Before async def retrieve_tweets(pool, count=50): ... A er A er async def retrieve_tweets( pool: Pool, count: int=50 ) -> Sequence[Mapping[str, Any]]: ...
  29. Stubs Stubs Hard to maintain Hard to maintain Easy to

    get into situation, when implementation != stub Completely same problems as with tests & documentation
  30. Incomplete Stubs Incomplete Stubs import asyncio def main() -> int:

    loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(some_async_def), asyncio.ensure_future(some_other_async_def), ] loop.run_until_complete(asyncio.gather(*tasks, loop=loop)) loop.close() return 0
  31. Incomplete Stubs Incomplete Stubs $ mypy -c '...' <string>: note:

    In function "main": <string>:6: error: "module" has no attribute "ensure_future" <string>:7: error: "module" has no attribute "ensure_future" <string>:9: error: "module" has no attribute "gather" <string>:6: error: Name 'some_async_def' is not defined <string>:7: error: Name 'some_other_async_def' is not defined
  32. Incomplete Stubs Incomplete Stubs from lxml import etree def fetch_data(url:

    str) -> etree._Element: ... def use_data(): data = fetch_data(URL) for item in data.iterfind(...): # Will fail with "has no attribute" error ...
  33. # noqa: F401 # noqa: F401 from typing import Dict,

    List, Set def process_data(data: Dict[str, str]) -> List[int]: uniques = set() # type: Set[int] for value in data.values(): uniques.add(int(value)) return list(uniques) module.py:1:1: F401 'typing.Set' imported but unused
  34. Solution to Solution to # noqa: F401 # noqa: F401

    PEP-526 implements syntax for variable annotations. uniques: Set[int] = set() number: int # Works even without assignment class Schema(object): name: str data: Dict[str, str] = {} Included in Python 3.6 Included in Python 3.6
  35. Circular Imports Circular Imports project/models.py project/models.py from .utils import some_func

    class Model(object): def shortcut_for_some_func(self) -> int: return some_func(self) project/utils.py project/utils.py import typing if typing.TYPE_CHECKING: from .models import Model def some_func(model: 'Model') -> int: ...
  36. # type: ignore # type: ignore Sooner or later, but

    you'll need to use # type: ingore asyncio.gather(*tasks, loop=loop) # type: ignore for item in data.iterfind('...'): # type: ignore mypy is still experimental and you'll need to use # type: ignore a er yelling WTF !
  37. Additional Notes Additional Notes

  38. pytype pytype google/pytype Type checker from Google Needs Python 2.7

    to run Able to type check Python 3 code though
  39. enforce enforce RussBaz/enforce Runtime type checking Designed to use at

    tests or for data validation import enforce @enforce.runtime_validation def hello(name: str) -> str: return 'Hello, {0}!'.format(name) hello('world') hello(1) # Will fail with RuntimeTypeError
  40. mypy-django mypy-django machinails/mypy-django Type hints bindings for Django Experimental Not

    full coverage of Django internals
  41. Conclusion Conclusion " " # # # # $ $

    % % & & ' '
  42. Questions? Questions? Twi er: Twi er: @playpausenstop @playpausenstop GitHub: GitHub:

    @playpauseandstop @playpauseandstop Made in Ukraine