Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Oh my py. Type hints in Python

Oh my py. Type hints in Python

Slides for Dnepr.py#5 talk about type hints / type annotations in Python 3 & mypy. How to use them, when and what the common problems

Igor Davydenko

November 02, 2016

More Decks by Igor Davydenko

Other Decks in Programming


  1. Type hints Type hints in Python in Python And how

    to use them And how to use them Igor Davydenko 2016, Dnepr.py#5
  2. Brief History aka Brief History aka PEP-484 PEP-484 Designed on

    top of PEP-3107 (Function Annotations) Created on September 2014 Became part of standard library in Python 3.5 Endorsed by Guido van Rossum
  3. Basics Basics from decimal import Decimal, ROUND_UP PRECISION = Decimal('.01')

    def quantize_payment(hours: float, rate: Decimal) -> Decimal: return (Decimal(hours) * rate).quantize(PRECISION, ROUND_UP)
  4. str, bytes, Text, AnyStr str, bytes, Text, AnyStr from typing

    import AnyStr, Text def response(body: bytes=None, text: str=None) -> bytes: ... def response(content: AnyStr=None) -> bytes: ... def render_template(path: Text) -> str: ...
  5. list, List, Tuple, Set, Sequence list, List, Tuple, Set, Sequence

    from typing import List, Sequence, Set, Tuple def avg(data: list) -> float: return sum(data, 0.) / len(data) def avg(data: List[float]) -> float: return sum(data, 0.) / len(data) def split_name(value: str) -> Tuple[str, str]: return tuple(value.split(' ', 1)) def avg_tuple(data: Tuple[float, ...]) -> float: return sum(data, 0.) / len(data) def unique(data: List[str]) -> Set[str]: return set(data) def avg(data: Sequence[float]) -> float: return sum(data, 0.) / len(data)
  6. dict, Dict, Mapping dict, Dict, Mapping from typing import Dict,

    Mapping def read_data(slug: str) -> dict: ... def read_data(slug: str) -> Dict[str, str]: ... def process_data(data: Mapping[str, str]) -> None: for key, value in data.items(): ...
  7. Any, Union, Optional Any, Union, Optional from typing import Any,

    Optional, Sequence, Union def read_data(slug: str) -> Dict[str, Union[int, str]]: ... def read_data(slug: str) -> Dict[str, Any]: ... def avg(data: Sequence[float]) -> Optional[float]: if len(data): return sum(data, 0.) / len(data) return None
  8. Callable Callable from random import randint from typing import Callable

    def random_int_factory(min: int=0, max: int=100) -> Callable[[], int]: def random_int(): return randint(min, max) return random_int
  9. Aliases Aliases from aiohttp import web View = Callable[[web.Request], web.Response]

    def middleware() -> Callable[[web.Application, View], Awaitable[View]]: async def factory(app: web.Application, handler: View) -> View: async def middleware(request: web.Request) -> web.Response: ...
  10. Classes Classes class Schema(object): name = None # type: str

    def __init__(self, name: str=None) -> None: self.name = name or self.name def clone(self) -> 'Schema': return Schema(self.name) def load(self) -> bool: with open(SCHEMA_PATH / '{0}.json'.format(self.name)) as handler: return json.loads(handler.read()) def validate(self, data: Dict[str, Any]) -> Dict[str, Any]: json_schema = self.load() return fastjsonschema.compile(json_schema).validate(data)
  11. Types Types from typing import NewType UserID = NewType('UserID', int)

    def fetch_user(user_id: UserID) -> Any: ... fetch_user(UserID(1)) # OK fetch_user(1) # Type check will fail
  12. Inline Type Hints Inline Type Hints WEEKS = defaultdict(lambda: {

    False: list(range(8)), True: list(range(16)), }) # type: Dict[str, Dict[bool, List[int]]]
  13. Stubs Stubs package/module.py package/module.py def avg(data): if not len(data): return

    None return sum(data, 0.) / len(data) def avg_unique(data): return avg(unique(data)) def unique(data): return set(data)
  14. Stubs Stubs package/module.pyi package/module.pyi def avg(data: Sequence[float]) -> Optional[float]: ...

    def avg_unique(data: Sequence[float]) -> Optional[float]: ... def unique(data: Sequence[float]) -> Set[float]: ...
  15. Typeshed Typeshed python/typeshed Provide stubs for standard library And some

    widely-used shared libraries Have stubs for Python 2 & Python 3 Curated by PSF
  16. Python 2 Type Hints Python 2 Type Hints def hello(name):

    # type: (name: str) -> str return 'Hello, {0}'.format(name) def multi_line_annotations(address, # type: Union[str, List[str]] sender, # type: str subject, # type: str body # type: str ): # type: (...) -> bool ... More info at mypy docs.
  17. Introduction Introduction Static type checker for Python Static type checker

    for Python Works with Python 3 & Python 2 code Still experimental Still experimental Developed at Dropbox Again. Endorsed by Guido
  18. Configuration. Step 1 Configuration. Step 1 mypy.ini mypy.ini [mypy] fast_parser

    = True check_untyped_defs = True warn_redundant_casts = True
  19. Configuration. Step 2 Configuration. Step 2 mypy.ini mypy.ini silent_imports =

    True [mypy] fast_parser = True check_untyped_defs = True warn_redundant_casts = True
  20. Configuration. Step 3 Configuration. Step 3 mypy.ini mypy.ini disallow_untyped_defs =

    True [mypy] fast_parser = True silent_imports = True check_untyped_defs = True warn_redundant_casts = True
  21. Output Output env/bin/mypy project/ project/signals.py: note: In function "cache_api_urls": project/signals.py:25:

    error: Argument 2 to "api_url" has incompatible type "str"; expected "int"
  22. No Hype No Hype mypy is not widely used No

    viable benefits for users Hard to migrate large codebase to type hints
  23. Long Lines / Ugly Code Long Lines / Ugly Code

    Before Before async def retrieve_tweets(pool, count=50): ... A er A er async def retrieve_tweets( pool: Pool, count: int=50 ) -> Sequence[Mapping[str, Any]]: ...
  24. Stubs Stubs Hard to maintain Hard to maintain Easy to

    get into situation, when implementation != stub Completely same problems as with tests & documentation
  25. Incomplete Stubs Incomplete Stubs import asyncio def main() -> int:

    loop = asyncio.get_event_loop() tasks = [ asyncio.ensure_future(some_async_def), asyncio.ensure_future(some_other_async_def), ] loop.run_until_complete(asyncio.gather(*tasks, loop=loop)) loop.close() return 0
  26. Incomplete Stubs Incomplete Stubs $ mypy -c '...' <string>: note:

    In function "main": <string>:6: error: "module" has no attribute "ensure_future" <string>:7: error: "module" has no attribute "ensure_future" <string>:9: error: "module" has no attribute "gather" <string>:6: error: Name 'some_async_def' is not defined <string>:7: error: Name 'some_other_async_def' is not defined
  27. Incomplete Stubs Incomplete Stubs from lxml import etree def fetch_data(url:

    str) -> etree._Element: ... def use_data(): data = fetch_data(URL) for item in data.iterfind(...): # Will fail with "has no attribute" error ...
  28. # noqa: F401 # noqa: F401 from typing import Dict,

    List, Set def process_data(data: Dict[str, str]) -> List[int]: uniques = set() # type: Set[int] for value in data.values(): uniques.add(int(value)) return list(uniques) module.py:1:1: F401 'typing.Set' imported but unused
  29. Solution to Solution to # noqa: F401 # noqa: F401

    PEP-526 implements syntax for variable annotations. uniques: Set[int] = set() number: int # Works even without assignment class Schema(object): name: str data: Dict[str, str] = {} Included in Python 3.6 Included in Python 3.6
  30. Circular Imports Circular Imports project/models.py project/models.py from .utils import some_func

    class Model(object): def shortcut_for_some_func(self) -> int: return some_func(self) project/utils.py project/utils.py import typing if typing.TYPE_CHECKING: from .models import Model def some_func(model: 'Model') -> int: ...
  31. # type: ignore # type: ignore Sooner or later, but

    you'll need to use # type: ingore asyncio.gather(*tasks, loop=loop) # type: ignore for item in data.iterfind('...'): # type: ignore mypy is still experimental and you'll need to use # type: ignore a er yelling WTF !
  32. pytype pytype google/pytype Type checker from Google Needs Python 2.7

    to run Able to type check Python 3 code though
  33. enforce enforce RussBaz/enforce Runtime type checking Designed to use at

    tests or for data validation import enforce @enforce.runtime_validation def hello(name: str) -> str: return 'Hello, {0}!'.format(name) hello('world') hello(1) # Will fail with RuntimeTypeError