Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Oh my py. Type hints in Python

Oh my py. Type hints in Python

Slides for Dnepr.py#5 talk about type hints / type annotations in Python 3 & mypy. How to use them, when and what the common problems

Igor Davydenko

November 02, 2016
Tweet

More Decks by Igor Davydenko

Other Decks in Programming

Transcript

  1. Oh my py
    Oh my py
    !
    !
    Igor Davydenko
    2016, Dnepr.py#5

    View full-size slide

  2. Type hints
    Type hints
    in Python
    in Python
    And how to use them
    And how to use them
    Igor Davydenko
    2016, Dnepr.py#5

    View full-size slide

  3. Brief History aka
    Brief History aka PEP-484
    PEP-484
    Designed on top of PEP-3107 (Function Annotations)
    Created on September 2014
    Became part of standard library in Python 3.5
    Endorsed by Guido van Rossum

    View full-size slide

  4. Basics
    Basics
    def hello(text: str) -> str:
    return 'Hello, {0}'.format(text)
    def sum(a: int, b: int) -> int:
    return a + b

    View full-size slide

  5. Basics
    Basics
    from decimal import Decimal, ROUND_UP
    PRECISION = Decimal('.01')
    def quantize_payment(hours: float, rate: Decimal) -> Decimal:
    return (Decimal(hours) * rate).quantize(PRECISION, ROUND_UP)

    View full-size slide

  6. str, bytes, Text, AnyStr
    str, bytes, Text, AnyStr
    from typing import AnyStr, Text
    def response(body: bytes=None, text: str=None) -> bytes:
    ...
    def response(content: AnyStr=None) -> bytes:
    ...
    def render_template(path: Text) -> str:
    ...

    View full-size slide

  7. list, List, Tuple, Set, Sequence
    list, List, Tuple, Set, Sequence
    from typing import List, Sequence, Set, Tuple
    def avg(data: list) -> float:
    return sum(data, 0.) / len(data)
    def avg(data: List[float]) -> float:
    return sum(data, 0.) / len(data)
    def split_name(value: str) -> Tuple[str, str]:
    return tuple(value.split(' ', 1))
    def avg_tuple(data: Tuple[float, ...]) -> float:
    return sum(data, 0.) / len(data)
    def unique(data: List[str]) -> Set[str]:
    return set(data)
    def avg(data: Sequence[float]) -> float:
    return sum(data, 0.) / len(data)

    View full-size slide

  8. dict, Dict, Mapping
    dict, Dict, Mapping
    from typing import Dict, Mapping
    def read_data(slug: str) -> dict:
    ...
    def read_data(slug: str) -> Dict[str, str]:
    ...
    def process_data(data: Mapping[str, str]) -> None:
    for key, value in data.items():
    ...

    View full-size slide

  9. Any, Union, Optional
    Any, Union, Optional
    from typing import Any, Optional, Sequence, Union
    def read_data(slug: str) -> Dict[str, Union[int, str]]:
    ...
    def read_data(slug: str) -> Dict[str, Any]:
    ...
    def avg(data: Sequence[float]) -> Optional[float]:
    if len(data):
    return sum(data, 0.) / len(data)
    return None

    View full-size slide

  10. Callable
    Callable
    from random import randint
    from typing import Callable
    def random_int_factory(min: int=0, max: int=100) -> Callable[[], int]:
    def random_int():
    return randint(min, max)
    return random_int

    View full-size slide

  11. Aliases
    Aliases
    from aiohttp import web
    View = Callable[[web.Request], web.Response]
    def middleware() -> Callable[[web.Application, View], Awaitable[View]]:
    async def factory(app: web.Application, handler: View) -> View:
    async def middleware(request: web.Request) -> web.Response:
    ...

    View full-size slide

  12. Classes
    Classes
    class Schema(object):
    name = None # type: str
    def __init__(self, name: str=None) -> None:
    self.name = name or self.name
    def clone(self) -> 'Schema':
    return Schema(self.name)
    def load(self) -> bool:
    with open(SCHEMA_PATH / '{0}.json'.format(self.name)) as handler:
    return json.loads(handler.read())
    def validate(self, data: Dict[str, Any]) -> Dict[str, Any]:
    json_schema = self.load()
    return fastjsonschema.compile(json_schema).validate(data)

    View full-size slide

  13. Types
    Types
    from typing import NewType
    UserID = NewType('UserID', int)
    def fetch_user(user_id: UserID) -> Any:
    ...
    fetch_user(UserID(1)) # OK
    fetch_user(1) # Type check will fail

    View full-size slide

  14. Inline Type Hints
    Inline Type Hints
    WEEKS = defaultdict(lambda: {
    False: list(range(8)),
    True: list(range(16)),
    }) # type: Dict[str, Dict[bool, List[int]]]

    View full-size slide

  15. Stubs
    Stubs
    package/module.py
    package/module.py
    def avg(data):
    if not len(data):
    return None
    return sum(data, 0.) / len(data)
    def avg_unique(data):
    return avg(unique(data))
    def unique(data):
    return set(data)

    View full-size slide

  16. Stubs
    Stubs
    package/module.pyi
    package/module.pyi
    def avg(data: Sequence[float]) -> Optional[float]: ...
    def avg_unique(data: Sequence[float]) -> Optional[float]: ...
    def unique(data: Sequence[float]) -> Set[float]: ...

    View full-size slide

  17. Typeshed
    Typeshed
    python/typeshed
    Provide stubs for standard library
    And some widely-used shared libraries
    Have stubs for Python 2 & Python 3
    Curated by PSF

    View full-size slide

  18. Python 2 Type Hints
    Python 2 Type Hints
    def hello(name): # type: (name: str) -> str
    return 'Hello, {0}'.format(name)
    def multi_line_annotations(address, # type: Union[str, List[str]]
    sender, # type: str
    subject, # type: str
    body # type: str
    ):
    # type: (...) -> bool
    ...
    More info at mypy docs.

    View full-size slide

  19. Introduction
    Introduction
    Static type checker for Python
    Static type checker for Python
    Works with Python 3 & Python 2 code
    Still experimental
    Still experimental
    Developed at Dropbox
    Again. Endorsed by Guido

    View full-size slide

  20. Usage
    Usage
    $ pip install mypy-lang typed-ast
    $ mypy ...
    $ mypy --fast-parser ...

    View full-size slide

  21. Configuration. Step 1
    Configuration. Step 1
    mypy.ini
    mypy.ini
    [mypy]
    fast_parser = True
    check_untyped_defs = True
    warn_redundant_casts = True

    View full-size slide

  22. Configuration. Step 2
    Configuration. Step 2
    mypy.ini
    mypy.ini
    silent_imports = True
    [mypy]
    fast_parser = True
    check_untyped_defs = True
    warn_redundant_casts = True

    View full-size slide

  23. Configuration. Step 3
    Configuration. Step 3
    mypy.ini
    mypy.ini
    disallow_untyped_defs = True
    [mypy]
    fast_parser = True
    silent_imports = True
    check_untyped_defs = True
    warn_redundant_casts = True

    View full-size slide

  24. Output
    Output
    env/bin/mypy project/
    project/signals.py: note: In function "cache_api_urls":
    project/signals.py:25: error: Argument 2 to "api_url" has incompatible type "str"; expected "int"

    View full-size slide

  25. Problems
    Problems

    View full-size slide

  26. No Hype
    No Hype
    mypy is not widely used
    No viable benefits for users
    Hard to migrate large codebase to type hints

    View full-size slide

  27. Long Lines / Ugly Code
    Long Lines / Ugly Code
    Before
    Before
    async def retrieve_tweets(pool, count=50):
    ...
    A er
    A er
    async def retrieve_tweets(
    pool: Pool,
    count: int=50
    ) -> Sequence[Mapping[str, Any]]:
    ...

    View full-size slide

  28. Stubs
    Stubs
    Hard to maintain
    Hard to maintain
    Easy to get into situation, when implementation != stub
    Completely same problems as with tests & documentation

    View full-size slide

  29. Incomplete Stubs
    Incomplete Stubs
    import asyncio
    def main() -> int:
    loop = asyncio.get_event_loop()
    tasks = [
    asyncio.ensure_future(some_async_def),
    asyncio.ensure_future(some_other_async_def),
    ]
    loop.run_until_complete(asyncio.gather(*tasks, loop=loop))
    loop.close()
    return 0

    View full-size slide

  30. Incomplete Stubs
    Incomplete Stubs
    $ mypy -c '...'
    : note: In function "main":
    :6: error: "module" has no attribute "ensure_future"
    :7: error: "module" has no attribute "ensure_future"
    :9: error: "module" has no attribute "gather"
    :6: error: Name 'some_async_def' is not defined
    :7: error: Name 'some_other_async_def' is not defined

    View full-size slide

  31. Incomplete Stubs
    Incomplete Stubs
    from lxml import etree
    def fetch_data(url: str) -> etree._Element:
    ...
    def use_data():
    data = fetch_data(URL)
    for item in data.iterfind(...): # Will fail with "has no attribute" error
    ...

    View full-size slide

  32. # noqa: F401
    # noqa: F401
    from typing import Dict, List, Set
    def process_data(data: Dict[str, str]) -> List[int]:
    uniques = set() # type: Set[int]
    for value in data.values():
    uniques.add(int(value))
    return list(uniques)
    module.py:1:1: F401 'typing.Set' imported but unused

    View full-size slide

  33. Solution to
    Solution to # noqa: F401
    # noqa: F401
    PEP-526 implements syntax for variable annotations.
    uniques: Set[int] = set()
    number: int # Works even without assignment
    class Schema(object):
    name: str
    data: Dict[str, str] = {}
    Included in Python 3.6
    Included in Python 3.6

    View full-size slide

  34. Circular Imports
    Circular Imports
    project/models.py
    project/models.py
    from .utils import some_func
    class Model(object):
    def shortcut_for_some_func(self) -> int:
    return some_func(self)
    project/utils.py
    project/utils.py
    import typing
    if typing.TYPE_CHECKING:
    from .models import Model
    def some_func(model: 'Model') -> int:
    ...

    View full-size slide

  35. # type: ignore
    # type: ignore
    Sooner or later, but you'll need to use # type: ingore
    asyncio.gather(*tasks, loop=loop) # type: ignore
    for item in data.iterfind('...'): # type: ignore
    mypy is still experimental and you'll need to use # type: ignore a er yelling WTF !

    View full-size slide

  36. Additional Notes
    Additional Notes

    View full-size slide

  37. pytype
    pytype
    google/pytype
    Type checker from Google
    Needs Python 2.7 to run
    Able to type check Python 3 code though

    View full-size slide

  38. enforce
    enforce
    RussBaz/enforce
    Runtime type checking
    Designed to use at tests or for data validation
    import enforce
    @enforce.runtime_validation
    def hello(name: str) -> str:
    return 'Hello, {0}!'.format(name)
    hello('world')
    hello(1) # Will fail with RuntimeTypeError

    View full-size slide

  39. mypy-django
    mypy-django
    machinails/mypy-django
    Type hints bindings for Django
    Experimental
    Not full coverage of Django internals

    View full-size slide

  40. Conclusion
    Conclusion
    "
    " #
    # #
    # $
    $ %
    % &
    & '
    '

    View full-size slide

  41. Questions?
    Questions?
    Twi er:
    Twi er: @playpausenstop
    @playpausenstop
    GitHub:
    GitHub: @playpauseandstop
    @playpauseandstop
    Made in Ukraine

    View full-size slide