Oh my py. Type hints in Python

Slides for Dnepr.py#5 talk about type hints / type annotations in Python 3 & mypy. How to use them, when and what the common problems

Transcript

1. Oh my py
   Oh my py
   !
   !
   Igor Davydenko
   2016, Dnepr.py#5
   

   View Slide

2. Type hints
   Type hints
   in Python
   in Python
   And how to use them
   And how to use them
   Igor Davydenko
   2016, Dnepr.py#5
   

   View Slide

3. Brief History aka
   Brief History aka PEP-484
   PEP-484
   Designed on top of PEP-3107 (Function Annotations)
   Created on September 2014
   Became part of standard library in Python 3.5
   Endorsed by Guido van Rossum
   

   View Slide

4. Basics
   Basics
   def hello(text: str) -> str:
   return 'Hello, {0}'.format(text)
   def sum(a: int, b: int) -> int:
   return a + b
   

   View Slide

5. Basics
   Basics
   from decimal import Decimal, ROUND_UP
   PRECISION = Decimal('.01')
   def quantize_payment(hours: float, rate: Decimal) -> Decimal:
   return (Decimal(hours) * rate).quantize(PRECISION, ROUND_UP)
   

   View Slide

6. str, bytes, Text, AnyStr
   str, bytes, Text, AnyStr
   from typing import AnyStr, Text
   def response(body: bytes=None, text: str=None) -> bytes:
   ...
   def response(content: AnyStr=None) -> bytes:
   ...
   def render_template(path: Text) -> str:
   ...
   

   View Slide

7. list, List, Tuple, Set, Sequence
   list, List, Tuple, Set, Sequence
   from typing import List, Sequence, Set, Tuple
   def avg(data: list) -> float:
   return sum(data, 0.) / len(data)
   def avg(data: List[float]) -> float:
   return sum(data, 0.) / len(data)
   def split_name(value: str) -> Tuple[str, str]:
   return tuple(value.split(' ', 1))
   def avg_tuple(data: Tuple[float, ...]) -> float:
   return sum(data, 0.) / len(data)
   def unique(data: List[str]) -> Set[str]:
   return set(data)
   def avg(data: Sequence[float]) -> float:
   return sum(data, 0.) / len(data)
   

   View Slide

8. dict, Dict, Mapping
   dict, Dict, Mapping
   from typing import Dict, Mapping
   def read_data(slug: str) -> dict:
   ...
   def read_data(slug: str) -> Dict[str, str]:
   ...
   def process_data(data: Mapping[str, str]) -> None:
   for key, value in data.items():
   ...
   

   View Slide

9. Any, Union, Optional
   Any, Union, Optional
   from typing import Any, Optional, Sequence, Union
   def read_data(slug: str) -> Dict[str, Union[int, str]]:
   ...
   def read_data(slug: str) -> Dict[str, Any]:
   ...
   def avg(data: Sequence[float]) -> Optional[float]:
   if len(data):
   return sum(data, 0.) / len(data)
   return None
   

   View Slide

10. Callable
    Callable
    from random import randint
    from typing import Callable
    def random_int_factory(min: int=0, max: int=100) -> Callable[[], int]:
    def random_int():
    return randint(min, max)
    return random_int
    

    View Slide

11. Aliases
    Aliases
    from aiohttp import web
    View = Callable[[web.Request], web.Response]
    def middleware() -> Callable[[web.Application, View], Awaitable[View]]:
    async def factory(app: web.Application, handler: View) -> View:
    async def middleware(request: web.Request) -> web.Response:
    ...
    

    View Slide

12. Classes
    Classes
    class Schema(object):
    name = None # type: str
    def __init__(self, name: str=None) -> None:
    self.name = name or self.name
    def clone(self) -> 'Schema':
    return Schema(self.name)
    def load(self) -> bool:
    with open(SCHEMA_PATH / '{0}.json'.format(self.name)) as handler:
    return json.loads(handler.read())
    def validate(self, data: Dict[str, Any]) -> Dict[str, Any]:
    json_schema = self.load()
    return fastjsonschema.compile(json_schema).validate(data)
    

    View Slide

13. Types
    Types
    from typing import NewType
    UserID = NewType('UserID', int)
    def fetch_user(user_id: UserID) -> Any:
    ...
    fetch_user(UserID(1)) # OK
    fetch_user(1) # Type check will fail
    

    View Slide

14. Inline Type Hints
    Inline Type Hints
    WEEKS = defaultdict(lambda: {
    False: list(range(8)),
    True: list(range(16)),
    }) # type: Dict[str, Dict[bool, List[int]]]
    

    View Slide

15. Stubs
    Stubs
    package/module.py
    package/module.py
    def avg(data):
    if not len(data):
    return None
    return sum(data, 0.) / len(data)
    def avg_unique(data):
    return avg(unique(data))
    def unique(data):
    return set(data)
    

    View Slide

16. Stubs
    Stubs
    package/module.pyi
    package/module.pyi
    def avg(data: Sequence[float]) -> Optional[float]: ...
    def avg_unique(data: Sequence[float]) -> Optional[float]: ...
    def unique(data: Sequence[float]) -> Set[float]: ...
    

    View Slide

17. Typeshed
    Typeshed
    python/typeshed
    Provide stubs for standard library
    And some widely-used shared libraries
    Have stubs for Python 2 & Python 3
    Curated by PSF
    

    View Slide

18. Python 2 Type Hints
    Python 2 Type Hints
    def hello(name): # type: (name: str) -> str
    return 'Hello, {0}'.format(name)
    def multi_line_annotations(address, # type: Union[str, List[str]]
    sender, # type: str
    subject, # type: str
    body # type: str
    ):
    # type: (...) -> bool
    ...
    More info at mypy docs.
    

    View Slide

19. mypy
    mypy
    

    View Slide

20. Introduction
    Introduction
    Static type checker for Python
    Static type checker for Python
    Works with Python 3 & Python 2 code
    Still experimental
    Still experimental
    Developed at Dropbox
    Again. Endorsed by Guido
    

    View Slide

21. Usage
    Usage
    $ pip install mypy-lang typed-ast
    $ mypy ...
    $ mypy --fast-parser ...
    

    View Slide

22. Configuration. Step 1
    Configuration. Step 1
    mypy.ini
    mypy.ini
    [mypy]
    fast_parser = True
    check_untyped_defs = True
    warn_redundant_casts = True
    

    View Slide

23. Configuration. Step 2
    Configuration. Step 2
    mypy.ini
    mypy.ini
    silent_imports = True
    [mypy]
    fast_parser = True
    check_untyped_defs = True
    warn_redundant_casts = True
    

    View Slide

24. Configuration. Step 3
    Configuration. Step 3
    mypy.ini
    mypy.ini
    disallow_untyped_defs = True
    [mypy]
    fast_parser = True
    silent_imports = True
    check_untyped_defs = True
    warn_redundant_casts = True
    

    View Slide

25. Output
    Output
    env/bin/mypy project/
    project/signals.py: note: In function "cache_api_urls":
    project/signals.py:25: error: Argument 2 to "api_url" has incompatible type "str"; expected "int"
    

    View Slide

26. Problems
    Problems
    

    View Slide

27. No Hype
    No Hype
    mypy is not widely used
    No viable benefits for users
    Hard to migrate large codebase to type hints
    

    View Slide

28. Long Lines / Ugly Code
    Long Lines / Ugly Code
    Before
    Before
    async def retrieve_tweets(pool, count=50):
    ...
    A er
    A er
    async def retrieve_tweets(
    pool: Pool,
    count: int=50
    ) -> Sequence[Mapping[str, Any]]:
    ...
    

    View Slide

29. Stubs
    Stubs
    Hard to maintain
    Hard to maintain
    Easy to get into situation, when implementation != stub
    Completely same problems as with tests & documentation
    

    View Slide

30. Incomplete Stubs
    Incomplete Stubs
    import asyncio
    def main() -> int:
    loop = asyncio.get_event_loop()
    tasks = [
    asyncio.ensure_future(some_async_def),
    asyncio.ensure_future(some_other_async_def),
    ]
    loop.run_until_complete(asyncio.gather(*tasks, loop=loop))
    loop.close()
    return 0
    

    View Slide

31. Incomplete Stubs
    Incomplete Stubs
    $ mypy -c '...'
    : note: In function "main":
    :6: error: "module" has no attribute "ensure_future"
    :7: error: "module" has no attribute "ensure_future"
    :9: error: "module" has no attribute "gather"
    :6: error: Name 'some_async_def' is not defined
    :7: error: Name 'some_other_async_def' is not defined
    

    View Slide

32. Incomplete Stubs
    Incomplete Stubs
    from lxml import etree
    def fetch_data(url: str) -> etree._Element:
    ...
    def use_data():
    data = fetch_data(URL)
    for item in data.iterfind(...): # Will fail with "has no attribute" error
    ...
    

    View Slide

33. # noqa: F401
    # noqa: F401
    from typing import Dict, List, Set
    def process_data(data: Dict[str, str]) -> List[int]:
    uniques = set() # type: Set[int]
    for value in data.values():
    uniques.add(int(value))
    return list(uniques)
    module.py:1:1: F401 'typing.Set' imported but unused
    

    View Slide

34. Solution to
    Solution to # noqa: F401
    # noqa: F401
    PEP-526 implements syntax for variable annotations.
    uniques: Set[int] = set()
    number: int # Works even without assignment
    class Schema(object):
    name: str
    data: Dict[str, str] = {}
    Included in Python 3.6
    Included in Python 3.6
    

    View Slide

35. Circular Imports
    Circular Imports
    project/models.py
    project/models.py
    from .utils import some_func
    class Model(object):
    def shortcut_for_some_func(self) -> int:
    return some_func(self)
    project/utils.py
    project/utils.py
    import typing
    if typing.TYPE_CHECKING:
    from .models import Model
    def some_func(model: 'Model') -> int:
    ...
    

    View Slide

36. # type: ignore
    # type: ignore
    Sooner or later, but you'll need to use # type: ingore
    asyncio.gather(*tasks, loop=loop) # type: ignore
    for item in data.iterfind('...'): # type: ignore
    mypy is still experimental and you'll need to use # type: ignore a er yelling WTF !
    

    View Slide

37. Additional Notes
    Additional Notes
    

    View Slide

38. pytype
    pytype
    google/pytype
    Type checker from Google
    Needs Python 2.7 to run
    Able to type check Python 3 code though
    

    View Slide

39. enforce
    enforce
    RussBaz/enforce
    Runtime type checking
    Designed to use at tests or for data validation
    import enforce
    @enforce.runtime_validation
    def hello(name: str) -> str:
    return 'Hello, {0}!'.format(name)
    hello('world')
    hello(1) # Will fail with RuntimeTypeError
    

    View Slide

40. mypy-django
    mypy-django
    machinails/mypy-django
    Type hints bindings for Django
    Experimental
    Not full coverage of Django internals
    

    View Slide

41. Conclusion
    Conclusion
    "
    " #
    # #
    # $
    $ %
    % &
    & '
    '
    

    View Slide

42. Questions?
    Questions?
    Twi er:
    Twi er: @playpausenstop
    @playpausenstop
    GitHub:
    GitHub: @playpauseandstop
    @playpauseandstop
    Made in Ukraine
    

    View Slide