Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Getting Started with Statically Typed Programming in Python 3.10

Getting Started with Statically Typed Programming in Python 3.10

Peacock

July 29, 2021
Tweet

More Decks by Peacock

Other Decks in Programming

Transcript

  1. Getting Started with
    Statically Typed
    Programming in Python 3.10
    Peacock (Yoichi Takai), at EuroPython 2021

    View Slide

  2. Prolog
    Self-introduction, Table of contents
    2 / 49

    View Slide

  3. Slide is uploaded. You can see it via QR code
    or, via URL
    Chat(Element): #Conference 1: Optiver
    If you have a Combined or Conference Ticket, you can
    PEP8 styles in the sample code are ignored
    Due to space limitations. I'm sorry for hard to see
    Notes
    3 / 49

    View Slide

  4. Name: Peacock / Yoichi Takai
    Twitter / GitHub / Facebook: peacock0803sz
    Please call me Peacock
    I'm Attending from Japan, now it's 17:30 in JST
    Thanks for considering the timezone!
    Favourites: Music (Most is classical), Skiing, Gadgets
    Self-introduction
    4 / 49

    View Slide

  5. Company: CMScom (Full-time since 2020/06 ~)
    We're the only company in Japan that uses Plone
    Member of PloneJP (Plone User's Group in Japan)
    We created a short video to introducing Plone 6!
    https:/
    /youtu.be/CtpccSyRJaY
    Operating Member of PyCon JP Association
    Staff of PyCon JP 2020, 2021
    PyCon JP TV's director
    YouTube live about PyCons and local events held once a
    month
    5 / 49

    View Slide

  6. 1. Why do I talk about typing?
    2. Introduction of typing, How to write basically (I most want to say)
    3. Generics, User-Defined types (Best practice included)
    4. Updates overview recently & Backward compatibility for 3.9 or before
    5. Overview of new features on 3.10
    Today's topic
    6 / 49

    View Slide

  7. It's been five years since typing appeared
    In Python 3.5, at 2015
    Several big PEPs were adopted and updated over the years
    Even now, I think many people don't know where to start
    Because there is little coherent information
    Why do I talk about typing?
    7 / 49

    View Slide

  8. Developing library with typing
    Configures and options of mypy
    How to use them in CI, ex: GitHub actions, Circle CI and etc...
    History of type hinting
    Implementation of typing, mypy
    Abstract Syntax Tree (AST)
    I will not talk about
    8 / 49

    View Slide

  9. Introduction of typing
    How to write basically
    9 / 49

    View Slide

  10. It knows the type when you reference it in the editor.
    It gets angry when I try to give it the wrong one.
    The completion will work when accessing the return value of a
    function using a dot.
    What makes you happy?
    10 / 49

    View Slide

  11. Without the type hint
    We don't know the error...
    11 / 49

    View Slide

  12. With the type hint
    Editor tells a wrong argument
    12 / 49

    View Slide

  13. The reviewer can know variables or function returns types.
    Without the type hint
    Boss < What type does this function return?

    You < Humm... str or False or None ...?

    Boss < THAT'S TOO MANY TYPES!

    You < :-(
    def need_new_post():
    if ...: retrun None

    elif ...: retrun False

    else: return post_id # this is str

    In a code review
    13 / 49

    View Slide

  14. With the type hint
    Boss < It looks like this function may return 3 types... Isn't that too
    much?

    You < I see. That could be a bad design. Let me fix it.

    Boss < Sure, please.
    def need_new_post() -> None | False | str:

    if ...: retrun None

    elif ...: retrun False

    else: return post_id # this is str

    14 / 49

    View Slide

  15. After the arguments, write colon and type
    Before the colon at the end of the function definition, write an
    arrow and type
    Let's start with function definitions
    15 / 49

    View Slide

  16. Using built-in types
    bool , bytes , float , int , str
    you don't need to do anything to use them.
    None : used for functions that return nothing.
    Escaping from type puzzles
    Any Can hold instances of any type.
    It's better not to use it.
    Import and use from typing when necessary.
    from typing import Any

    unknown_variable: Any

    16 / 49

    View Slide

  17. dict , frozenset , list , set , tuple
    Collections can be written with [] for the type inside.
    3.9 and later only
    3.7, 3.8 write from __future__ import annotaions (see
    below)
    3.6: import annotations starting with uppercase letters
    from typing (next section)
    ref: official documentation (English)
    Since 3.9: Generics in standard Collections
    17 / 49

    View Slide

  18. Until 3.8, it was from typing , but now it's depreciated.
    For __builtins__ start with lowercase without doing anything.
    Such as list , tuple , and dict etc...
    For collections (ex: deque, defaultdict, ...), import modules start
    with collections
    iterable, callable, and other protocol-related items import
    modules start with collections.abc .
    regular expressions from re .
    Context-related items are available in contextlib .
    18 / 49

    View Slide

  19. For Generics, until 3.9, you had to write from typing import ...
    Such as Dict , List and Tuple etc...
    From 3.9, it's deprecated.
    from typing import Dict, List, Tuple, ... # before 3.9

    def some_function() -> Tuple[List[int], Dict[str, bool]]: pass

    Since 3.9, no more need these import statement!
    def some_function() -> tuple[list[int], dict[str, bool]]: pass

    (Deprecated since 3.9) import from typing
    module
    19 / 49

    View Slide

  20. There are many types in collections.abc .
    It's better to use a collection with a few methods to increase
    portability.
    The following figure shows the relationship.
    The further to the left you go, the fewer methods it has.
    To the right, the more methods it has.
    It's a good idea to look at the methods used in your functions.
    Choose the types on the left side of this diagram as much as
    possible.
    Using different types of collections
    20 / 49

    View Slide

  21. Great method inheritance tree
    21 / 49

    View Slide

  22. Tuples are fixed up to the length information
    Specify the type for the number of elements
    Or you can mix types, such as tuple[int, str, float].
    A sequence, such as a list, has the same constraint for all
    elements in the element
    Can be used regardless of the length of the sequence by
    setting only one element.
    The difference between tuple and others
    Sequences
    22 / 49

    View Slide

  23. A little more advanced: Generics type
    23 / 49

    View Slide

  24. Union : merged type, can be represented by | since 3.10
    You've probably seen it on Haskell or TypeScript
    from __future__ import annotations

    def square(number: int | float) -> int | float:

    return number ** 2

    Union objects can be tested for equality with other union objects.
    (int | str) | float == int | str | float # Unions of unions are flattened

    int | str | int == int | str # Redundant types are removed

    int | str == str | int # the order is ignored
    int | str == typing.Union[int, str] # Compatible with typing.Union
    Union (Mager type)
    24 / 49

    View Slide

  25. Shorthand, Optional[T] is equivalent to Union with None.
    Behaves just like Union: T | None
    If you use it in a function return value or something, it will
    propagate, so be careful how you use it.
    from typing import Optional

    age: Optional[int]

    age = 17
    age = None # This is also valid

    Optional type
    25 / 49

    View Slide

  26. Optional is useful but causes code bloat.
    def get_content() -> str | None:
    r = request.get("https://example.com")

    if r.status_code != 200: # This is the guard (early return)

    logging.warning("HTTP response is %d!", r.status_code)

    return None

    return r.text

    When you use the up function, you might write another guard and
    return None .
    As a result, we need to write a guard to the previous method,
    which reduces readability.
    Avoid using Optional as much as possible
    26 / 49

    View Slide

  27. In this case
    It would be cleaner to raise a raise RuntimeError .
    The cost of raising exceptions in Python is (relatively) low
    The performance would be satisfactory.
    The lack of null-safe methods in Python is also a factor
    But if there were such methods, they would be abused.
    Null-safe means a method that does not raise an exception
    when passed None.
    27 / 49

    View Slide

  28. It can be used when writing functions that take a function as an
    argument, such as decorator functions.
    from collections.abc import Callable # since 3.9
    from typing import Callable # 3.8 and earlier

    from fuctools import wraps

    def validate(func: Callable) -> Callable[... , Callable | tuple[Response, Literal[400]]]:

    @wraps(func)
    def wrapper(*args, **kw) -> Callable | tuple[Response, Literal[400]]:

    try:
    j = request.json
    if j is None: raise BadRequest

    except BadRequest:

    return jsonify({"data": [], "errors": {"message": ERROR_MESSAGE, "code": 400}}), 400
    return func(*args, **kw)
    return wrapper

    Callable (callable object)
    28 / 49

    View Slide

  29. A generic type is typically declared by inheriting from an instantiation.
    Example: a generic mapping type
    from typing import TypeVar, Generic

    KT, VT = TypeVar("KT"), TypeVar("VT")

    class Mapping(Generic[KT, VT]):

    def __getitem__(self, key: KT) -> VT: pass

    This class can then be used as:
    X, Y = TypeVar("X"), TypeVar("Y")

    def lookup_name(mapping: Mapping[X, Y], key: X, default: Y) -> Y:

    try: return mapping[key]
    except KeyError: return default

    User-defined Generic types
    29 / 49

    View Slide

  30. Introducing and promotion from
    PyCon JP
    30 / 49

    View Slide

  31. PyCon JP 2020 was held online!
    31 / 49

    View Slide

  32. We are the biggest Python conference in Japan.
    Website: https:/
    /2021.pycon.jp/
    Blog: https:/
    /pyconjp.blogspot.com/
    Twitter: @pyconjapan
    Date(conference day): 10/15(Fri), 16(Sat)
    Sprint and Training is not determined
    Now, CfP is over
    In the process of review and adoption
    Announces about PyCon JP 2021 (1/2)
    32 / 49

    View Slide

  33. Venue: Online or Hybrid:
    On-site venue is Bellesalle Kanda, Tokyo
    10/15(Fri) starts in the afternoon
    It'll be available only in the afternoon
    It may change depending on the COVID-19 situation
    10/16(Sat): All online
    Sponsors application (second) is open: Blog post
    For the latest information, check our blog and Twitter
    Share this slide page with more Pythonistas around you!
    Announces about PyCon JP 2021 (2/2)
    33 / 49

    View Slide

  34. Updates Overview & How to use new
    features in older versions
    34 / 49

    View Slide

  35. https:/
    /www.python.org/downloads/
    ver. status release EOS PEP main new feature
    3.10 beta 4 2021-10-04 2026-10 619 Pattern matching
    3.9 bug fix 2020-10-05 2025-10 596 Union operators to dict
    3.8 security 2019-10-14 2024-10 569 = in f-string
    3.7 Security 2018-06-27 2023-06-27 537 Data classes
    3.6 Security 2016-12-23 2021-12-23 494 Literal string (f-string)
    Recent Python updates
    35 / 49

    View Slide

  36. It exists for backward compatibility.
    Using typing new feature in the older versions, write from
    __future__ import annotations
    It describes when disruptive changes are introduced and become
    mandatory.
    In addition to typing, it was also used to call 3.x features in 2.x.
    ex) print_func , unicode_literals etc ...
    refs: __future__, future statement
    What is the __future__ module: (dunder
    future)?
    36 / 49

    View Slide

  37. New Features Related to Type Hints
    in 3.10
    37 / 49

    View Slide

  38. The union above type can be used as an operator.
    You can also use it when asking isinstance() .
    More intuitive since TypeScipt and others use this notation.
    int | str == typing.Union[int, str] # Compatible with typing.Union

    PEP 604: New Type Union Operator
    38 / 49

    View Slide

  39. THIS TOPIC IS DIFFICULT!!!
    Motivation
    Tring to write a generic decorator, it's difficult to write the type
    Needed a way to represent a function that has the same
    arguments as the specified function
    PEP 612: Parameter Specification Variables
    39 / 49

    View Slide

  40. Approach
    Adding an argument type called ParameterSpecification solves
    the problem.
    It can be used with Callable to behave like a generic callable
    object.
    You can think of it as an argument version of TypeVar .
    40 / 49

    View Slide

  41. Example
    from typing import Callable, ParameterSpecification, TypeVar
    Ps, R = ParameterSpecification("Ps"), TypeVar("R")

    def add_logging(f: Callable[Ps, R]) -> Callable[Ps, R]:

    def inner(*args: Ps.args, **kwargs: Ps.kwargs) -> R:
    log_to_database()

    return f(*args, **kwargs)

    return inner
    @add_logging
    def foo(x: int, y: str) -> int: return x + 7
    41 / 49

    View Slide

  42. Motivation
    We consider global variables without type hints to be type aliases.
    This tends to cause problems with forwarding references,
    scoping, etc.
    So, we're going to make it possible to explicitly define type
    aliases.
    You can still define type aliases implicitly.
    PEP 613: TypeAlias
    42 / 49

    View Slide

  43. Approach
    Add a new typing.TypeAlias
    Write a variable of type alias type like T: TypeAlias = int
    Variables defined at the global level are considered type aliases.
    Using ForwardReference, you can write T: TypeAlias = "int" .
    Example
    x = 1 # untyped global expression

    x: int = 1 # typed global expression

    x = int # untyped global expression
    x: Type[int] = int # typed global expression

    x: TypeAlias = int # type alias
    x: TypeAlias = “MyClass” # type alias

    43 / 49

    View Slide

  44. Motivation
    Type checker tools use a technique called type narrowing to determine
    the type of information.

    In this example, the if statement and is None are used to
    automatically narrow down the type.
    def func(val: Optional[str]):

    # "is None" type guard

    if val is not None: # Type of val is narrowed to str
    pass
    else: # Type of val is narrowed to None

    pass
    PEP 647: User-Defined Type Guards
    44 / 49

    View Slide

  45. However, that will not work as intended if the user function is used.
    def is_str_list(val: List[object]) -> bool:

    """Determines whether all objects in the list are strings"""
    return all(isinstance(x, str) for x in val)

    def func1(val: List[object]):

    if is_str_list(val): print(" ".join(val)) # Error: invalid type
    TypeGuard allows you to define user-defined type guards via the
    new typing.
    By using user-defined type guards, it is easier to get support for
    type narrowing.
    45 / 49

    View Slide

  46. from typing import TypeGuard
    def is_str_list(val: List[object]) -> TypeGuard[List[str]]:

    return all(isinstance(x, str) for x in val) # this is vaild!

    And, type narrowing works like this:
    def is_two_element_tuple(val: Tuple[str, ...]) -> TypeGuard[Tuple[str, str]]:

    return len(val) == 2
    OneOrTwoStrs = Union[Tuple[str], Tuple[str, str]]

    def func(val: OneOrTwoStrs):
    if is_two_element_tuple(val): reveal_type(val) # Tuple[str, str]

    else: reveal_type(val) # OneOrTwoStrs

    46 / 49

    View Slide

  47. 1. Introduction
    i. Motivation, Let's start writing, Built-in types
    ii. Standard collection type hints starting with lowercase (3.9)
    2. Collections and Generics
    i. Union, Optional, Callable, User-defined Generics
    3. Updates Overview & How to use new features in older versions
    4. Python 3.10 style type hinting
    i. New Type Union Operator, Parameter Specific Variables,
    TypeAlias, User-Defined Type Guards
    Summary
    47 / 49

    View Slide

  48. https:/
    /docs.python.org/3/library/typing.html
    https:/
    /docs.python.org/3.10/whatsnew/3.10.html
    http:/
    /mypy-lang.org
    https:/
    /future-architect.github.io/articles/20201223 (ja)
    https:/
    /qiita.com/tk0miya/items/931da13af292060356b9 (ja)
    https:/
    /qiita.com/tk0miya/items/1b093c2eee7668fffb62 (ja)
    https:/
    /qiita.com/tk0miya/items/a27191f5d627601930ed (ja)
    Pages I used for reference (Thanks)
    48 / 49

    View Slide

  49. We look forward to seeing you
    again at PyCon JP 2021!
    49 / 49

    View Slide