Getting Started with Statically Typed Programming in Python 3.10 / PyCon APAC 2021

Transcript

1. Getting Started with Statically Typed Programming in Python 3.10 Peacock (Yoichi Takai), at PyCon APAC 2021
2. Prolog Self-introduction, Table of contents 2 / 49
3. Slide is uploaded. You can see it via QR code or, via Speaker Deck PEP8 styles in the sample code are ignored Due to space limitations. I'm sorry for hard to see Notes 3 / 49
4. Name: Peacock / Yoichi Takai Twitter / GitHub / Facebook: peacock0803sz Please call me Peacock I'm Attending from Japan, now it's 17:30 in JST Thanks for considering the timezone! Favourites: Music (Most is classical), Skiing, Gadgets Self-introduction 4 / 49
5. Company: CMScom (Full-time since 2020/06 ~) We're the only company in Japan that uses Plone Member of PloneJP (Plone User's Group in Japan) We created a short video to introducing Plone 6! https:/ /youtu.be/CtpccSyRJaY Operating Member of PyCon JP Association Staff of PyCon JP 2020, 2021 PyCon JP TV's director YouTube live about PyCons and local events held once a month 5 / 49
6. 1. Why do I talk about typing? 2. Introduction of typing, How to write basically (I most want to say) 3. Generics, User-Defined types (Best practice included) 4. Updates overview recently & Backward compatibility for 3.9 or before 5. Overview of new features on 3.10 Today's topic 6 / 49
7. It's been five years since typing appeared In Python 3.5, at 2015 Several big PEPs were adopted and updated over the years Even now, I think many people don't know where to start Because there is little coherent information Why do I talk about typing? 7 / 49
8. Developing library with typing Configures and options of mypy How to use them in CI, ex: GitHub actions, Circle CI and etc... History of type hinting Implementation of typing, mypy Abstract Syntax Tree (AST) I will not talk about 8 / 49
9. Introduction of typing How to write basically 9 / 49
10. It knows the type when you reference it in the editor. It gets angry when I try to give it the wrong one. The completion will work when accessing the return value of a function using a dot. What makes you happy? 10 / 49
11. Without the type hint We don't know the error... 11 / 49
12. With the type hint Editor tells a wrong argument 12 / 49
13. The reviewer can know variables or function returns types. Without the type hint Boss < What type does this function return? You < Humm... str or False or None ...? Boss < THAT'S TOO MANY TYPES! You < :-( def need_new_post(): if ...: retrun None elif ...: retrun False else: return post_id # this is str In a code review 13 / 49
14. With the type hint Boss < It looks like this function may return 3 types... Isn't that too much? You < I see. That could be a bad design. Let me fix it. Boss < Sure, please. def need_new_post() -> None | False | str: if ...: retrun None elif ...: retrun False else: return post_id # this is str 14 / 49
15. After the arguments, write colon and type Before the colon at the end of the function definition, write an arrow and type Let's start with function definitions 15 / 49
16. Using built-in types bool , bytes , float , int , str you don't need to do anything to use them. None : used for functions that return nothing. Escaping from type puzzles Any Can hold instances of any type. It's better not to use it. Import and use from typing when necessary. from typing import Any unknown_variable: Any 16 / 49
17. dict , frozenset , list , set , tuple Collections can be written with [] for the type inside. 3.9 and later only 3.7, 3.8 write from __future__ import annotaions (see below) 3.6: import annotations starting with uppercase letters from typing (next section) ref: official documentation (English) Since 3.9: Generics in standard Collections 17 / 49
18. Until 3.8, it was from typing , but now it's depreciated. For __builtins__ start with lowercase without doing anything. Such as list , tuple , and dict etc... For collections (ex: deque, defaultdict, ...), import modules start with collections iterable, callable, and other protocol-related items import modules start with collections.abc . regular expressions from re . Context-related items are available in contextlib . 18 / 49
19. For Generics, until 3.9, you had to write from typing import ... Such as Dict , List and Tuple etc... From 3.9, it's deprecated. from typing import Dict, List, Tuple, ... # before 3.9 def some_function() -> Tuple[List[int], Dict[str, bool]]: pass Since 3.9, no more need these import statement! def some_function() -> tuple[list[int], dict[str, bool]]: pass (Deprecated since 3.9) import from typing module 19 / 49
20. There are many types in collections.abc . It's better to use a collection with a few methods to increase portability. The following figure shows the relationship. The further to the left you go, the fewer methods it has. To the right, the more methods it has. It's a good idea to look at the methods used in your functions. Choose the types on the left side of this diagram as much as possible. Using different types of collections 20 / 49
21. Great method inheritance tree 21 / 49
22. Tuples are fixed up to the length information Specify the type for the number of elements Or you can mix types, such as tuple[int, str, float]. A sequence, such as a list, has the same constraint for all elements in the element Can be used regardless of the length of the sequence by setting only one element. The difference between tuple and others Sequences 22 / 49
23. A little more advanced: Generics type 23 / 49
24. Union : merged type, can be represented by | since 3.10 You've probably seen it on Haskell or TypeScript from __future__ import annotations def square(number: int | float) -> int | float: return number ** 2 Union objects can be tested for equality with other union objects. (int | str) | float == int | str | float # Unions of unions are flattened int | str | int == int | str # Redundant types are removed int | str == str | int # the order is ignored int | str == typing.Union[int, str] # Compatible with typing.Union Union (Mager type) 24 / 49
25. Shorthand, Optional[T] is equivalent to Union with None. Behaves just like Union: T | None If you use it in a function return value or something, it will propagate, so be careful how you use it. from typing import Optional age: Optional[int] age = 17 age = None # This is also valid Optional type 25 / 49
26. Optional is useful but causes code bloat. def get_content() -> str | None: r = request.get("https://example.com") if r.status_code != 200: # This is the guard (early return) logging.warning("HTTP response is %d!", r.status_code) return None return r.text When you use the up function, you might write another guard and return None . As a result, we need to write a guard to the previous method, which reduces readability. Avoid using Optional as much as possible 26 / 49
27. In this case It would be cleaner to raise a raise RuntimeError . The cost of raising exceptions in Python is (relatively) low The performance would be satisfactory. The lack of null-safe methods in Python is also a factor But if there were such methods, they would be abused. Null-safe means a method that does not raise an exception when passed None. 27 / 49
28. It can be used when writing functions that take a function as an argument, such as decorator functions. from collections.abc import Callable # since 3.9 from typing import Callable # 3.8 and earlier from fuctools import wraps def validate(func: Callable) -> Callable[... , Callable | tuple[Response, Literal[400]]]: @wraps(func) def wrapper(*args, **kw) -> Callable | tuple[Response, Literal[400]]: try: j = request.json if j is None: raise BadRequest except BadRequest: return jsonify({"data": [], "errors": {"message": ERROR_MESSAGE, "code": 400}}), 400 return func(*args, **kw) return wrapper Callable (callable object) 28 / 49
29. A generic type is typically declared by inheriting from an instantiation. Example: a generic mapping type from typing import TypeVar, Generic KT, VT = TypeVar("KT"), TypeVar("VT") class Mapping(Generic[KT, VT]): def __getitem__(self, key: KT) -> VT: pass This class can then be used as: X, Y = TypeVar("X"), TypeVar("Y") def lookup_name(mapping: Mapping[X, Y], key: X, default: Y) -> Y: try: return mapping[key] except KeyError: return default User-defined Generic types 29 / 49
30. Introducing and promotion from PyCon JP 30 / 49
31. PyCon JP 2020 was held online! 31 / 49
32. We are the biggest Python conference in Japan. Website: https:/ /2021.pycon.jp/ Blog: https:/ /pyconjp.blogspot.com/ Twitter: @pyconjapan Date(conference day): 10/15(Fri), 16(Sat) Sprint and Training is not determined Now, CfP is over In the process of review and adoption Announces about PyCon JP 2021 (1/2) 32 / 49
33. Venue: Online or Hybrid: On-site venue is Bellesalle Kanda, Tokyo 10/15(Fri) starts in the afternoon It'll be available only in the afternoon It may change depending on the COVID-19 situation 10/16(Sat): All online Sponsors application (second) is open: Blog post For the latest information, check our blog and Twitter Share this slide page with more Pythonistas around you! Announces about PyCon JP 2021 (2/2) 33 / 49
34. Updates Overview & How to use new features in older versions 34 / 49
35. https:/ /www.python.org/downloads/ ver. status release EOS PEP main new feature 3.10 beta 4 2021-10-04 2026-10 619 Pattern matching 3.9 bug fix 2020-10-05 2025-10 596 Union operators to dict 3.8 security 2019-10-14 2024-10 569 = in f-string 3.7 Security 2018-06-27 2023-06-27 537 Data classes 3.6 Security 2016-12-23 2021-12-23 494 Literal string (f-string) Recent Python updates 35 / 49
36. It exists for backward compatibility. Using typing new feature in the older versions, write from __future__ import annotations It describes when disruptive changes are introduced and become mandatory. In addition to typing, it was also used to call 3.x features in 2.x. ex) print_func , unicode_literals etc ... refs: __future__, future statement What is the __future__ module: (dunder future)? 36 / 49
37. New Features Related to Type Hints in 3.10 37 / 49
38. The union above type can be used as an operator. You can also use it when asking isinstance() . More intuitive since TypeScipt and others use this notation. int | str == typing.Union[int, str] # Compatible with typing.Union PEP 604: New Type Union Operator 38 / 49
39. THIS TOPIC IS DIFFICULT!!! Motivation Tring to write a generic decorator, it's difficult to write the type Needed a way to represent a function that has the same arguments as the specified function PEP 612: Parameter Specification Variables 39 / 49
40. Approach Adding an argument type called ParameterSpecification solves the problem. It can be used with Callable to behave like a generic callable object. You can think of it as an argument version of TypeVar . 40 / 49
41. Example from typing import Callable, ParameterSpecification, TypeVar Ps, R = ParameterSpecification("Ps"), TypeVar("R") def add_logging(f: Callable[Ps, R]) -> Callable[Ps, R]: def inner(*args: Ps.args, **kwargs: Ps.kwargs) -> R: log_to_database() return f(*args, **kwargs) return inner @add_logging def foo(x: int, y: str) -> int: return x + 7 41 / 49
42. Motivation We consider global variables without type hints to be type aliases. This tends to cause problems with forwarding references, scoping, etc. So, we're going to make it possible to explicitly define type aliases. You can still define type aliases implicitly. PEP 613: TypeAlias 42 / 49
43. Approach Add a new typing.TypeAlias Write a variable of type alias type like T: TypeAlias = int Variables defined at the global level are considered type aliases. Using ForwardReference, you can write T: TypeAlias = "int" . Example x = 1 # untyped global expression x: int = 1 # typed global expression x = int # untyped global expression x: Type[int] = int # typed global expression x: TypeAlias = int # type alias x: TypeAlias = “MyClass” # type alias 43 / 49
44. Motivation Type checker tools use a technique called type narrowing to determine the type of information. In this example, the if statement and is None are used to automatically narrow down the type. def func(val: Optional[str]): # "is None" type guard if val is not None: # Type of val is narrowed to str pass else: # Type of val is narrowed to None pass PEP 647: User-Defined Type Guards 44 / 49
45. However, that will not work as intended if the user function is used. def is_str_list(val: List[object]) -> bool: """Determines whether all objects in the list are strings""" return all(isinstance(x, str) for x in val) def func1(val: List[object]): if is_str_list(val): print(" ".join(val)) # Error: invalid type TypeGuard allows you to define user-defined type guards via the new typing. By using user-defined type guards, it is easier to get support for type narrowing. 45 / 49
46. from typing import TypeGuard def is_str_list(val: List[object]) -> TypeGuard[List[str]]: return all(isinstance(x, str) for x in val) # this is vaild! And, type narrowing works like this: def is_two_element_tuple(val: Tuple[str, ...]) -> TypeGuard[Tuple[str, str]]: return len(val) == 2 OneOrTwoStrs = Union[Tuple[str], Tuple[str, str]] def func(val: OneOrTwoStrs): if is_two_element_tuple(val): reveal_type(val) # Tuple[str, str] else: reveal_type(val) # OneOrTwoStrs 46 / 49
47. 1. Introduction i. Motivation, Let's start writing, Built-in types ii. Standard collection type hints starting with lowercase (3.9) 2. Collections and Generics i. Union, Optional, Callable, User-defined Generics 3. Updates Overview & How to use new features in older versions 4. Python 3.10 style type hinting i. New Type Union Operator, Parameter Specific Variables, TypeAlias, User-Defined Type Guards Summary 47 / 49
48. https:/ /docs.python.org/3/library/typing.html https:/ /docs.python.org/3.10/whatsnew/3.10.html http:/ /mypy-lang.org https:/ /future-architect.github.io/articles/20201223 (ja) https:/ /qiita.com/tk0miya/items/931da13af292060356b9 (ja) https:/ /qiita.com/tk0miya/items/1b093c2eee7668fffb62 (ja) https:/ /qiita.com/tk0miya/items/a27191f5d627601930ed (ja) Pages I used for reference (Thanks) 48 / 49
49. We look forward to seeing you again at PyCon JP 2021! 49 / 49