Getting Started with Statically Typed Programming in Python 3.10 / PyConUS2022

Transcript

1. Getting Started with Statically Typed Programming in Python 3.10 Peacock (Yoichi Takai), at PyCon US 2022 (2022/05/01)
2. Prolog Self-Introduction, Table of Contents
3. Slides: slides.peacock0803sz.com/pyconus2022/index.html Expected: Google Chrome Feel free to Take pictures or Screenshots and Tweet Hashtag: #pyconus2022 3 / 57 ` ` ^ Slides ^
4. Self-Introduction Name: Peacock / Yoichi Takai Social media IDs: peacock0803sz Twitter, GitHub, Facebook… Please call me Peacock 4 / 57 ` `
5. Hobbies & Favourites Playing the Clarinet Listening Music (Most is classical) Skiing , Gadgets… 21-years old, This is my first trip abroad 5 / 57
6. My Works and Communities Company: CMScom (Full-time since 2020/06 ~) Web application developer, Flask / Pyramid / Plone etc… Experience learning Haskell and TypeScript Now, reading "Types and Programming Languages" As known as "TaPL" Member of PloneJP (Plone User’s Group in Japan) 6 / 57
7. My Activities Related to PyCon JP Staff of PyCon JP (since 2020 - ), 2022: Vice-Chair Operating Member of PyCon JP Association Director of PyCon JP TV It’s the YouTube live about PyCons and local events held once a month 7 / 57
8. Today’s Topics 1. Why do I talk about typing? 2. First step of typing, How to write basically That is what I most want to say 3. Generics, User-Defined types (Best practice included) 4. Updates overview recently & Backward compatibility for 3.9 or before 5. Overview of new features on 3.10 8 / 57
9. Why do I talk about Typing? It’s been 7 years since typing appeared In Python 3.5, at 2015 Many big PEPs were adopted and updated over the years Even now, I think many people don’t know where to start Because there is little coherent information 9 / 57
10. What I will NOT talk about Developing library with typing Configures and options of mypy How to use them in CI, ex: GitHub actions, Circle CI and etc… History of type hinting Implementation of typing, mypy Abstract Syntax Tree (AST) 10 / 57
11. 11 / 57 First step of Typing Basic Typing
12. What makes you happy? It knows the type when you reference it in the editor It gets angry when I try to give it the wrong one The completion will work when accessing the return value of a function using a dot. 12 / 57
13. Without the Typing We don’t know the error… 13 / 57
14. With the Typing and, the editor can tell the argument is wrong Editor tells a wrong argument 14 / 57
15. In a code review, without Typing… Boss < What type does this function return? You < Humm… str or False or None …? Boss < THAT’S TOO MANY TYPES! You < :-( 4 else: return post_id # this is str 15 / 57 1 def need_new_post(): 2 if ...: retrun None 3 elif ...: retrun False
16. With Typing… Boss < It looks like this function may return 3 types… Isn’t that too much? You < I see. That could be a bad design. Let me fix it. Boss < Sure, please. 1 def need_new_post() -> None | False | str: 16 / 57 2 if ...: retrun None 3 elif ...: retrun False 4 else: return post_id # this is str
17. Let’s start with function definitions Nothing too difficult! Just write type After the arguments, write colon and the type Before the colon at the end of the function definition, write an arrow and the type 17 / 57
18. Using built-in types bool , bytes , float , int , str You don’t need to do anything to use them. None : Used for functions that return nothing. 18 / 57 ` ` ` ` ` ` ` ` ` ` ` `
19. Escaping from complex type puzzles Any Can hold instances of any type. It’s better not to use it Import and use from typing when necessary. 1 from typing import Any 2 very_dangerous_last_resort: Any 19 / 57 ` ` ` `
20. (Since 3.9) Generics in standard Collections Before 3.9, from typing import Dict, List … Now, it’s depreciated Use dict , frozenset , list , set , tuple Collections can be written with [] for the type inside. ref: Official Documentation 20 / 57 ` ` ` ` ` ` ` ` ` ` ` ` ` `
21. Module Availabilities in standard library… collections : (ex: deque, defaultdict) collections.abc : Iterable , Callable , or other Protocol-Related re : Regular Expressions contextlib : Context-Related items 21 / 57 ` ` ` ` ` ` ` ` ` ` ` `
22. Using types from collections.abc differently There are many types in collections.abc . It’s better to use a collection with a few methods to increase portability. Ex. Iterable is better than list . It’s a good idea to look at the methods used in your functions. The figure on the next page shows the relationship… 22 / 57 ` ` ` ` ` ` ` `
23. Great method inheritance tree 23 / 57
24. The differences between tuple and other Sequences tuple is fixed up to the length of information Specify the type for the number of elements Or you can mix types, such as tuple[int, str, float] . 24 / 57 ` ` ` `
25. A sequence, such as a list , has the same constraint for all elements Can be used regardless of the length of the Sequence by setting only one. 25 / 57 ` `
26. 26 / 57 Advanced: Generic Types
27. Union (multiple types) union can be represented by | since 3.10 1 def square(number: int | float) -> int | float: 7 # Compatible with typing.Union 8 int | str == typing.Union[int, str] 27 / 57 ` ` ` ` 2 return number ** 2 1 # Nested unions are flattened 2 (int | str) | float == int | str | float 3 # Redundant types are removed 4 int | str | int == int | str 5 # Orders are ignored 6 int | str == str | int
28. Optional type Shorthand, Optional[T] is equivalent to Union with None. Behaves just like Union: T | None If you use it in a function return value or something, it will propagate, so be careful when you use it. 3 age: Optional[int] 4 age = 17 5 age = None # This is also valid 28 / 57 ` ` ` ` 1 from typing import Optional 2
29. Avoid using Optional as much as possible 6 return None When you use the function, you might write another guard and return None . 29 / 57 1 def get_content() -> str | None: 2 r = request.get("https://example.com") 3 if r.status_code != 200: 4 # Guard section (early return) 5 logging.warning("Response: %d", r.status_code) 7 return r.text ` `
30. In this case… It would be cleaner to raise a raise RuntimeError . The cost of raising exceptions in Python is (relatively) low Little performance impact, unlike Java. The lack of null-safe methods in Python is also a factor, Unlike JavaScript. But, if there were such methods, they would be abused. 30 / 57 ` `
31. Callable (callable objects) 13 return func(*args, **kw) 31 / 57 1 from collections.abc import Callable # since 3.9 2 from functools import wraps 3 4 def validate(func: Callable 5 ) -> Callable[..., Callable]: 6 @wraps(func) 7 def wrapper(*args, **kw) -> Callable: 8 try: 9 j = request.json 10 if j is None: raise BadRequest 11 except BadRequest: 12 return abort(400) 14 return wrapper
32. User-defined Generic types Example: a generic mapping type 13 except KeyError: return default 32 / 57 1 from typing import TypeVar, Generic 2 KT, VT = TypeVar("KT"), TypeVar("VT") 3 4 class Mapping(Generic[KT, VT]): 5 def __getitem__(self, key: KT) -> VT: pass 6 7 X, Y = TypeVar("X"), TypeVar("Y") 8 9 def lookup_name( # Using Mapping 10 mapping: Mapping[X, Y], key: X, default: Y 11 ) -> Y: 12 try: return mapping[key]
33. 33 / 57 How to use new features in older versions
34. What is the __future__ module? Using typing new feature in the older versions, write from __future__ import annotations In addition to typing, it was also used to call 3.x features in 2.x. ex) print_func , unicode_literals etc … ref: __future__, future statement 34 / 57 ` ` ` ` ` ` ` `
35. 35 / 57 Recent Update
36. Recent Python updates Ver. Status Release EoS Schedule Main new feature 3.11 Alpha 7 2022-10 2027-10 PEP 619 Performance tuning 3.10 Bugfix 2021-10 2026-10 PEP 619 Pattern Matching 3.9 Bugfix 2020-10 2025-10 PEP 596 Union Operators 3.8 Security 2019-10 2024-10 PEP 569 = in f-string 3.7 Security 2018-06 2023-06 PEP 537 dataclasses 2.7 EoL 2010-07 2020-01 PEP 373 Improving Decimals https://www.python.org/downloads/ 36 / 57 ` `
37. New Features on Type Hints in 3.10 PEP 604: New Type Union Operator PEP 612: Parameter Specification Variables PEP 613: TypeAlias PEP 647: User-Defined Type Guards See Also: https://docs.python.org/3/whatsnew/3.10.html#new- features-related-to-type-hints 37 / 57
38. PEP 604: New Type Union Operator | The union above type can be used as an operator. You can also use it when asking isinstance() . 1 int | str == typing.Union[int, str] 2 # Compatible with typing.Union Optional[T] can also be written as T | None using this. 38 / 57 ` ` ` ` ` ` ` `
39. PEP 612: Parameter Specification Variables Motivation Needed a way to represent a function that has the same arguments as the specified function 1 from typing import Callable, TypeVar 2 R = TypeVar("R") 3 4 def add_logging(f: Callable[..., R] 5 ) -> Callable[..., R]: 6 def inner(*args: object, **kw: object) -> R: 7 log_to_database() 8 return f(*args, **kwargs) 9 return inner 39 / 57
40. 1 @add_logging 2 def takes_int_str(x: int, y: str) -> int: 3 return x + 7 4 5 await takes_int_str(1, "A") 6 await takes_int_str("B", 2) # fails at runtime Approach Adding an argument type called ParameterSpecification solves the problem. It can be used with Callable to behave like a generic callable object. 40 / 57 ` ` ` `
41. Before 11 12 @add_logging 13 def foo(x: int, y: str) -> int: 14 return x + 7 41 / 57 1 from typing import Callable, TypeVar 2 Ps, R = TypeVar("Ps"), TypeVar("R") 3 4 def add_logging(f: Callable[Ps, R] 5 ) -> Callable[Ps, R]: 6 # args: tuple...?, kwargs: dict...? 7 def inner(*args, **kwargs) -> R: 8 log_to_database(*args, **kwargs) 9 return f(*args, **kwargs) 10 return inner
42. After 10 11 @add_logging 12 def foo(x: int, y: str) -> int: return x + 7 42 / 57 1 from typing import Callable, ParameterSpecification, T 2 Ps, R = ParameterSpecification("Ps"), TypeVar("R") 3 4 def add_logging(f: Callable[Ps, R] 5 ) -> Callable[Ps, R]: 6 def inner(*args: Ps.args, **kwargs: Ps.kwargs) -> R: 7 log_to_database(*args, **kwargs) 8 return f(*args, **kwargs) 9 return inner
43. PEP 613: TypeAlias Motivation We consider global variables without type hints to be type aliases. This tends to cause problems with forwarding references, scoping, etc. So, we’re going to make it possible to explicitly define type aliases. 43 / 57
44. Approach Add a new typing.TypeAlias You can write like this T: TypeAlias = int You can write like: T: TypeAlias = "int" Using ForwardReference 44 / 57 ` ` ` ` ` `
45. Before & After 7 x: TypeAlias = int # type alias 8 x: TypeAlias = “MyClass” # type alias 45 / 57 1 x = 1 # untyped global expression 2 x: int = 1 # typed global expression 3 4 x = int # untyped global expression 5 x: Type[int] = int # typed global expression 6
46. PEP 647: User-Defined Type Guards Motivation Type checker tools use a technique called type narrowing to determine the type of information. 5 else: pass # Type of val is narrowed to None 46 / 57 1 def func(val: Optional[str]): 2 # "is None" type guard 3 if val is not None: 4 pass # Type of val is narrowed to str
47. However, that will not work as intended if the user function is used. 6 if is_str_list(val): print(" ".join(val)) TypeGuard allows you to define user-defined type guards via the new typing. 47 / 57 1 def is_str_list(val: List[object]) -> bool: 2 """Determines whether all objects in the list are st 3 return all(isinstance(x, str) for x in val) 4 5 def func1(val: List[object]): 7 # => Error: invalid type ` `
48. 1 from typing import TypeGuard 2 3 def is_str_list(val: List[object] 4 ) -> TypeGuard[List[str]]: 5 return all(isinstance(x, str) for x in val) And, type narrowing works like next: 48 / 57
49. 10 else: reveal_type(val) # OneOrTwoStrs 49 / 57 1 def is_two_element_tuple( 2 val: Tuple[str, ...] 3 ) -> TypeGuard[Tuple[str, str]]: 4 return len(val) == 2 5 6 OneOrTwoStrs = Union[Tuple[str], Tuple[str, str]] 7 def func(val: OneOrTwoStrs): 8 if is_two_element_tuple(val): 9 reveal_type(val) # Tuple[str, str]
50. New Feature on Type Hints in 3.11 Python 3.11 is focused on performance tuning, but there are also new features in Typing… PEP 673: Self Type, It’s a way to annotate methods that return an instance of their class 50 / 57
51. Motivation Calling set_scale on a subclass of Shape , the type checker still infers the return type to be Shape . 7 Circle().set_scale(0.5).set_radius(2.7) 8 # => error: Shape has no attribute "set_radius" 51 / 57 ` ` ` ` ` ` 1 class Circle(Shape): 2 def set_radius(self, r: float) -> Circle: 3 self.radius = r 4 return self 5 6 Circle().set_scale(0.5) # *Shape*, not Circle
52. Workaround, but it’s unintuitive… 15 Circle().set_scale(0.5).set_radius(2.7) # => Circle 52 / 57 1 from typing import TypeVar 2 TShape = TypeVar("TShape", bound="Shape") 3 4 class Shape: 5 def set_scale(self: TShape, scale: float 6 ) -> TShape: 7 self.scale = scale 8 return self 9 10 class Circle(Shape): 11 def set_radius(self, radius: float) -> Circle: 12 self.radius = radius 13 return self 14
53. typing.Self resolves that unintuitively 12 Circle().set_scale(0.5) # => It would be *Circle*! 53 / 57 ` ` 1 from typing import Self 2 class Shape: 3 def set_scale(self, scale: float) -> Self: 4 self.scale = scale 5 return self 6 7 class Circle(Shape): 8 def set_radius(self, radius: float) -> Self: 9 self.radius = radius 10 return self 11
54. Summary 1. Benefits, starting with def , Built-in types 2. Standard collections (since 3.9) 3. Collections, Union, Optional, Callable, User-defined Generics 4. Updates Overview, __future__ for compatibility 54 / 57 ` ` ` `
55. 5. New features on Python 3.10 and 3.11 1. 3.10: New Type Union Operator, Parameter Specific Variables, TypeAlias, User-Defined Type Guards 2. 3.11: Self Type 55 / 57
56. Pages I used for reference (Thanks) There are links that I referenced. Thank you! https://docs.python.org/3/library/typing.html https://docs.python.org/3.10/whatsnew/3.10.html https://docs.python.org/3.11/whatsnew/3.11.html https://future-architect.github.io/articles/20201223 (ja) https://qiita.com/tk0miya/items/931da13af292060356b9 (ja) https://qiita.com/tk0miya/items/1b093c2eee7668fffb62 (ja) https://qiita.com/tk0miya/items/a27191f5d627601930ed (ja) 56 / 57
57. Thank you See you again next time in Japan! Powered by Slidev, using theme-briks