Demystify Python Types for PEP 729

Uncover the why behind PEP 729. This talk demystifies Python's type system, bridging theory and practice, and explores the challenges across type checkers.

Transcript

1. Demystify Python Types for PEP 729 Kir Chou Ver. Github 1
2. 2 PEP 729 – Typing governance process PEP = Python Enhancement Proposals
3. PEP 729 – Typing governance process (Discussion) "This PEP proposes a new way to govern the Python type system: a council that is responsible for maintaining and developing the Python type system." "This PEP was endorsed by maintainers of all major type checkers, including Rebecca Chen (pytype), Eric Traut (Pyright), and privately by maintainers of mypy and Pyre." 3
4. Initial council members (typing-council) Eric Traut (Pyright; author of PEP 647, PEP 681, and PEP 695) Jelle Zijlstra (core developer; typeshed; pyanalyze; author of PEP 688 and PEP 702) Shantanu Jain (core developer; typeshed; mypy) 4 Guido van Rossum (core developer; author of PEP 484 and PEP 526) Rebecca Chen (*pytype) *As of today, pytype is maintained by others.
5. Motivation 1. PEPs are the only specification 2. It’s hard to evolve the specification 3. The type system is underspecified 4. The Steering Council is not well-placed to solve the above problems 5 "... a user of typing has to also think about an external tool, the type checker." "... individual type checkers to decide how to navigate underspecified areas." "… that is often too heavy a process for a small change."
6. 6 ⭐Specification for the Python type system⭐
7. 7 Type? New process? 🧅 🔪
8. 8 Type theory Gradual typing theory Type checkers New specifications Table of contents
9. What is "Type" in Python or a programming language? Type theory 9
10. >>> a = True >>> b = False >>> print(a, b) True False >>> a == b False >>> not a False 10 What does a Boolean type actually mean in a programming language?
11. If you've studied other theories, e.g., Set theory, Category theory, Logic and Proof Theory, and Homotopy Type Theory (HoTT), … HoTT bridges them. 11 Type theory is "a presentation of a specific type system" Ref: Was soll HoTT? by Jacob Neumann
12. Expression x ∈ A ∈ = is an element of 12
13. x ∈ A in Type theory can be read in the following ways (Per Martin-Löf's theory): 1. x is an element of the set A <- Programming language uses this typically! ◦ A Python programmer may already know: ▪ True is a boolean type, so True ∈ bool ▪ False is a boolean type, so False ∈ bool 2. x is a proof object for the proposition A 3. x is a program satisfying the specification A 4. x is a solution satisfying the problem A <- This is my favorite one! 13 Expression x ∈ A ∈ = is an element of
14. a ∈ A 14 Q1: How to define a Set A?
15. x ∈ A 15 Condition1 for Q1: What's in A?
16. x ∈ A 16 y ∈ A = z ∈ A ≠ Condition2 for Q1: In A, what conditions make elements equal?
17. 󰝢 (-3kg) ∈ Lose weight 17 What can lose weight? 🏃 (-5kg) ∈ Lose weight 🥗 (-5kg) ∈ Lose weight 1. 󰝢🏃🥗 are solutions to the problem. 2. 🏃 and 🥗 solves the same problem (lose 5kg). 󰝢 solves a different problem (lose 3kg). = ≠ 4. x is an solution in the problem A.
18. z = False ∈ Bool x = True ∈ Bool 18 y = True ∈ Bool = ≠ 2. True ≠ False True = True 1. {True, False} ∈ Bool What is Bool? 1. x is an element in the set A.
19. a ∈ A 19 Q2: How to tell Set A = Set B? b ∈ B
20. x ∈ A 20 Condition1 for Q2: If x is in A, x is in B. x ∈ B
21. x ∈ A 21 x ∈ B x = x' ∈ A x = x' ∈ B Condition1 for Q2: If x is in A, x is in B. Condition2 for Q2: if x=x' in A, x=x' in B.
22. 22 y ∈ B y ∈ A y = y' ∈ B y = y' ∈ A Condition3 for Q2: If y in B, y in A. Condition4 for Q2: if y=y' in B, y=y' in A.
23. 23 4. x is an solution in the problem A. 󰝢≈🏃≠🥗 ∈ Improve fitness 󰝢≠🏃=🥗 ∈ Lost weight ≠ Lost weight = Improve fitness? When you can find… 1. Two solutions to solve both problems. 2. Another equivalent solution for solutions in 1 to solve the same problem. But you can't (as each solution's effect is different).
24. 24 String Set = Bool Set? When you can find… 1. A variable in both sets. 2. Two equal variables in each set. But you can't. a = True ∈ Bool b = "s" ∈ String ≠ 1. x is an element in the set A.
25. Q. How does a mathematician define a Set? 25
26. Q. How does a mathematician define a Set? 1. Formation rule - Problem definition (Set). ◦ A is a set. 2. Introduction rule - Possible solutions (Element). ◦ a is in the set A. 3. Elimination rule - If we pick a solution, how it solves the problem. ◦ What is A, and under what condition a, b, c, … is in A? 4. Equality rule - If we pick a specific solution, it solves the problem. ◦ If we define a and A, then a is in A. 26 A. General rules! (Per Martin-Löf's theory)
27. General rules for Bool by a mathematician • Formation rule ◦ BOOL set • Introduction rule ◦ true∈BOOL, fasle∈BOOL • Elimination rule ◦ Premise: b∈BOOL, C(v) set [v∈BOOL], c∈C(true), d∈C(false) ◦ Conclusion: (if b then c else d)∈C(b) • Equality rule ◦ Premise: C(v) set [v∈BOOL], c∈C(true), d∈C(false) ◦ Conclusion: (if true then c)∈C(true) / (if false then d)∈C(false) 27 True Set, False Set Selector to build conditional set
28. 28 Python Math
29. 29 Types in programming languages are behind math's theory! Python Math
30. Everything is PyObject* in Python? 30 "s" = True = NewType() ∈ PyObject* ? ∀ ∈ PyObject* ?
31. Everything is PyObject* in Python with type information 1. PyObject* header defines the type information (ob_type), the interpreter uses it to determine the valid operations in runtime. 2. type(), isinstance(), issubclass() can be used to check the type in runtime. >>> assert isinstance(∀, object) >>> assert issubclass(bool, object) # bool ∈ PyObject* >>> assert not issubclass(object, bool) >>> assert isinstance(True, bool) # True ∈ bool >>> assert isinstance(False, bool) # False ∈ bool 31
32. Bool by a Pythonista (PEP 285 - Mar 2002 by Guido) • Context ◦ Python didn't support a boolean type, while most languages did. ◦ People wrote code like this: True = 1; False = 0; • CPython: Include/boolobject.h / Objects/boolobject.c 32
33. PyTypeObject PyBool_Type 33 Base type: Bool ∈ Int Built-in functions (not, and, or, xor, doc) Built-in functions (__init__, __del__) Bool
34. 34 What is "Annotation" in Python? Gradual Typing Theory
35. def rect_area(length, width): return length * width def rect_area(length: float, width): return length * width def rect_area(length: float, width: float) -> float: return length * width 35 Annotation! Fully-typed Partially-typed Untyped Gradual typing
36. def bill_sum(bill1, bill2): return bill1 + bill2 >>> bill_sum(960, 110) 1070 >>> bill_sum(960.0, 110) 1070.0 >>> bill_sum("960", "110") 960110 >>> bill_sum([960], [110]) [960, 110] 36 Unexpected!! Which one? both? Why gradual typing?
37. A type may be partially known/unknown at compile time. Ref 1. Gradual Typing for Functional Languages by Jeremy Siek (2006) 2. What is gradual typing by Jeremy Siek (2014, March) 3. PEP 483 – The Theory of Type Hints (2014, Dec) 37 Gradual typing theory is "a pay-as-you-go type system"
38. A type may be partially known/unknown at compile time. Ref 1. PEP 483 – The Theory of Type Hints 2. Gradual Typing for Functional Languages by Jeremy Siek (2006) 3. What is gradual typing by Jeremy Siek (2014, March) 38 Gradual typing theory is "a pay-as-you-go type system"
39. 39 Type system Input Output Elements in Sets? x ∈ A ? y ∈ B ? Type consistency Yes, each element in the right set! x ∈ A ok y ∈ B ok Type consistency
40. Python's type consistency (Before PEP 483) 40 Python code CPython RuntimeError? Runtime type consistency Execution Result PS: You may not notice the code is considered "type consistent" by CPython. Type system Input Output
41. Python's type consistency (After PEP 483) 41 Untyped code RuntimeError? Partially-typed code Fully-typed code Type checker TypeError? Static type consistency Execution Result CPython Type system Input Output
42. 42 Are they type consistent? b: int = 1 a = 1
43. Gradual typing theory - rule 1 43 1. All program parameters are default assigned the type Any ("?"). ("Any" type is consistent to all types.) b: int = 1 a: any = 1 Whether the types of variables are the same.
44. 44 Type consistent! b: int = 1 a: any = 1 1. All program parameters are default assigned the type Any ("?"). ("Any" type is consistent to all types.) Gradual typing theory - rule 1
45. 45 Are they type consistent? def f(x): … def g(x: int) -> int: …
46. 46 Whether the types of arguments and return value of functions are the same. def f(x: any) -> any: … def g(x: int) -> int: … Gradual typing theory - rule 1 1. All program parameters are default assigned the type Any ("?"). ("Any" type is consistent to all types.)
47. 47 Type consistent! def f(x: any) -> any: … def g(x: int) -> int: … Gradual typing theory - rule 1 1. All program parameters are default assigned the type Any ("?"). ("Any" type is consistent to all types.)
48. 48 Is the type for variable c "int"? a = 1 b: int = 1 c = a + b
49. 49 c is any type because c is not annotated, so any + int = any. a: any = 1 b: int = 1 c = a + b Gradual typing theory - rule 1 1. All program parameters are default assigned the type Any ("?"). ("Any" type is consistent to all types.) 🚨 Note: the major type checker you used today may "infer" the type.
50. Type inference? (mypy vs pytype) s_typed: str = 'asdf' reveal_type(s_typed) # => builtins.str vs str s_untyped = 'asdf' reveal_type(s_untyped) # => builtins.str vs str reveal_type('asdf') # => Literal['asdf'] vs str def f(): return 'asdf' reveal_type(f()) # => Any vs str
51. 51 a: int = 1 def f(x): … f(a) 2. Reject programs that have inconsistencies in the known parts of types (at static time). a: int = 1 def f(x: str): … f(a) OK NG (Incorrect arg-type) Gradual typing theory - rule 2
52. 52 def f(x: str): return '(' + x.split() + ')' s: str = 'asdf' f(s) 3. Unknown types with errors can still be caught (at runtime). Ref: mypy:no-errors-reported-for-obviously-wrong-code Python: TypeError: can only concatenate str (not "list") to str Some type checkers (e.g., mypy): No issues Gradual typing theory - rule 3
53. 53 Takeaways: Python's type systems are based on Type theory and Gradual typing theory!
54. Type checkers 54
55. PEP 729 – Typing governance process (Discussion) "This PEP proposes a new way to govern the Python type system: a council that is responsible for maintaining and developing the Python type system." "This PEP was endorsed by maintainers of all major type checkers, including Rebecca Chen (pytype), Eric Traut (Pyright), and privately by maintainers of mypy and Pyre." 55
56. 56 Company Advantages mypy Dropbox, OG (Now) CPython 2012 Dec *Fast new features adoption. (As of today, pyright provides the best support - ref) pytype Google 2015 Mar Why not mypy? ref 1. Coverage: for untyped codes. 2. Type inference. PyCon 2019 lightning talk by rebecca pyre-check Meta 2017 Nov Why not mypy? ref 1, 2 1. Large-scale code (Instagram) gradual typing adoption. 2. Based on existing solutions in HACK (PHP in meta). pyright Microsoft 2019 Mar Why not mypy? ref 1. IDE integration, language protocol. 2. Speed: A “lazy” or “just-in-time” type evaluator. 3. Not written in python. Static type checkers comparison 👑
57. How do python's type systems resolve types? 57 Untyped code RuntimeError? Partially-typed code Fully-typed code Type checkers TypeError? Execution Result CPython Type system Input Output
58. Supported in type checkers + Supported in CPython from 3.11. *_type(...) may return different results in different type checkers and CPython. 58 reveal_type() / assert_type() in type systems # Before Python 3.11 if typing.TYPE_CHECK: reveal_type(...)
59. Python 3 Types in the Wild: A Tale of Two Type Systems TL;DR: A research by RPI and IBM to compare mypy and pytype in 2020. 59 Assignment variable x = e class variable x.attr = e list x[e1] = e2
60. 60 variable x = e Assignment with different type systems (untyped) class variable x.attr = e list x[e1] = e2 a = 1 reveal_type(a) a = 's' reveal_type(a) a = [1] reveal_type(a) a.append('s') reveal_type(a) class A: attr = 1 a = A() reveal_type(a.attr) a.attr = 's' reveal_type(a.attr) gh-113255
61. a = 1 reveal_type(a) a = 's' reveal_type(a) 61 a = [1] reveal_type(a) a.append('s') reveal_type(a) class A: attr = 1 a = A() reveal_type(a.attr) a.attr = 's' reveal_type(a.attr) Type inference via static analysis ❌ ❌ ❌
62. 62 variable x = e Assignment with different type systems (typed) class variable x.attr = e list x[e1] = e2 a: int|str = 1 reveal_type(a) a = 's' reveal_type(a) a: list[int|str] = [1] reveal_type(a) a.append('s') reveal_type(a) class A: attr: int|str = 1 a = A() reveal_type(a.attr) a.attr = 's' reveal_type(a.attr) gh-113255
63. 63 a: int|str = 1 reveal_type(a) a = 's' reveal_type(a) class A: attr: int|str = 1 a = A() reveal_type(a.attr) a.attr = 's' reveal_type(a.attr) Type inference via static analysis a: list[int|str] = [1] reveal_type(a) a.append('s') reveal_type(a)
64. 64 "Arguably, having two fundamentally different type systems violates the Zen of Python , which famously states that “There should be one– and preferably only one –obvious way to do it.”" – Python 3 Types in the Wild: A Tale of Two Type Systems "Can we rule the way to develop type checkers?" Takeaways: Problems even though the type systems are based on Type theory and Gradual typing
65. New specifications in type systems 65
66. 66 Type Type Specifications • Types define the kind of data a variable can hold. • They are checked at runtime. • e.g., ◦ PEP 285 – Adding a bool type • Type specifications provide optional hints about the expected types of variables, function parameters, and function return values. • They are not always checked at runtime but used by type checkers for static analysis. • e.g., ◦ PEP 484 - Type Hints ◦ PEP 544 - Protocols: Structural subtyping ◦ PEP 673 - Self Type New Specifications
67. For example: PEP 673 - Self Type Why PEP 673? 67 Before PEP673 After PEP673
68. PEP 673 - Self Type impl in CPython Lib/typing.py (Showing 5 changed files with 116 additions and 2 deletions.) # 1. You can use "from typing import Self". __all__ = [ …, 'Self', …] # 2. You can check the Self type and return it. def _type_check(arg, …): if arg in (..., Self, …): return arg … # 3. Explains how to use Self type. @_SpecialForm def Self(self, parameters): … raise TypeError(f"{self} is not subscriptable") 68 That's it!
69. Commit (Showing 28 changed files with 897 additions and 57 deletions.) ~8x efforts than CPython • mypy/semanal.py ◦ Semantic analyzer (bind names to definitions + do simple consistency checks). ◦ 📝 Support self type in semantic analysis (in mypy/typeanal.py). • mypy/checker.py ◦ Type checker (do various checks). ◦ 📝 Support expand_self_type (in mypy/expandtype.py). • mypy/nodes.py ◦ Nodes of the abstract syntax tree (AST). ◦ 📝 Support new self type node. 69 PEP 673 - Self Type impl in mypy
70. 70 PEP 673 - Self Type impl in pytype?
71. Timeline to support a new specification 1. 2021-Nov-10: PEP 673 Created for CPython. 2. 2021-Dec: Bugged for mypy. 3. 2022-Feb: Bugged and Implemented in CPython. 4. 2022-Nov: Implemented in mypy. 5. 2022-Sep: Bugged for pytype. 6. 2023-Nov: Implemented in pytype. 71 10M Fun fact: Self type annotation is implemented in 3 months, while it can be checked after… 10+ months for mypy, 21+ months for pytype. 3M 21M
72. Link to full Report
73. 73 "Arguably, having two fundamentally different type systems violates the Zen of Python , which famously states that “There should be one– and preferably only one –obvious way to do it.”" – Python 3 Types in the Wild: A Tale of Two Type Systems "Can we rule the way to develop type checkers?" "Can we shorten this noticeable time gap to support a new specification to different type checkers?" Takeaways: Problems even though the type systems are based on Type theory and Gradual typing
74. 74 PEP 729 - a solution? typing-council
75. ref
76. 76 🔪 Bonus?
77. 77 Bonus: A mini type checker?
78. 78 Untyped code Partially-typed code Fully-typed code TypeError? Type system Input Output
79. 79 I/O Arg-type input test cases Assignment input test cases Return-Value input test cases NG output def add_one(a: int): return a + 1 add_one("1") def add_one(a) -> int: return a + 1 b: str = add_one(1) def reflex(a: int) -> int: return a def reflex_twice() -> str: return reflex(1) OK output def add_one(a: int): return a + 1 add_one(1) def add_one(a) -> int: return a + 1 b: int = add_one(1) def reflex(a: int) -> int: return a def reflex_twice() -> int: return reflex(1)
80. 80 Untyped code Partially-typed code Fully-typed code Mini type checker TypeError? Type system Input Output
81. 81 Mini type checker LibCST to parse code to tree! • Parse 1st time for semantic analysis • Parse 2nd time for type check Also check untyped/partial-typed code Pytype/Pyre are based on LibCST.
82. ⭐Specification for the Python type system⭐ Recommended: Generics 82 More inputs? OK NG Generics T = TypeVar("whatever") def f(a: T) -> T: return a x: int = 1 y: int = f(x) T = TypeVar("whatever") def f(a: T) -> T: return a x: int = 2 y: str = f(x)
83. 83 Thank you! github.com/note35/TypeSystem-Learning More about me, and my talks: note35.github.io/about