What's new in Python 3.7?

Python 3.7 is out since June 2017. In this talk you will see all the new features of Python 3.7 with some examples.

Transcript

1. What's new in Python 3.7? by Stéphane Wirtel 1 / 97
2. I am Stéphane 2 / 97
3. What's new in Python 3.7? 3 / 97
4. A new Syntax! public class Python37 { public static void main(String[] args) { System.out.println("Hello World!"); } } 4 / 97
5. 5 / 97
6. PEP 537 PEP 553, Built-in breakpoint() PEP 557, Data Classes PEP 562, Module __getattr__ and __dir__ PEP 563, Postponed Evaluation of Annotations PEP 564, Time functions with nanosecond resolution PEP 565, Show DeprecationWarning in __main__ PEP 567, Context Variables PEP 540, UTF-8 mode https://www.python.org/dev/peps/pep-0537/ 6 / 97
7. PEP 553 Built-in breakpoint() https://www.python.org/dev/peps/pep-0553/ 7 / 97
8. Built-in breakpoint() A wonderful function def divide(e, f): return f / e a, b = 0, 1 print(divide(a, b)) https://www.python.org/dev/peps/pep-0553/ 8 / 97
9. Built-in breakpoint() A wonderful function def divide(e, f): return f / e a, b = 0, 1 print(divide(a, b)) and of course, you have a crash... Traceback (most recent call last): File "bugs.py", line 5, in <module> print(divide(a, b)) File "bugs.py", line 2, in divide return f / e ZeroDivisionError: division by zero https://www.python.org/dev/peps/pep-0553/ 9 / 97
10. 10 / 97
11. Built-in breakpoint() So, how can I debug it? maybe with debugger (hello JS)? def divide(e, f): debugger; return f / e https://www.python.org/dev/peps/pep-0553/ 11 / 97
12. Built-in breakpoint() So, how can I debug it? maybe with debugger (hello JS)? def divide(e, f): debugger; return f / e No, it is not debugger, then what is the name of my debugger and how to add a breakpoint? pdb ipdb pudb pdbpp ... https://www.python.org/dev/peps/pep-0553/ 12 / 97
13. Built-in breakpoint() for example, with pdb def divide(e, f): import pdb; pdb.set_trace() return f / e a, b = 0, 1 print(divide(a, b)) https://www.python.org/dev/peps/pep-0553/ 13 / 97
14. Built-in breakpoint() for example, with pdb def divide(e, f): import pdb; pdb.set_trace() return f / e a, b = 0, 1 print(divide(a, b)) or with pudb def divide(e, f): import pudb; pudb.set_trace() return f / e a, b = 0, 1 print(divide(a, b)) https://www.python.org/dev/peps/pep-0553/ 14 / 97
15. Built-in breakpoint() or just with breakpoint() def divide(e, f): breakpoint() return f / e a, b = 0, 1 print(divide(a, b)) https://www.python.org/dev/peps/pep-0553/ 15 / 97
16. Built-in breakpoint() or just with breakpoint() def divide(e, f): breakpoint() return f / e a, b = 0, 1 print(divide(a, b)) $ python bugs.py > /tmp/bugs.py(3)divide() -> return f / e (Pdb) By default, breakpoint will execute pdb.set_trace() https://www.python.org/dev/peps/pep-0553/ 16 / 97
17. Built-in breakpoint() can be configured $ PYTHONBREAKPOINT=0 python bugs.py $ PYTHONBREAKPOINT='' python bugs.py $ PYTHONBREAKPOINT=pudb.set_trace python bugs.py $ PYTHONBREAKPOINT=IPython.embed python bugs.py ... $ pip install web-pdb $ PYTHONBREAKPOINT=’web_pdb.set_trace’ python bugs.py https://www.python.org/dev/peps/pep-0553/ 17 / 97
18. Built-in breakpoint() For example with web-pdb https://www.python.org/dev/peps/pep-0553/ 18 / 97
19. 19 / 97
20. PEP 557 Data Classes https://www.python.org/dev/peps/pep-0557/ 20 / 97
21. Data Classes - Introduction Numerous attempts to define classes which exist primarily to store values Examples collections.namedtuple The attrs project (Hi Hynek!) with python classes ... https://www.python.org/dev/peps/pep-0557/ 21 / 97
22. Data Classes - Bad examples (tuple, dict) tuple >>> person = ('Foo', 'Bar') >>> person[0] # is it the firstname or the lastname? dict >>> person = {'firstname': 'Foo', 'lastname': 'Bar'} >>> person['firstname'] # would prefer the . notation https://www.python.org/dev/peps/pep-0557/ 22 / 97
23. Data Classes - Bad examples (namedtuple) from collections import namedtuple Person = namedtuple('Person', ['firstname', 'lastname']) person = Person('Foo', 'Bar') person.firstname person == Person('Foo', 'Bar') person == ('Foo', 'Bar') # but person.firstname = 'Foooooooooo' # will crash https://www.python.org/dev/peps/pep-0557/ 23 / 97
24. Data Classes - Bad examples (class) class Person: def __init__(self, firstname, lastname, age): self.firstname = firstname self.lastname = lastname def __repr__(self): return f'Person(firstname={self.firstname}, lastname={self.lastname})' def __eq__(self, o): return (self.firstname == o.firstname and self.lastname == o.lastname) def __lt__(self, o): return (self.firstname < o.firstname and self.lastname < o.lastname) person = Person('Foo', 'Bar') person.firstname >>> person == Person('Foo', 'Bar') True >>> person < Person('Foo', 'Bar') False https://www.python.org/dev/peps/pep-0557/ 24 / 97
25. Data Classes - Good example from dataclasses import dataclass @dataclass class Person: firstname: str lastname: str person = Person('Foo', 'Bar') person.firstname >>> person == Person('Foo', 'Bar') True https://www.python.org/dev/peps/pep-0557/ 25 / 97
26. Data Classes - make_dataclass from dataclasses import make_dataclass Person = make_dataclass('Person', ['firstname', 'lastname']) person = Person('Foo', 'Bar') https://www.python.org/dev/peps/pep-0557/ 26 / 97
27. Data Classes - default values from dataclasses import dataclass @dataclass class Position: name: str latitude: float = 0.0 longitude: float = 0.0 >>> europython = Position('Charleroi') Position(name='Charleroi', latitude=0.0, longitude=0.0) https://www.python.org/dev/peps/pep-0557/ 27 / 97
28. Data Classes - default values from typing import List from dataclasses import dataclass, field @dataclass class Attendee: firstname: str lastname: str def default_attendees() -> List[Attendees]: return [Attendee('Foo', 'Bar')] @dataclass class Conference: name: str locations: str attendees: List[Attendee] = field(default_factory=default_attendees) conference = Conference('PySS18', 'San Sebastiàn') assert isinstance(conference.attendees, list) assert len(conference.attendees) == 1 https://www.python.org/dev/peps/pep-0557/ 28 / 97
29. Data Classes - methods/properties from dataclasses import dataclass @dataclass class Person: firstname: str lastname: str age: int @property def fullname(self): return f'{self.firstname} {self.lastname}' https://www.python.org/dev/peps/pep-0557/ 29 / 97
30. Data Classes - con guration init=True => __init__ repr=True=> __repr__ eq=True => __eq__ order=False => __lt__, __le__, __gt__, __ge__ frozen=False => __setattr__, __getattr__ ... https://www.python.org/dev/peps/pep-0557/ 30 / 97
31. Data Classes - con guration - frozen from dataclasses import dataclass @dataclass(frozen=True) class Person: firstname: str lastname: str age: int >>> person = Person('Foo', 'Bar', 38) >>> person.age = 27 # dataclasses.FrozenInstanceError: cannot assign to field 'age https://www.python.org/dev/peps/pep-0557/ 31 / 97
32. Data Classes - con guration - order from dataclasses import dataclass @dataclass(order=True) class Person: firstname: str lastname: str age: int >>> person = Person('Foo', 'Bar', 38) >>> person < Person('Foo', 'Bar', 37) False https://www.python.org/dev/peps/pep-0557/ 32 / 97
33. Data Classes - inheritance @dataclass class Person: firstname: str lastname: str @dataclass class User(Person): username: str password: str person = Person('Foo', 'Bar') user = User('Foo', 'Bar', 'Us3rn4m3', 's3cr3t') https://www.python.org/dev/peps/pep-0557/ 33 / 97
34. Data Classes - mypy from dataclasses import dataclass @dataclass class Person: firstname: str lastname: str person = Person(1, 2) $ mypy demo_dataclass.py ...:8: error: Argument 1 to "Person" has incompatible type "int"; expected "str" ...:8: error: Argument 2 to "Person" has incompatible type "int"; expected "str" https://www.python.org/dev/peps/pep-0557/ 34 / 97
35. Data Classes - asdict, astuple You can also convert to a dict or a tuple >>> from dataclasses import asdict, astuple >>> person = Person('Foo', 'Bar') >>> asdict(person) {'firstname': 'Foo', 'lastname': 'Bar'} >>> astuple(person) ('Foo', 'Bar') https://www.python.org/dev/peps/pep-0557/ 35 / 97
36. Data Classes - Conclusion replace tuples, namedtuples, simple class use type annotation -> hello mypy accept default values can have logic via methods/properties inheritance helpers https://www.python.org/dev/peps/pep-0557/ 36 / 97
37. 37 / 97
38. PEP 562 Module __getattr__ and __dir__ https://www.python.org/dev/peps/pep-0562/ 38 / 97
39. Module __getattr__ and __dir__ Customize the access to an attribute of a module, example, a deprecated function # main.py from lib import old_function # Works, but emits the warning $ python3 -Wd main.py /tmp/demo/lib.py:11: DeprecationWarning: old_function is deprecated https://www.python.org/dev/peps/pep-0562/ 39 / 97
40. Module __getattr__ and __dir__ # lib.py from warnings import warn def old_function(*args, **kwargs): warn('old_function is deprecated', DeprecationWarning) ... def old_other_function(*args, **kwargs): warn('old_other_function is deprecated', DeprecationWarning) ... https://www.python.org/dev/peps/pep-0562/ 40 / 97
41. Module __getattr__ and __dir__ # lib.py from warnings import warn deprecated_names = ['old_function', 'old_other_function'] def _deprecated_old_function(*args, **kwargs): ... def _deprecated_old_other_function(*args, **kwargs): ... def __getattr__(name): if name in deprecated_names: warn(f'{name} is deprecated', DeprecationWarning) return globlals()[f'_deprectated_{name}'] raise AttributeError(f'module {__name__} has no attribute {name}') https://www.python.org/dev/peps/pep-0562/ 41 / 97
42. Module __getattr__ and __dir__ Customize the listing of the items from a module # lib.py deprecated_names = ['old_function'] __all__ = ['new_function_one', 'new_function_two'] def new_function_one(arg, other): ... def new_function_two(arg, other): ... def __dir__() -> List[str]: return sorted(__all__ + deprecated_names) https://www.python.org/dev/peps/pep-0562/ 42 / 97
43. Module __getattr__ and __dir__ Customize the listing of the items from a module # lib.py deprecated_names = ['old_function'] __all__ = ['new_function_one', 'new_function_two'] def new_function_one(arg, other): ... def new_function_two(arg, other): ... def __dir__() -> List[str]: return sorted(__all__ + deprecated_names) # main.py import lib dir(lib) # prints ['new_function_one', 'new_function_two', 'old_function', ...] https://www.python.org/dev/peps/pep-0562/ 43 / 97
44. PEP 563 Postponed Evaluation of Annotations https://www.python.org/dev/peps/pep-0563/ 44 / 97
45. PEP 563: Postponed Evaluation of Annotations Without the annotations class Node: def __init__(self, left: Node, right: Node) -> None: self.left = left self.right = right https://www.python.org/dev/peps/pep-0563/ 45 / 97
46. PEP 563: Postponed Evaluation of Annotations Without the annotations class Node: def __init__(self, left: Node, right: Node) -> None: self.left = left self.right = right Traceback (most recent call last): File "test_node.py", line 1, in <module> class Node: File "test_node.py", line 2, in Node def __init__(self, left: Node, right: Node) -> None: NameError: name 'Node' is not defined https://www.python.org/dev/peps/pep-0563/ 46 / 97
47. PEP 563: Postponed Evaluation of Annotations Problem with the forward reference, for that, we have to define the class like that class Node: def __init__(self, left: 'Node', right: 'Node') -> None: self.left = left self.right = right https://www.python.org/dev/peps/pep-0563/ 47 / 97
48. PEP 563: Postponed Evaluation of Annotations Now, with the annotations from __future__ import annotations class Node: def __init__(self, left: Node, right: Node) -> None: self.left = left self.right = right and we do not need a forward reference. https://www.python.org/dev/peps/pep-0563/ 48 / 97
49. PEP 563: Postponed Evaluation of Annotations With the annotations and the dataclasses from __future__ import annotations from dataclasses import dataclass from typing import Optional @dataclass class Node: left: Optional[Node] = None right: Optiona[Node] = None root = Tree() print(root) https://www.python.org/dev/peps/pep-0563/ 49 / 97
50. PEP 564 Time functions with nanosecond resolution https://www.python.org/dev/peps/pep-0564 50 / 97
51. Time functions with nanosecond resolution Float type limited to 104 days The Python time.time() function returns the current time as a floating-point numberself. Limited to 64 bits and in the IEEE 754 format. With this limitation, the float type starts to lose nanoseconds after 104 days. -> Precision loss Example On Python microbenchmarks, it's common to see function calls taking less than 100ns. A difference of a few nanoseconds might become significant. https://www.python.org/dev/peps/pep-0564 51 / 97
52. Time functions with nanosecond resolution time.clock_gettime_ns() time.clock_settime_ns() time.monotonic_ns() time.perf_counter_ns() time.process_time_ns() time.time_ns() https://www.python.org/dev/peps/pep-0564 52 / 97
53. Time functions with nanosecond resolution time.clock_gettime_ns() time.clock_settime_ns() time.monotonic_ns() time.perf_counter_ns() time.process_time_ns() time.time_ns() >>> time.monotonic() 154630.068913333 https://www.python.org/dev/peps/pep-0564 53 / 97
54. Time functions with nanosecond resolution time.clock_gettime_ns() time.clock_settime_ns() time.monotonic_ns() time.perf_counter_ns() time.process_time_ns() time.time_ns() >>> time.monotonic() 154630.068913333 >>> time.monotonic_ns() 154631543907219 https://www.python.org/dev/peps/pep-0564 54 / 97
55. PEP 565 Show DeprecationWarning in main https://www.python.org/dev/peps/pep-0565/ 55 / 97
56. Show DeprecationWarning in main Since Python 3.2, the DeprecationWarning was hidden by default but: Side Effect only visible then running tests https://www.python.org/dev/peps/pep-0565/ 56 / 97
57. Show DeprecationWarning in main Since Python 3.2, the DeprecationWarning was hidden by default but: Side Effect only visible then running tests with python 3.7 DeprecationWarning: displayed by default only in __main__ and when running tests. https://www.python.org/dev/peps/pep-0565/ 57 / 97
58. Show DeprecationWarning in main with python 3.6 $ $PWD/python --version Python 3.6.7+ $ make PYTHON=$PWD/python -C Doc/ venv $PWD/python -m venv ./venv ./venv/bin/python3 -m pip install -U Sphinx blurb Collecting Sphinx https://www.python.org/dev/peps/pep-0565/ 58 / 97
59. Show DeprecationWarning in main with python 3.6 $ $PWD/python --version Python 3.6.7+ $ make PYTHON=$PWD/python -C Doc/ venv $PWD/python -m venv ./venv ./venv/bin/python3 -m pip install -U Sphinx blurb Collecting Sphinx with python 3.7 $ $PWD/python --version Python 3.7.1+ $ make PYTHON=$PWD/python -C Doc/ venv $PWD/python -m venv ./venv ./venv/bin/python3 -m pip install -U Sphinx blurb pip/_vendor/urllib3/util/selectors.py:14: DeprecationWarning: Using or ... pip/_vendor/urllib3/_collections.py:2: DeprecationWarning: Using or ... ... Collecting Sphinx https://www.python.org/dev/peps/pep-0565/ 59 / 97
60. 60 / 97
61. PEP 567 Context Variables https://www.python.org/dev/peps/pep-0567/ 61 / 97
62. PEP 567: Context Variables For a synchronous environment, TLS is perfect but not for an asynchronous environment, which in this case, the asynchronous tasks execute concurrently in the same OS thread. This concept is similar to the TLS, but allows correctly keeping track of values per asynchronous task. one new module: contextvars two objects: Context and ContextVar https://www.python.org/dev/peps/pep-0567/ 62 / 97
63. PEP 567: Context Variables import contextvars current_user = contextsvars.ContextVar('current_user', default=None) async def inner(): log.debug('Current User: %s', current_user.get()) @routes.get('/{name}') async def handler(request): current_user.set(request.match_info['name']) await inner() https://www.python.org/dev/peps/pep-0567/ 63 / 97
64. PEP 540 UTF-8 Mode https://www.python.org/dev/peps/pep-0540/ 64 / 97
65. UTF-8 Mode > python -X utf8 your_script.py > PYTHONUTF8=1 python your_script.py disabled by default but automatically enabled if LC_ALL=POSIX or C if enabled: Python will use the utf-8 encoding, regardless the locale of the current platform. sys.getfilesystemencoding() returns UTF-8 locale.getpreferredencoding() return UTF-8 sys.stdin & sys.stdout error handlers to surrogateescape (avoid a UnicodeDecodeError) https://www.python.org/dev/peps/pep-0540/ 65 / 97
66. PEP 545 Python documentation translations https://www.python.org/dev/peps/pep-0545/ 66 / 97
67. Python documentation translations Of course, we have the official documentation in English But you can find it also in: French Japanese Korean Maybe in the future, YOU could add German Polish Spanish Ukrainian ... We are waiting for your contributions ;-) https://www.python.org/dev/peps/pep-0545/ 67 / 97
68. Not in PEPs 68 / 97
69. importlib.resources 69 / 97
70. importlib.resources Load binary artifact from a package. = to pkg_resources in features, but > in performance we can read from File System, a zip file or anywhere. https://docs.python.org/fr/3.7/library/importlib.html https://www.youtube.com/watch?v=ZsGFU2qh73E 70 / 97
71. importlib.resources Load binary artifact from a package. = to pkg_resources in features, but > in performance we can read from File System, a zip file or anywhere. My package email/ ├── __init__.py └── tests ├── data │ ├── __init__.py │ └── message.eml └── __init__.py https://docs.python.org/fr/3.7/library/importlib.html https://www.youtube.com/watch?v=ZsGFU2qh73E 71 / 97
72. importlib.resources read email/tests/data/message.eml without importlib.resources dirname = pathlib.Path(__file__).parent data_path = dirname / 'tests' / 'data' / 'message.eml' eml = data_path.read_bytes() 72 / 97
73. importlib.resources read email/tests/data/message.eml without importlib.resources dirname = pathlib.Path(__file__).parent data_path = dirname / 'tests' / 'data' / 'message.eml' eml = data_path.read_bytes() with importlib.resources eml = resources.read_binary('email.tests.data', 'message.eml') 73 / 97
74. importlib.resources read email/tests/data/message.eml without importlib.resources dirname = pathlib.Path(__file__).parent data_path = dirname / 'tests' / 'data' / 'message.eml' eml = data_path.read_bytes() with importlib.resources eml = resources.read_binary('email.tests.data', 'message.eml') or with a context manager with resources.path('email.tests.data', 'message.eml') as eml: do_something(eml) 74 / 97
75. importlib.resources read email/tests/data/message.eml without importlib.resources dirname = pathlib.Path(__file__).parent data_path = dirname / 'tests' / 'data' / 'message.eml' eml = data_path.read_bytes() with importlib.resources eml = resources.read_binary('email.tests.data', 'message.eml') or with a context manager with resources.path('email.tests.data', 'message.eml') as eml: do_something(eml) or import email.tests.data with resources.parth(email.tests.data, 'message.eml') as eml: do_something(eml) 75 / 97
76. "Dict keeps insertion order" is the ruling! Introduced in Python 3.6. Thanks to INADA Naoki https://mail.python.org/pipermail/python-dev/2017- December/151283.html 76 / 97
77. "Dict keeps insertion order" is the ruling! >>> d = {} >>> d['name'] = 'pycon' >>> d['location'] = 'san sebastian, spain' >>> d['year'] = 2018 >>> d {'name': 'pycon', 'location': 'san sebastian, spain', 'year': 2018} >>> 77 / 97
78. from ... import ... 78 / 97
79. from ... import ... $ python3 -c "from itertools import demo" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: cannot import name 'demo' 79 / 97
80. from ... import ... $ python3 -c "from itertools import demo" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: cannot import name 'demo' $ /opt/cpython-3.7/bin/python3 -c "from itertools import demo" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: cannot import name 'demo' from 'itertools' (unknown location) 80 / 97
81. async & await are keywords! 81 / 97
82. async & await are keywords! For async async = True $ python /tmp/test_async.py File "/tmp/test_async.py", line 1 async = True ^ SyntaxError: invalid syntax 82 / 97
83. async & await are keywords! For async async = True $ python /tmp/test_async.py File "/tmp/test_async.py", line 1 async = True ^ SyntaxError: invalid syntax and await def await(): pass $ python /tmp/test_await.py File "test_await.py", line 1 def await(): ^ SyntaxError: invalid syntax 83 / 97
84. AsyncIO Improvements 84 / 97
85. asyncio has a lot of improvements Welcome to the asyncio.run function. Before, when you wanted to execute a coroutine for asyncio, you needed to create a loop, etc... async def amain(): async with aiohttp.ClientSession() as session: async with session.get('https://www.python.org') as resp: content = await resp.text() print(content[:30]) 85 / 97
86. asyncio has a lot of improvements Welcome to the asyncio.run function. Before, when you wanted to execute a coroutine for asyncio, you needed to create a loop, etc... async def amain(): async with aiohttp.ClientSession() as session: async with session.get('https://www.python.org') as resp: content = await resp.text() print(content[:30]) # before loop = asyncio.get_event_loop() try: loop.run_until_complete(amain()) finally: loop.close() 86 / 97
87. asyncio has a lot of improvements Welcome to the asyncio.run function. Before, when you wanted to execute a coroutine for asyncio, you needed to create a loop, etc... async def amain(): async with aiohttp.ClientSession() as session: async with session.get('https://www.python.org') as resp: content = await resp.text() print(content[:30]) # before loop = asyncio.get_event_loop() try: loop.run_until_complete(amain()) finally: loop.close() # now asyncio.run(amain()) 87 / 97
88. asyncio has a lot of improvements New functions asyncio supports contextvars asyncio.create_task() -> asyncio.get_event_loop().create_task() loop.start_tls() asyncio.current_task() asyncio.all_tasks() loop.sock_sendfile(): use the os.sendfile (when possible + performance) ... Performance improvements asyncio.get_event_loop (+- 15 times faster). asyncio.Future callback management has been optimized asyncio.gather() - 15% faster asyncio.sleep() - 2x faster ... 88 / 97
89. Gifts for the developers 89 / 97
90. Gifts for the developers > python -X importtime -c 'import asyncio' import time: self [us] | cumulative | imported package import time: 381 | 381 | zipimport import time: 2044 | 2044 | _frozen_importlib_external import time: 197 | 197 | _codecs import time: 1515 | 1712 | codecs ... 90 / 97
91. Gifts for the developers > python -X dev enable the "development mode" add additional runtime checks install debug hooks for the memory allocators enable the faulthandler module for a beautiful Python dump on a crash ;-) enable the asyncio debug mode 91 / 97
92. Performances 92 / 97
93. and many improvements, bug xes... 93 / 97
94. and ... 94 / 97
95. Python 3.7 is a great vintage! 95 / 97
96. What's new in Python 3.7? stephane.wirtel@mgx.io @matrixixe 96 / 97
97. stephane.wirtel@mgx.io @matrixixe 97 / 97