What's new in Python 3.8? October 2019 Stéphane Wirtel Python: 3.8 1 / 68

I am Stéphane I am Stéphane 2 / 68 2 / 68

Hi, I am Stéphane Python lover since 2001 Freelancer CPython Core Dev PSF Fellow {Fellowship WG, Marketing WG} Board Member of EuroPython Society (EuroPython 2020 - 20th/26th July 2020) Former co-organiser of PythonFOSDEM (1st Feb 2020) 3 / 68

Schedule Schedule 4 / 68 4 / 68

Schedule in this talk PEP 570: Python Positional-Only Parameters PEP 572: Assignment Expressions PEP 578: Python Runtime Audit Hooks Others points f-string{=}, ... PEP 574: Pickle protocol 5 with out-of-band data PEP 587: Python Initialization Configuration PEP 590: Vectorcall: a fast calling protocol for CPython https://python.org/dev/peps/pep-0569 5 / 68

Release 3.8 Release 3.8 6 / 68 6 / 68

Schedule of 3.8.0 3.8 development begins: 2018-01-29 Alpha 3.8.0 alpha 1: 2019-02-03 3.8.0 alpha 2: 2019-02-25 3.8.0 alpha 3: 2019-03-25 3.8.0 alpha 4: 2019-05-06 Beta 3.8.0 beta 1: 2019-06-04 3.8.0 beta 2: 2019-07-04 3.8.0 beta 3: 2019-07-29 3.8.0 beta 4: 2019-08-30 7 / 68

Schedule of 3.8.0 Candidate 3.8.0 candidate 1: 2019-10-01 8 / 68

Schedule of 3.8.0 Candidate 3.8.0 candidate 1: 2019-10-01 Release 3.8.0 final: 2019-10-14 (maybe next Monday) ;-) Thank you to Łukasz Langa, our Release Manager for Python 3.8 and his first #Python Baby ;-) 9 / 68

PEPs {570, 572, 578} PEPs {570, 572, 578} 10 / 68 10 / 68

PEP 570: Python Positional-Only parameters Currently, we can pass the arguments by position or by name. BUT sometimes, we would like to restrict the call to a function by passing the parameters by position and not by name For that, we have introduced a new syntax for the definition of the functions/methods with the '/'. API Designer Point of View: We can rename the name of a parameter and keep the backward compatibility. def name(pos1, pos2, /, pos_or_kwd, *, kwd1, kwd2): pass pos1 and pos2 are positional-only arguments pos_or_kwd is a positional or keyword argument kwd1, kwd2 are only keyword arguments https://www.python.org/dev/peps/pep-0570/ 11 / 68

PEP 570: Python Positional-Only parameters Example def pow(x, y, /, mod=None): r = x ** y if mod is not None: r %= mod return r >>> pow(2, 10) 1024 >>> pow(2, 10, 17) 4 >>> pow(2, 10, mod=17) 4 >>> pow(x=2, y=10, mod=17) Traceback (most recent call last): File "", line 1, in TypeError: pow() got some positional-only arguments passed as keyword arguments: 'x, y' >>> https://www.python.org/dev/peps/pep-0570/ 12 / 68

PEP 572: Assignment Expressions (aka Walrus Operator) Before >>> params = {'foo': 'bar'} >>> x = params.get('foo') >>> if x is not None: ... print(x) 'foo' https://www.python.org/dev/peps/pep-0572/ 13 / 68

PEP 572: Assignment Expressions (aka Walrus Operator) Before >>> params = {'foo': 'bar'} >>> x = params.get('foo') >>> if x is not None: ... print(x) 'foo' After Say Hello to := The Walrus Operator >>> params = {'foo': 'bar'} >>> if x := params.get('foo'): ... print(x) https://www.python.org/dev/peps/pep-0572/ 14 / 68

PEP 572: Assignment Expressions (aka Walrus Operator) Before match = pattern.search(data) if match is not None: do_something(match) https://www.python.org/dev/peps/pep-0572/ 15 / 68

PEP 572: Assignment Expressions (aka Walrus Operator) Before match = pattern.search(data) if match is not None: do_something(match) After if (match := pattern.search(data)): do_something(match) https://www.python.org/dev/peps/pep-0572/ 16 / 68

PEP 572: Assignment Expressions (aka Walrus Operator) Before n = len(a) if n > 10: print(f"List is too long ({n} elements, expected <= 10)") https://www.python.org/dev/peps/pep-0572/ 17 / 68

PEP 572: Assignment Expressions (aka Walrus Operator) Before n = len(a) if n > 10: print(f"List is too long ({n} elements, expected <= 10)") After if (n := len(a)) > 10: print(f"List is too long ({n} elements, expected <= 10)") https://www.python.org/dev/peps/pep-0572/ 18 / 68

PEP 572: Assignment Expressions (aka Walrus Operator) Example 1: commit e1d455f3a3b82c2e08d5e133bcbab5a181b66cfb Author: Julien Palard Date: Wed Sep 11 15:01:18 2019 +0200 Doc: Use walrus operator in example. (GH-15934) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst index 48bc35ca76..db26d56af3 100644 --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -516,10 +516,7 @@ Here is an example session that uses the ``GET`` method:: >>> # The following example demonstrates reading data in chunks. >>> conn.request("GET", "/") >>> r1 = conn.getresponse() - >>> while True: - ... chunk = r1.read(200) # 200 bytes - ... if not chunk: - ... break + >>> while chunk := r1.read(200): ... print(repr(chunk)) b'\n

PEP 572: Assignment Expressions (aka Walrus Operator) Examples 2 & 3: >>> for angle in range(360): ... print(f'{angle=}\N{degree sign} {(theta:=radians(angle))=:.3f} {sin(theta)=:.3f}') >>> filtered_data = [y for x in data if (y := f(x)) is not None] Credit: Raymond Hettinger https://www.python.org/dev/peps/pep-0572/ 20 / 68

PEP 572: Assignment Expressions (aka Walrus Operator) Example 4 https://www.python.org/dev/peps/pep-0572/ 21 / 68

PEP 572: Assignment Expressions (aka Walrus Operator) Example 4 >>> [(t:=(t[1], sum(t)) if i else (0,1))[0] for i in range(20)] [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181] — Raymond Hettinger (@raymondh) October 13, 2019 https://www.python.org/dev/peps/pep-0572/ 22 / 68

PEP 578: Python Runtime Audit Hooks What's that? Allow to generete event for some (C-API and Python) functions Allow the monitoring of low-level access The events can be watched with hooks A Hook can't be removed or replaced https://www.python.org/dev/peps/pep-0578/ 23 / 68

PEP 578: Python Runtime Audit Hooks Declare a audit hook sys.addaudithook(hook: Callable(Undefined, Undefined)) import sys from functools import reduce from contextlib import supress def audit_hook(event, args): if event in ['urllib.Request']: print(f'Network {event=} {args=}') sys.addaudithook(audit_hook) def product(series): with supress(Exception): urllib.request.urlopen('http://example.com') return reduce(lambda acc, num: acc * num, series) result = product(range(10)) print(f'{result=}') Network event='urllib.Request' args=('http://example.com', None, {}, 'GET') result=0 24 / 68

PEP 578: Python Runtime Audit Hooks Emit an event sys.audit(str, *args) import sys from functools import reduce def make_request(url): sys.audit('make_request', url) def product(series): make_request('http://www.example.com') return reduce(lambda acc, num: acc * num, series) def audit_hook(event, args): if event in ('make_request',): print(f'Network {event=} {args=}') sys.addaudithook(audit_hook) result = product(range(1, 10)) print(f'{result=}') Network event='make_request' args=('http://www.example.com',) result=362880 25 / 68

PEP 578: Python Runtime Audit Hooks List of Events Hook Description sys.addaudithook Detect when new audit hooks are being added compile Detect dynamic code compilation exec Detect dynamic execution of code objects import Detect when modules are imported io.open Detect when a file is about to be opened object.setattr Detect monkey patching object.delattr Detect deletion of object attributes object.getattr Detect access to restricted attributes urllib.Request Detect URLS request etc... 26 / 68

PEP 578: Python Runtime Audit Hooks Basic Example import sys def audit_hook(event, args): printf(f'{event=} {args=}') sys.addaudithook(audit_hook) from collections import defaultdict event='import' args=('functools', None, ['/home/stephane/src/github.com/python/cpython', '/home/stephane/src/github. event='os.listdir' args=('/home/stephane/src/github.com/python/cpython',) event='open' args=('/home/stephane/src/github.com/python/cpython/Lib/__pycache__/functools.cpython-39.pyc', 'r', 524 event='exec' args=( at 0x7fad7c5801e0, file "/home/stephane/src/github.com/python/cpython/Lib/f event='import' args=('collections', None, ['/home/stephane/src/github.com/python/cpython', '/home/stephane/src/githu event='open' args=('/home/stephane/src/github.com/python/cpython/Lib/collections/__pycache__/__init__.cpython-39.pyc event='exec' args=( at 0x7fad7c5276c0, file "/home/stephane/src/github.com/python/cpython/Lib/c event='import' args=('operator', None, ['/home/stephane/src/github.com/python/cpython', '/home/stephane/src/github.c event='open' args=('/home/stephane/src/github.com/python/cpython/Lib/__pycache__/operator.cpython-39.pyc', 'r', 5242 27 / 68

PEP 578: Python Runtime Audit Hooks About Performance? Depends of the cases but the Python Performance Benchmark Suite shows no significant impact. Between 1.05% faster and 1.05% slower. https://github.com/python/pyperformance 28 / 68

Not in the PEPs Not in the PEPs 29 / 68 29 / 68

Feature: Asyncio REPL Python 3.4 import asyncio @asyncio.coroutine def say(what, when): await asyncio.sleep(when) print(what) loop = asyncio.get_event_loop() loop.run_until_complete(say('hello world', 1)) loop.close() At the beginning of Asyncio (Python 3.4 in 2014) 30 / 68

Feature: Asyncio REPL Python 3.5 import asyncio async def say(what, when): await asyncio.sleep(when) print(what) loop = asyncio.get_event_loop() loop.run_until_complete(say('hello world', 1)) loop.close() 31 / 68

Feature: Asyncio REPL with Python 3.7 >>> import asyncio >>> async def say(what, when): ... await asyncio.sleep(when) ... print(what) ... >>> asyncio.run(say('hello world', 1)) >>> 32 / 68

Feature: Asyncio REPL with Python 3.7 >>> import asyncio >>> async def say(what, when): ... await asyncio.sleep(when) ... print(what) ... >>> asyncio.run(say('hello world', 1)) >>> 3.8+, Welcome to the AsyncIO REPL python -m asyncio >>> async def say(what, when): ... await asyncio.sleep(when) ... print(what) ... >>> await say('hello world', 1)) >>> 33 / 68

Feature: Do you remember the f-strings in 3.6? They are really useful, example: Before f-strings first_name = 'Stephane' print("Hello %s" % (first_name,)) print("Hello {}".format(first_name)) print("Hello {name}".format(name=first_name)) 34 / 68

Feature: Do you remember the f-strings in 3.6? They are really useful, example: Before f-strings first_name = 'Stephane' print("Hello %s" % (first_name,)) print("Hello {}".format(first_name)) print("Hello {name}".format(name=first_name)) with f-strings first_name = 'Stephane' print(f"Hello {first_name}") 35 / 68

Feature: f-strings debug Before >>> x = 10 >>> f'x: {x}' 36 / 68

Feature: f-strings debug Before >>> x = 10 >>> f'x: {x}' Now >>> x = 10 >>> f'{x=}' 37 / 68

Feature: f-string debug Of course, you can use the string formatting. >>> user='stephane' >>> member_since = date(1980, 9, 15) >>> delta = date.today() - member_since >>> f'{user=!s} {delta.days=:,d}' 'user=stephane delta.days=14,269' >>> 38 / 68

Feature: json.tools --json-lines json.tool Add the option --json-lines to the module json.tool Allow to format the content of a multi-line json. echo -e '{"ingredients":["frog", "water", "chocolate", "glucose"]}\n{"ingredients":["chocolate","steel bolts"]}' \ | ./python -m json.tool --json-lines { "ingredients": [ "frog", "water", "chocolate", "glucose" ] } { "ingredients": [ "chocolate", "steel bolts" ] } 39 / 68

Feature: Support of pax pax (POSIX.1-2001) is the default format: shutil.make_archive tarfile 40 / 68

Feature: unittest adds AsyncMock Add AsyncMock for the support of asynchronous code >>> import asyncio >>> from unittest.mock import AsyncMock >>> mock_instance = AsyncMock() >>> mock_instance.__aiter__.return_value = [1, 2, 3] >>> async def amain(): ... return [i async for i in mock_instance] ... >>> asyncio.run(amain()) [1, 2, 3] 41 / 68

Feature: unittest adds AsyncMock >>> import asyncio >>> from unittest.mock import MagicMock >>> class AsyncContextManager: ... async def __aenter__(self): ... return self ... async def __aexit__(self, exc_type, exc, tb): ... pass ... >>> mock_instance = MagicMock(AsyncContextManager()) >>> async def amain(): ... async with mock_instance as result: ... pass ... >>> asyncio.run(amain()) >>> mock_instance.__aenter__.assert_awaited_once() >>> mock_instance.__aexit__.assert_awaited_once() 42 / 68

Feature: unittest adds AsyncMock import unittest class TestRequest(unittest.IsolatedAsyncioTestCase): async def asyncSetUp(self): self.connection = await AsyncConnection() async def test_get(self): response = await self.connection.get('https://example.com') self.assertEqual(response.status_code, 200) async def asyncTearDown(self): await self.connection.close() if __name__ == '__main__': unittest.main() 43 / 68

Feature: Exception Hook for threading import sys import threading def log_exception(*args): print(f'got exception {args}') sys.excepthook = log_exception def foo(): a = 1 / 0 threading.Thread(target=foo).start() 44 / 68

Feature: Exception Hook for threading import sys import threading def log_exception(*args): print(f'got exception {args}') sys.excepthook = log_exception def foo(): a = 1 / 0 threading.Thread(target=foo).start() Exception in thread Thread-1: Traceback (most recent call last): File "cpython/Lib/threading.py", line 944, in _bootstrap_inner self.run() File "cpython/Lib/threading.py", line 882, in run self._target(*self._args, **self._kwargs) File "demo-fail-excepthook.py", line 10, in foo a = 1 / 0 ZeroDivisionError: division by zero 45 / 68

Feature: Exception Hook for threading Add function threading.excepthook import sys import threading def log_exception(*args): print(f'got exception {args}') threading.excepthook = log_exception def foo(): a = 1 / 0 threading.Thread(target=foo).start() got exception args=(_thread.ExceptHookArgs( exc_type=, exc_value=ZeroDivisionError('division by zero'), exc_traceback=, thread=),) 46 / 68

Feature: sys.unraisablehook Handle an unraisable exception. Called when an exception has occurred but there is no way for Python to handle it. Example: a destructor raises an exception or during garbage collection (gc.collect()). import sys def unraisablehook(args): sys.stderr.write(f'{args=}') sys.unraisablehook = unraisablehook class Demo: def __del__(self): raise Exception(':/') demo = Demo() del demo 47 / 68

Feature: Add \N{EM DASH} in re module Now, it's possible to parse a string with a unicode. >>> import re >>> notice = 'I love 3.8' >>> PATTERN = re.compile(r'I love \N{SNAKE} ([0-9\.]*)') >>> if (match := PATTERN.search(notice)): ... print(match.group(1)) ... 3.8 SNAKE is a name of the Unicode Character 'SNAKE' (U+1F40D): Updated to Unicode 12.1.0 https://www.fileformat.info/info/unicode/char/1f40d/index.htm 48 / 68

Feature: Generalized iterable unpacking in yield and return Before >>> def parse(family): ... lastname, *members = family.split() ... return lastname.upper(), *members File "", line 3 return lastname.upper(), *members ^ SyntaxError: invalid syntax 49 / 68

Feature: Generalized iterable unpacking in yield and return Before >>> def parse(family): ... lastname, *members = family.split() ... return lastname.upper(), *members File "", line 3 return lastname.upper(), *members ^ SyntaxError: invalid syntax Now >>> def parse(family): ... lastname, *members = family.split() ... return lastname.upper(), *members ... >>> parse('simpsons homer marge bart lisa sally') ('SIMPSONS', 'homer', 'marge', 'bart', 'lisa', 'sally') 50 / 68

Feature: SyntaxWarning when missing comma Python < 3.8 >>> [(1, 2, 3)(4, 5)] Traceback (most recent call last): File "", line 1, in TypeError: 'tuple' object is not callable Python >= 3.8 >>> [(1, 2, 3)(4, 5)] :1: SyntaxWarning: 'tuple' object is not callable; perhaps you missed a comma? Traceback (most recent call last): File "", line 1, in TypeError: 'tuple' object is not callable 51 / 68

Feature: New Module Add importlib.metadata >>> from importlib.metdata import version, requires, files >>> version('requests') '2.22.0' >>> list(requires('requests')) ['chardet (<3.1.0,>=3.0.2)'] >>> list(files('requests'))[:2] [PackagePath('requests-2.22.0.dist-info/INSTALLER'), PackagePath('requests-2.22.0.dist-info/LICENSE')] You can also play with importlib.resources 52 / 68

Feature: Hi multiprocessing.shared_memory Processes are conventionally limited to only have access to their own process memory space shared memory permits the sharing of data between processes avoiding the need to instead send messages between processes. Sharing data directly via memory can provide significant performance benefits compared to sharing data via disk or socket or other communications requiring the serialization/deserialization and copying of data. 53 / 68

Feature: Hi multiprocessing.shared_memory >>> # process 1 >>> from multiprocessing import shared_memory >>> a = shared_memory.ShareableList(range(5)) >>> print(a.shm.name) psm_8c3af53a >>> print(a) ShareableList([0, 1, 2, 3, 4], name='psm_8c3af53a') 54 / 68

Feature: Hi multiprocessing.shared_memory >>> # process 1 >>> from multiprocessing import shared_memory >>> a = shared_memory.ShareableList(range(5)) >>> print(a.shm.name) psm_8c3af53a >>> print(a) ShareableList([0, 1, 2, 3, 4], name='psm_8c3af53a') >>> # process 2 >>> from multiprocessing import shared_memory >>> b = shared_memory.ShareableList(name='psm_8c3af53a') >>> print(b) ShareableList([0, 1, 2, 3, 4], name='psm_8c3af53a') >>> b[-1] = 10 >>> print(b) ShareableList([0, 1, 2, 3, 10], name='psm_8c3af53a') 55 / 68

Feature: Hi multiprocessing.shared_memory >>> # process 1 >>> from multiprocessing import shared_memory >>> a = shared_memory.ShareableList(range(5)) >>> print(a.shm.name) psm_8c3af53a >>> print(a) ShareableList([0, 1, 2, 3, 4], name='psm_8c3af53a') >>> # process 2 >>> from multiprocessing import shared_memory >>> b = shared_memory.ShareableList(name='psm_8c3af53a') >>> print(b) ShareableList([0, 1, 2, 3, 4], name='psm_8c3af53a') >>> b[-1] = 10 >>> print(b) ShareableList([0, 1, 2, 3, 10], name='psm_8c3af53a') >>> # process 1 >>> print(a) ShareableList([0, 1, 2, 3, 10], name='psm_8c3af53a') 56 / 68

Feature: Math Add math.dist for the Euclidean distance between two points. math.prod for the product of an iterable of numbers >>> math.prod([1, 2, 3, 4]) 24 math.isqrt for integer square roots >>> math.isqrt(4) 2 Changes math.hypot supports multiple dimensions math.factorial only for int. >>> math.factorial(4) 24 57 / 68

Feature: socket: Create a server Before import socket sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1) sock.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0) sock.bind(("", 8080)) sock.listen() 58 / 68

Feature: socket: Create a server Before import socket sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1) sock.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0) sock.bind(("", 8080)) sock.listen() New function: socket.create_server Now import socket addr = ("", 8080) if socket.has_dualstack_ipv6(): s = socket.create_server(addr, family=socket.AF_INET6, dualstack_ipv6=True) else: s = socket.create_server(addr) 59 / 68

Feature: functools.cached_property Before class DataSet: def __init__(self, sequence_of_numbers): self.numbers = sequence_of_numbers self._variance = None @property def variance(self): if self._variance is None: self._variance = statistics.variance(self.numbers) return self._variance 60 / 68

Feature: functools.cached_property Before class DataSet: def __init__(self, sequence_of_numbers): self.numbers = sequence_of_numbers self._variance = None @property def variance(self): if self._variance is None: self._variance = statistics.variance(self.numbers) return self._variance Now import functools class DataSet: def __init__(self, sequence_of_numbers): self.numbers = sequence_of_numbers @functools.cached_property def variance(self): return statistics.variance(self.numbers) 61 / 68

Deprecations & Removals Deprecations & Removals 62 / 68 62 / 68

Deprecations bdist_wininst -> bdist_wheel threading.Thread.isAlive() ... Removals Module macpath platform.popen() -> os.popen() time.clock() -> time.process_time() pyvenv script -> python -m venv ... 63 / 68

Performance Performance 64 / 68 64 / 68

Optimizations subprocess will call os.posix_spawn on Linux and macOS if conditions are met: close_fds is False preexec_fn, pass_fds, cwd and start_new_sessions parameters are not set. the executable path contains a directory shutil.copyfile, shutil.copy, shutil.copy2, shutil.copytree and shutil.move use the "fast-copy" syscalls on Linux and macOS (implemented by the Kernel) outfd.write(infd.read()) By default, the pickle module uses Protocol 4, better performance and smaller size compared to Protocol 3 uuid.UUID uses __slots__ -> reduce memory footprint operator.itemgetter by 33%. etc... 65 / 68

Optimizations 66 / 68

Questions? Questions? 67 / 68 67 / 68

What's new in Python 3.8? Stéphane Wirtel [email protected] https://twitter.com/matrixise 68 / 68