Tweet
Tweet

Slide 1

Slide 1 text

Asyncio Evolved: Enhanced Exception Handling with TaskGroup in Python 3.11 Junya Fukuda PyCon Taiwan 2023

Slide 2

Slide 2 text

Who am I? 👤 •Junya Fukudaʢ@JunyaFffʣcalled “Jun” not “Junior” •Photos 📷 Share 🐦 👍 •Software Engineer at Groove X, inc. Here is today's code ෱ా൏໵

Slide 3

Slide 3 text

Who am I? 👤 •book •Python Practical Recipes Here is today's code •Junya Fukudaʢ@JunyaFffʣcalled “Jun” not “Junior” •Photos 📷 Share 🐦 👍 •Software Engineer at Groove X, inc. ෱ా൏໵

Slide 4

Slide 4 text

Who am I? 👤 •book •Python Practical Recipes •Expert Python Programming - Fourth Edition •Written by Tarek Ziadé, Michał Jaworski Here is today's code •Junya Fukudaʢ@JunyaFffʣcalled “Jun” not “Junior” •Photos 📷 Share 🐦 👍 •Software Engineer at Groove X, inc. ෱ా൏໵

Slide 5

Slide 5 text

Who am I? 👤 •book •Python Practical Recipes •Expert Python Programming - Fourth Edition •Written by Tarek Ziadé, Michał Jaworski Here is today's code •Junya Fukudaʢ@JunyaFffʣcalled “Jun” not “Junior” •Photos 📷 Share 🐦 👍 •Software Engineer at Groove X, inc. ෱ా൏໵

Slide 6

Slide 6 text

asyncio's new features and New "Hello-ish world"

Slide 7

Slide 7 text

Today's Goal

Slide 8

Slide 8 text

Today's Goal •Focus on exception handling and cancellation. •Introducing the new API asyncio.TaskGroup.

Slide 9

Slide 9 text

Today's Goal •Focus on exception handling and cancellation. •Introducing the new API asyncio.TaskGroup. •Writing asyncio code in a "reliable" and "safe" way •The new standard "Hello-ish world"

Slide 10

Slide 10 text

Why are we talking about thisʁ

Slide 11

Slide 11 text

Why are we talking about thisʁ There are two main reasons.

Slide 12

Slide 12 text

Motivation (why are we talking about this?) •There are two main reasons.

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

async def main(): print(f"{time.ctime()} Hello!") await asyncio.sleep(1.0) print(f"{time.ctime()} GoodBye!") loop = asyncio.get_event_loop() task = loop.create_task(main()) loop.run_until_complete(task) pending = asyncio.all_tasks(loop=loop) for task in pending: task.cancel() group = asyncio.gather(*pending, return_exceptions=True) loop.run_until_complete(group) loop.close()

Slide 17

Slide 17 text

Motivation (why are we talking about this?) •I would like to revamp this code with asyncio.TaskGroup

Slide 18

Slide 18 text

Agenda •Introduction •Capturing concurrent exceptions •Cancelling tasks on exception •The new standard "Hello-ish world"

Slide 19

Slide 19 text

Scope of Today's Talk •Talk. •Exception handling in asyncio (also new except* syntax) •Cancel Task •asyncio's New Hello World •Don't talk. •Speci fi c examples in the application •What's happening at ASGI Here is today's code

Slide 20

Slide 20 text

asyncio, are you using it? ✋

Slide 21

Slide 21 text

Overview of asyncio •asyncio: Python's async library. •Continues to evolve. •Handling multiple tasks concurrently. •Useful for I/O-bound tasks like database or HTTP requests.

Slide 22

Slide 22 text

🤔

Slide 23

Slide 23 text

Why Not Using Asyncio? Reasons to Consider •Synchronous processing, which we're used to, is suf fi cient. •Primarily CPU-bound tasks: •Asynchronous processing seems like it might behave unpredictably. •We might choose to write in another language Mainly focusing on machine learning and data analysis with minimal IO (such as Go, JavaScript (nodejs/deno..), Rust, C#, Swift, etc.)

Slide 24

Slide 24 text

Why Not Using Asyncio? Reasons to Consider •Synchronous processing, which we're used to, is suf fi cient. •Primarily CPU-bound tasks: •Asynchronous processing seems like it might behave unpredictably. •We might choose to write in another language Mainly focusing on machine learning and data analysis with minimal IO (such as Go, JavaScript (nodejs/deno..), Rust, C#, Swift, etc.)

Slide 25

Slide 25 text

Why Not Using Asyncio? Reasons to Consider •Synchronous processing, which we're used to, is suf fi cient. •Primarily CPU-bound tasks: •Asynchronous processing seems like it might behave unpredictably. •We might choose to write in another language Mainly focusing on machine learning and data analysis with minimal IO (such as Go, JavaScript (nodejs/deno..), Rust, C#, Swift, etc.)

Slide 26

Slide 26 text

Why Not Using Asyncio? Reasons to Consider •Synchronous processing, which we're used to, is suf fi cient. •Primarily CPU-bound tasks: •Asynchronous processing seems like it might behave unpredictably. •We might choose to write in another language Mainly focusing on machine learning and data analysis with minimal IO (such as Go, JavaScript (nodejs/deno..), Rust, C#, Swift, etc.) Ultimately, it depends on the requirements. If it's for the web, it's mostly about I/O. It might be a good idea to create it with asyncio.

Slide 27

Slide 27 text

Why Not Using Asyncio? Reasons to Consider •Synchronous processing, which we're used to, is suf fi cient. •Primarily CPU-bound tasks: •Asynchronous processing seems like it might behave unpredictably. •We might choose to write in another language Mainly focusing on machine learning and data analysis with minimal IO (such as Go, JavaScript (nodejs/deno..), Rust, C#, Swift, etc.)

Slide 28

Slide 28 text

"What happens if an exception occurs in this process?"

Slide 29

Slide 29 text

Have you ever mentioned or been asked this during a review? "What happens if an exception occurs in this process?" ʢ´-`ʣ.ŇoO

Slide 30

Slide 30 text

Are you using asyncio? •Asynchronous processing seems to have unpredictable behavior. •that's true •Seems like it might behave unpredictably

Slide 31

Slide 31 text

Are you using asyncio? •Asynchronous processing seems to have unpredictable behavior. •that's true •Seems like it might behave unpredictably •When an exception occurs, it's unclear what happens •“What happens to this concurrently running task if an exception occursʁ” •During an exception, what on earth happens to the other tasks...

Slide 32

Slide 32 text

Are you using asyncio? •Asynchronous processing seems to have unpredictable behavior. •that's true •Seems like it might behave unpredictably •When an exception occurs, it's unclear what happens •“What happens to this concurrently running task if an exception occursʁ” •During an exception, what on earth happens to the other tasks... •A new feature that solves these issues

Slide 33

Slide 33 text

asyncio.TaskGroup

Slide 34

Slide 34 text

•Let's start with the of fi cial documentation •The basic feature of asyncio.TaskGroup •multiple tasks are handled concurrently. •Similar functionality was available even before Python 3.10. •asyncio.gather() and asyncio.wait() asycio.TaskGroup

Slide 35

Slide 35 text

asycio.TaskGroup import asyncio async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(some_coro(...)) task2 = tg.create_task(another_coro(...)) print("Both tasks have completed now.")

Slide 36

Slide 36 text

asycio.TaskGroup import asyncio async def main(): await asyncio.gather( some_coro(...), another_coro(...) ) print("Both tasks have completed now.")

Slide 37

Slide 37 text

Features of asyncio.TaskGroup •Capturing concurrent exceptions

Slide 38

Slide 38 text

Features of asyncio.TaskGroup •Capturing concurrent exceptions •Cancelling tasks on exception

Slide 39

Slide 39 text

Features of asyncio.TaskGroup •Capturing concurrent exceptions •Cancelling tasks on exception

Slide 40

Slide 40 text

•Handle multiple tasks at once Capturing concurrent exceptions •Naturally, exceptions may occur at the same time •Let's see how to capture it in Python 3.10 or earlier •Using the high-level API asyncio.gather() added in Python 3.7 •Options for exceptions to asyncio.gather()ʮreturn_exceptionsʯ •return_exceptions False •return_exceptions True •Simultaneous execution of speci fi ed tasks (asynchronous functions)

Slide 41

Slide 41 text

Here is today's code

Slide 42

Slide 42 text

•Prepare asynchronous functions to run concurrently Let's get set up

Slide 43

Slide 43 text

async def coro_success(): return "DONE" async def coro_value_err(): raise ValueError async def coro_type_err(): raise TypeError

Slide 44

Slide 44 text

Capturing concurrent exceptions - Python 3.10 or earlier •Options for exceptions to asyncio.gather() ”return_exceptions” •return_exceptions False •return_exceptions True

Slide 45

Slide 45 text

async def main(): “""return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}") asyncio.run(main()) asyncio.gather - return_exceptions False Capturing Concurrent Exceptions

Slide 46

Slide 46 text

async def main(): “""return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}") asyncio.run(main()) asyncio.gather - return_exceptions False Capturing Concurrent Exceptions

Slide 47

Slide 47 text

async def main(): “""return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}") asyncio.run(main()) asyncio.gather - return_exceptions False Capturing Concurrent Exceptions

Slide 48

Slide 48 text

$ python310 1_asyncio_gather_except.py async def main(): “""return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}") asyncio.run(main()) asyncio.gather - return_exceptions False Capturing Concurrent Exceptions

Slide 49

Slide 49 text

$ python310 1_asyncio_gather_except.py err=ValueError() $ python310 async def main(): “""return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}") asyncio.run(main()) asyncio.gather - return_exceptions False Capturing Concurrent Exceptions

Slide 50

Slide 50 text

$ python310 1_asyncio_gather_except.py err=ValueError() $ python310 async def main(): “""return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}") asyncio.run(main()) asyncio.gather - return_exceptions False Capturing Concurrent Exceptions ValueError only TypeError cannot be caught

Slide 51

Slide 51 text

$ python310 1_asyncio_gather_except.py err=ValueError() $ python310 ValueError only TypeError cannot be caught Only the fi rst exception is captured The rest are suppressed. async def main(): “""return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}") asyncio.run(main()) asyncio.gather - return_exceptions False Capturing Concurrent Exceptions

Slide 52

Slide 52 text

Capturing concurrent exceptions - Python 3.10 or earlier •Options for exceptions to asyncio.gather() ”return_exceptions” •return_exceptions False •return_exceptions True

Slide 53

Slide 53 text

Capturing concurrent exceptions - Python 3.10 or earlier •Options for exceptions to asyncio.gather() ”return_exceptions” •return_exceptions False •return_exceptions True return_exceptions If speci fi ed Get a list of results after all tasks have been executed Need to check the return list.

Slide 54

Slide 54 text

async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") asyncio.gather - return_exceptions True Capturing Concurrent Exceptions

Slide 55

Slide 55 text

async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") asyncio.gather - return_exceptions True Capturing Concurrent Exceptions

Slide 56

Slide 56 text

async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") asyncio.gather - return_exceptions True Capturing Concurrent Exceptions

Slide 57

Slide 57 text

$ python310 2_asyncio_gather_except_true.py async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") asyncio.gather - return_exceptions True Capturing Concurrent Exceptions

Slide 58

Slide 58 text

$ python310 2_asyncio_gather_except_true.py async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") $ python310 asyncio.gather - return_exceptions True Capturing Concurrent Exceptions results=['success', ValueError(), TypeError()]

Slide 59

Slide 59 text

$ python310 2_asyncio_gather_except_true.py async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") results=['success', ValueError(), TypeError()] $ python310 Get all results in a list Need to see what exceptions were raised asyncio.gather - return_exceptions True Capturing Concurrent Exceptions

Slide 60

Slide 60 text

for result in results: match result: case ValueError(): print("ValueError") case TypeError(): print("TypeError") async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") asyncio.gather - return_exceptions True Capturing Concurrent Exceptions

Slide 61

Slide 61 text

for result in results: match result: case ValueError(): print("ValueError") case TypeError(): print("TypeError") async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") asyncio.gather - return_exceptions True Capturing Concurrent Exceptions

Slide 62

Slide 62 text

$ python310 2_asyncio_gather_except_true.py async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") for result in results: match result: case ValueError(): print("ValueError") case TypeError(): print("TypeError") asyncio.gather - return_exceptions True Capturing Concurrent Exceptions

Slide 63

Slide 63 text

$ python310 2_asyncio_gather_except_true.py async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") results=[‘DONE', ValueError(), TypeError()] ValueError() for result in results: match result: case ValueError(): print("ValueError") case TypeError(): print("TypeError") TypeError() asyncio.gather - return_exceptions True Capturing Concurrent Exceptions

Slide 64

Slide 64 text

asyncio.gather - return_exceptions True $ python310 2_asyncio_gather_except_true.py async def main(): """return_exceptions=True""" results = await asyncio.gather( coro_success(), coro_value_err(), coro_type_err(), return_exceptions=True) print(f"{results=}") results=[‘DONE', ValueError(), TypeError()] ValueError() for result in results: match result: case ValueError(): print("ValueError") case TypeError(): print("TypeError") TypeError() Need to check results All tasks need to be performed Capturing Concurrent Exceptions

Slide 65

Slide 65 text

Capturing Concurrent Exceptions Python 3.10 and earlier issues •Unable to capture except for the fi rst exception. •Cannot log all exceptions that occur •return_exceptions False

Slide 66

Slide 66 text

Capturing Concurrent Exceptions Python 3.10 and earlier issues •Unable to capture except for the fi rst exception. •Cannot log all exceptions that occur •Need to check the return list. •All tasks must be performed •return_exceptions False •return_exceptions True

Slide 67

Slide 67 text

Capturing Concurrent Exceptions Python 3.10 and earlier issues The solution is asyncio.TaskGroup and the new "except*" syntax

Slide 68

Slide 68 text

•I will execute the same async function as before. Let's get set up

Slide 69

Slide 69 text

async def coro_success(): return "DONE" async def coro_value_err(): raise ValueError async def coro_type_err(): raise TypeError

Slide 70

Slide 70 text

async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(coro_success()) task2 = tg.create_task(coro_value_err()) task3 = tg.create_task(coro_type_err()) results = [task1.result(), task2.result(), task3.result()] Capturing Exceptions in asycio.TaskGroup

Slide 71

Slide 71 text

try: async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(coro_success()) task2 = tg.create_task(coro_value_err()) task3 = tg.create_task(coro_type_err()) results = [task1.result(), task2.result(), task3.result()] Capturing Exceptions in asycio.TaskGroup

Slide 72

Slide 72 text

async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(coro_success()) task2 = tg.create_task(coro_value_err()) task3 = tg.create_task(coro_type_err()) results = [task1.result(), task2.result(), task3.result()] except* ValueError as err: print(f"{err=}") try: Capturing Exceptions in asycio.TaskGroup

Slide 73

Slide 73 text

try: async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(coro_success()) task2 = tg.create_task(coro_value_err()) task3 = tg.create_task(coro_type_err()) results = [task1.result(), task2.result(), task3.result()] except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f"{err=}") Capturing Exceptions in asycio.TaskGroup

Slide 74

Slide 74 text

try: async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(coro_success()) task2 = tg.create_task(coro_value_err()) task3 = tg.create_task(coro_type_err()) results = [task1.result(), task2.result(), task3.result()] except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f"{err=}") Using except* Capturing Exceptions in asycio.TaskGroup

Slide 75

Slide 75 text

try: async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(coro_success()) task2 = tg.create_task(coro_value_err()) task3 = tg.create_task(coro_type_err()) results = [task1.result(), task2.result(), task3.result()] except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f"{err=}") Capturing Exceptions in asycio.TaskGroup

Slide 76

Slide 76 text

try: async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(coro_success()) task2 = tg.create_task(coro_value_err()) task3 = tg.create_task(coro_type_err()) results = [task1.result(), task2.result(), task3.result()] except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f"{err=}") $ python311 3_asyncio_taskgroup_except.py Capturing Exceptions in asycio.TaskGroup

Slide 77

Slide 77 text

try: async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(coro_success()) task2 = tg.create_task(coro_value_err()) task3 = tg.create_task(coro_type_err()) results = [task1.result(), task2.result(), task3.result()] except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f"{err=}") $ python311 3_asyncio_taskgroup_except.py err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()]) err=ExceptionGroup('unhandled errors in a TaskGroup', [TypeError()]) $ python311 Capturing Exceptions in asycio.TaskGroup

Slide 78

Slide 78 text

try: async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(coro_success()) task2 = tg.create_task(coro_value_err()) task3 = tg.create_task(coro_type_err()) results = [task1.result(), task2.result(), task3.result()] except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f"{err=}") $ python311 3_asyncio_taskgroup_except.py err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()]) err=ExceptionGroup('unhandled errors in a TaskGroup', [TypeError()]) $ python311 Two Exception are captured. Capturing Exceptions in asycio.TaskGroup

Slide 79

Slide 79 text

asyncio.TaskGroup and new syntax except* •Capturing Exceptions •Easy to read

Slide 80

Slide 80 text

asyncio.TaskGroup and new syntax except* •Capturing Exceptions •Easy to read This is the fi rst feature of asyncio.TaskGroup

Slide 81

Slide 81 text

try: async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(coro_success()) task2 = tg.create_task(coro_value_err()) task3 = tg.create_task(coro_type_err()) results = [task1.result(), task2.result(), task3.result()] except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f"{err=}") Capturing Exceptions in asycio.TaskGroup

Slide 82

Slide 82 text

except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f"{err=}") except* except* Capturing Exceptions in asycio.TaskGroup

Slide 83

Slide 83 text

Capturing Exceptions in asycio.TaskGroup except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f"{err=}") except* except* err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()]) err=ExceptionGroup('unhandled errors in a TaskGroup', [TypeError()])

Slide 84

Slide 84 text

except* except* •PEP 654: Exception Groups and except* •Python 3.11new syntax •Use with ExceptionGroup new syntax

Slide 85

Slide 85 text

new syntax except* except* •PEP654 Motivation 5 cases •Concurrent errors Born for what? •There is no good way to handle exceptions sent by multiple tasks. •asyncio.gather()’s issues

Slide 86

Slide 86 text

asycio.TaskGroup IUUQTHJUIVCDPNQZUIPODQZUIPOCMPCNBJO-JCBTZODJPUBTLHSPVQTQZ class TaskGroup: ... async def __aexit__(self, et, exc, tb): ... if self._errors: # Exceptions are heavy objects that can have object # cycles (bad for GC); let's not keep a reference to # a bunch of them. errors = self._errors self._errors = None me = BaseExceptionGroup('unhandled errors in a TaskGroup', errors) raise me from None ExceptionGroup

Slide 87

Slide 87 text

class TaskGroup: ... async def __aexit__(self, et, exc, tb): ... if self._errors: # Exceptions are heavy objects that can have object # cycles (bad for GC); let's not keep a reference to # a bunch of them. errors = self._errors self._errors = None me = BaseExceptionGroup('unhandled errors in a TaskGroup', errors) raise me from None IUUQTHJUIVCDPNQZUIPODQZUIPOCMPCNBJO-JCBTZODJPUBTLHSPVQTQZ asycio.TaskGroup ExceptionGroup

Slide 88

Slide 88 text

IUUQTHJUIVCDPNQZUIPODQZUIPOCMPCNBJO-JCBTZODJPUBTLHSPVQTQZ class TaskGroup: ... async def __aexit__(self, et, exc, tb): ... if self._errors: # Exceptions are heavy objects that can have object # cycles (bad for GC); let's not keep a reference to # a bunch of them. errors = self._errors self._errors = None me = BaseExceptionGroup('unhandled errors in a TaskGroup', errors) raise me from None asycio.TaskGroup ExceptionGroup

Slide 89

Slide 89 text

•Exception Groups and except: Irit Katriel at PyCon UK •https://www.youtube.com/watch?v=uARIj9eAZcQ •The 2021 Python Language Summit: PEP 654 — Exception Groups and except* •https://pyfound.blogspot.com/2021/05/the-2021-python-language-summit-pep-654.html • How Exception Groups Will Improve Error Handling in AsyncIO - Łukasz Langa | Power IT Conference •https://www.youtube.com/watch?v=Lfe2zsGS0Js reference

Slide 90

Slide 90 text

Features of asyncio.TaskGroup •Capturing concurrent exceptions •Cancelling tasks on exception

Slide 91

Slide 91 text

Cancel task when exception occurs •Concurrent task behavior during exceptionsʁ •Tasks may continue running. •Example: asyncio.gather()

Slide 92

Slide 92 text

•Add one asynchronous function - wait a bit Let's get set up

Slide 93

Slide 93 text

async def coro_success(): return "DONE" async def coro_value_err(): raise ValueError async def coro_type_err(): raise TypeError

Slide 94

Slide 94 text

async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return “DONEʁ"

Slide 95

Slide 95 text

async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return “DONEʁ"

Slide 96

Slide 96 text

Cases where tasks remain - asyncio.gather - return_exceptions false async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f"{err=}") print("DONE") asyncio.run(main())

Slide 97

Slide 97 text

Cases where tasks remain - asyncio.gather - return_exceptions false async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f"{err=}") print("DONE") asyncio.run(main())

Slide 98

Slide 98 text

async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f"{err=}") print("DONE") asyncio.run(main()) Cases where tasks remain - asyncio.gather - return_exceptions false

Slide 99

Slide 99 text

Cases where tasks remain - asyncio.gather - return_exceptions false async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f"{err=}") print("DONE") asyncio.run(main())

Slide 100

Slide 100 text

$ python310 4_remaining_tasks.py Cases where tasks remain - asyncio.gather - return_exceptions false async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f"{err=}") print("DONE") asyncio.run(main())

Slide 101

Slide 101 text

$ python310 4_remaining_tasks.py Cases where tasks remain - asyncio.gather - return_exceptions false async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f"{err=}") print("DONE") asyncio.run(main()) err=ValueError() DONE

Slide 102

Slide 102 text

$ python310 4_remaining_tasks.py err=ValueError() DONE coro_long: Task is still running. Cases where tasks remain - asyncio.gather - return_exceptions false async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f"{err=}") print("DONE") asyncio.run(main())

Slide 103

Slide 103 text

$ python310 4_remaining_tasks.py err=ValueError() DONE coro_long: Task is still running. If an exception occurs Other tasks remain running Hard to predict behavior Cases where tasks remain - asyncio.gather - return_exceptions false async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f"{err=}") print("DONE") asyncio.run(main())

Slide 104

Slide 104 text

$ python310 4_remaining_tasks.py err=ValueError() DONE coro_long: Task is still running. I want to stop. I want to roll back Cases where tasks remain - asyncio.gather - return_exceptions false If an exception occurs Other tasks remain running Hard to predict behavior async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f"{err=}") print("DONE") asyncio.run(main())

Slide 105

Slide 105 text

Cases where tasks remain - asyncio.gather - return_exceptions false $ python310 4_remaining_tasks.py coro_long: Task is still running. ɾࢭΊ͍ͨ ɾϩʔϧόοΫ͍ͨ͠ ྫ֎͕ൃੜͨ͠৔߹ ଞͷλεΫ͸ಈ͍ͨ·· 
 ಈ࡞Λ༧ଌͮ͠Β͍ I want to stop. I want to roll back If an exception occurs Other tasks remain running Hard to predict behavior async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f"{err=}") print("DONE") asyncio.run(main()) err=ValueError() DONE

Slide 106

Slide 106 text

Cancel task when exception occurs •If an exception occurs, what happens to the remaining tasks that were running at the same timeʁ •In some cases, the remaining tasks are executed as they are.

Slide 107

Slide 107 text

Cancel task when exception occurs •In some cases, the remaining tasks are executed as they are. •Cancel a task (Cancellations were not added on 3.11) •Cancellation Basics •Cancel when using asyncio.gather() before 3.10 •Let's fi x the process that leaves the task •If an exception occurs, what happens to the remaining tasks that were running at the same timeʁ

Slide 108

Slide 108 text

Cancel task when exception occurs •Let's fi x the process that leaves the task •Asynchronous function (coro_long()) to be canceled •Instructions to check and cancel remaining tasks •Wait for post-cancellation processing to complete. Added post-processing for cancellations •There are three perspectives for modi fi cation

Slide 109

Slide 109 text

Cancel task when exception occurs •Let's fi x the process that leaves the task •Asynchronous function (coro_long()) to be canceled •Instructions to check and cancel remaining tasks •Wait for post-cancellation processing to complete. Added post-processing for cancellations •There are three perspectives for modi fi cation

Slide 110

Slide 110 text

Cancel task when exception occurs •Let's fi x the process that leaves the task •Asynchronous function (coro_long()) to be canceled •Instructions to check and cancel remaining tasks •Wait for post-cancellation processing to complete. Added post-processing for cancellations •There are three perspectives for modi fi cation

Slide 111

Slide 111 text

Cancel task when exception occurs •Let's fi x the process that leaves the task •Asynchronous function (coro_long()) to be canceled •Instructions to check and cancel remaining tasks •Wait for post-cancellation processing to complete. Added post-processing for cancellations •There are three perspectives for modi fi cation

Slide 112

Slide 112 text

•task.cancel() •Cancel request to a task Added post-processing for cancellations

Slide 113

Slide 113 text

•task.cancel() •Cancel request to a task •CancelledError exception is sent to the cancelled task •receive asyncio.CancelledError in an asynchronous function •Resend asyncio.CancelledError Added post-processing for cancellations

Slide 114

Slide 114 text

•task.cancel() •Cancel request to a task •CancelledError exception is sent to the cancelled task •receive asyncio.CancelledError in an asynchronous function •Resend asyncio.CancelledError •I'm a little confused. Let's try to implement it. Added post-processing for cancellations

Slide 115

Slide 115 text

Let's get set up •receive asyncio.CancelledError in an asynchronous function •Resend asyncio.CancelledError

Slide 116

Slide 116 text

async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?"

Slide 117

Slide 117 text

async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" try:

Slide 118

Slide 118 text

async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try:

Slide 119

Slide 119 text

async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError

Slide 120

Slide 120 text

Cancel task when exception occurs •Let's fi x the process that leaves the task •Asynchronous function (coro_long()) to be canceled •Instructions to check and cancel remaining tasks •Wait for post-cancellation processing to complete. Added post-processing for cancellations •There are three perspectives for modi fi cation

Slide 121

Slide 121 text

Cancel task when exception occurs •Let's fi x the process that leaves the task •Asynchronous function (coro_long()) to be canceled •Instructions to check and cancel remaining tasks •Wait for post-cancellation processing to complete. Added post-processing for cancellations •There are three perspectives for modi fi cation

Slide 122

Slide 122 text

•Request a task to cancel - Task.cancel() Instructions to check and cancel remaining tasks •Determine completion of a task - Task.done() •Wait for post-cancel process to complete - asyncio.sleep() •Let's modify the code.

Slide 123

Slide 123 text

async def main(): """return_exceptions False""" try: print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) asyncio.run(main()) Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() print("DONE")

Slide 124

Slide 124 text

async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) print("DONE") Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather()

Slide 125

Slide 125 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) Easy to check task status The process is the same print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() print("DONE")

Slide 126

Slide 126 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: Identifying un fi nished tasks with Task.done() Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() print("DONE")

Slide 127

Slide 127 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: task.cancel() # Cancel unfinished tasks Task.cancel() for un fi nished tasks Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() print("DONE")

Slide 128

Slide 128 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: Waiting for completion asyncio.sleep() await asyncio.sleep(1) Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() print("DONE") task.cancel() # Cancel unfinished tasks

Slide 129

Slide 129 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: After modi fi cation await asyncio.sleep(1) Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() print("DONE") task.cancel() # Cancel unfinished tasks

Slide 130

Slide 130 text

Reviewing the Code Before Changes Output

Slide 131

Slide 131 text

async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) print("DONE") asyncio.run(main()) $ python310 4_remaining_tasks.py err=ValueError() DONE coro_long: Task is still running. Cases where tasks remain - asyncio.gather - return_exceptions false Review Execution before modi fi cation

Slide 132

Slide 132 text

async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) print("DONE") asyncio.run(main()) $ python310 4_remaining_tasks.py coro_long: Task is still running. Cases where tasks remain - asyncio.gather - return_exceptions false Review Execution before modi fi cation Output from remaining a task err=ValueError() DONE

Slide 133

Slide 133 text

Cases where tasks remain - asyncio.gather - return_exceptions false async def main(): """return_exceptions False""" try: results = await asyncio.gather( coro_success(), coro_value_err(), coro_long() ) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) print("DONE") asyncio.run(main()) $ python310 4_remaining_tasks.py coro_long: Task is still running. Review Execution before modi fi cation Output from remaining a task err=ValueError() DONE

Slide 134

Slide 134 text

async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError

Slide 135

Slide 135 text

async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError

Slide 136

Slide 136 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: task.cancel() # Cancel unfinished tasks print("DONE") await asyncio.sleep(1) Run it async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather()

Slide 137

Slide 137 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) $ python310 5_canceling_task_gather.py async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() task.cancel() # Cancel unfinished tasks print("DONE")

Slide 138

Slide 138 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) $ python310 5_canceling_task_gather.py err=ValueError() async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() task.cancel() # Cancel unfinished tasks print("DONE")

Slide 139

Slide 139 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) $ python310 5_canceling_task_gather.py err=ValueError() coro_long: Task was cancelled. async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() task.cancel() # Cancel unfinished tasks print("DONE")

Slide 140

Slide 140 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) $ python310 5_canceling_task_gather.py err=ValueError() coro_long: Task was cancelled. async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError DONE Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() task.cancel() # Cancel unfinished tasks print("DONE")

Slide 141

Slide 141 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) If canceled On the asynchronous function side Catch Cancel $ python310 5_canceling_task_gather.py err=ValueError() coro_long: Task was cancelled. async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() task.cancel() # Cancel unfinished tasks print("DONE") DONE

Slide 142

Slide 142 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) asyncio.run(main()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) $ python310 5_canceling_task_gather.py err=ValueError() coro_long: Task was cancelled. async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError basic principle of cancellation Cases where tasks remain → Add instructions to con fi rm and cancel tasks Python 3.10 or earlier - asyncio.gather() task.cancel() # Cancel unfinished tasks print("DONE") DONE If canceled On the asynchronous function side Catch Cancel

Slide 143

Slide 143 text

Cancel task when exception occurs •Let's fi x the process that leaves the task •Asynchronous function (coro_long()) to be canceled •Instructions to check and cancel remaining tasks •Wait for post-cancellation processing to complete. Added post-processing for cancellations •There are three perspectives for modi fi cation

Slide 144

Slide 144 text

•At the time of exiting the context manager, •Remaining tasks are canceled. •Wait for cancellation process to complete. Cancellation process in asycio.TaskGroup

Slide 145

Slide 145 text

•At the time of exiting the context manager, •Remaining tasks are canceled. •Wait for cancellation process to complete. •That's all! •In short, nothing special needs to be done. It will be canceled. Let's refactor the code we just wrote. Cancellation process in asycio.TaskGroup

Slide 146

Slide 146 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) asyncio.run(main()) print(f"{results=}") for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) Cancellation process in asycio.TaskGroup Let's refactor it. results = await asyncio.gather(*[task1, task2, task3]) except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) print("DONE") task.cancel() # Cancel unfinished tasks

Slide 147

Slide 147 text

async def main(): """return_exceptions False""" try: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) asyncio.run(main()) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) results = await asyncio.gather(*[task1, task2, task3]) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) Cancellation process in asycio.TaskGroup Let's refactor it. print("DONE") task.cancel() # Cancel unfinished tasks

Slide 148

Slide 148 text

async def main(): """Cancel in TaskGroup""" try: asyncio.run(main()) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) async with asyncio.TaskGroup() as g: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) results = await asyncio.gather(*[task1, task2, task3]) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) Cancellation process in asycio.TaskGroup Let's refactor it. print("DONE") task.cancel() # Cancel unfinished tasks

Slide 149

Slide 149 text

asyncio.run(main()) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) async with asyncio.TaskGroup() as g: task1 = asyncio.create_task(coro_success()) task2 = asyncio.create_task(coro_value_err()) task3 = asyncio.create_task(coro_long()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) Cancellation process in asycio.TaskGroup Let's refactor it. print("DONE") task.cancel() # Cancel unfinished tasks async def main(): """Cancel in TaskGroup""" try:

Slide 150

Slide 150 text

asyncio.run(main()) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) async with asyncio.TaskGroup() as g: task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) Cancellation process in asycio.TaskGroup Let's refactor it. print("DONE") task.cancel() # Cancel unfinished tasks async def main(): """Cancel in TaskGroup""" try:

Slide 151

Slide 151 text

asyncio.run(main()) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) async with asyncio.TaskGroup() as g: print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) Cancellation process in asycio.TaskGroup Let's refactor it. print("DONE") task.cancel() # Cancel unfinished tasks async def main(): """Cancel in TaskGroup""" try:

Slide 152

Slide 152 text

asyncio.run(main()) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) async with asyncio.TaskGroup() as g: print(f"{results=}") except ValueError as err: print(f"{err=}") except TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) Cancellation process in asycio.TaskGroup Let's refactor it. print("DONE") task.cancel() # Cancel unfinished tasks async def main(): """Cancel in TaskGroup""" try:

Slide 153

Slide 153 text

asyncio.run(main()) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) async with asyncio.TaskGroup() as g: print(f"{results=}") except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) Cancellation process in asycio.TaskGroup Let's refactor it. print("DONE") task.cancel() # Cancel unfinished tasks async def main(): """Cancel in TaskGroup""" try:

Slide 154

Slide 154 text

asyncio.run(main()) for task in [task1, task2, task3]: if task.done() is False: await asyncio.sleep(1) async with asyncio.TaskGroup() as g: print(f"{results=}") except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) Cancellation process in asycio.TaskGroup Let's refactor it. print("DONE") task.cancel() # Cancel unfinished tasks async def main(): """Cancel in TaskGroup""" try:

Slide 155

Slide 155 text

asyncio.run(main()) async with asyncio.TaskGroup() as g: print(f"{results=}") except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) Cancellation process in asycio.TaskGroup Let's refactor it. print("DONE") async def main(): """Cancel in TaskGroup""" try:

Slide 156

Slide 156 text

async def main(): """Cancel in TaskGroup""" try: asyncio.run(main()) async with asyncio.TaskGroup() as g: print(f"{results=}") except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) Cancellation process in asycio.TaskGroup Let's refactor it. print("DONE")

Slide 157

Slide 157 text

Let's run it. async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError asyncio.run(main()) async with asyncio.TaskGroup() as g: print(f"{results=}") except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) $ python311 6_cancelled_task_taskgroup.py Cancellation process in asycio.TaskGroup print("DONE") async def main(): """Cancel in TaskGroup""" try:

Slide 158

Slide 158 text

Let's run it. async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError asyncio.run(main()) async with asyncio.TaskGroup() as g: print(f"{results=}") except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) $ python311 6_cancelled_task_taskgroup.py coro_long: Task was cancelled. Cancellation process in asycio.TaskGroup print("DONE") async def main(): """Cancel in TaskGroup""" try:

Slide 159

Slide 159 text

Let's run it. async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError asyncio.run(main()) async with asyncio.TaskGroup() as g: print(f"{results=}") except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) $ python311 6_cancelled_task_taskgroup.py err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()]) coro_long: Task was cancelled. Cancellation process in asycio.TaskGroup print("DONE") async def main(): """Cancel in TaskGroup""" try:

Slide 160

Slide 160 text

Let's run it. async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError asyncio.run(main()) async with asyncio.TaskGroup() as g: print(f"{results=}") except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) $ python311 6_cancelled_task_taskgroup.py err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()]) coro_long: Task was cancelled. DONE Cancellation process in asycio.TaskGroup print("DONE") async def main(): """Cancel in TaskGroup""" try:

Slide 161

Slide 161 text

Capturing Exceptions and Processing after canceling The order of Capturing and processing after cancellation is swapped. async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError asyncio.run(main()) async with asyncio.TaskGroup() as g: print(f"{results=}") except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) $ python311 6_cancelled_task_taskgroup.py err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()]) coro_long: Task was cancelled. Cancellation process in asycio.TaskGroup print("DONE") DONE async def main(): """Cancel in TaskGroup""" try:

Slide 162

Slide 162 text

In TaskGroup complete Predictable behavior async def coro_long(): await asyncio.sleep(1) print("coro_long: Task is still running.") return "DONE?" except asyncio.CancelledError as err: try: print("coro_long: Task was cancelled.") raise asyncio.CancelledError asyncio.run(main()) async with asyncio.TaskGroup() as g: print(f"{results=}") except* ValueError as err: print(f"{err=}") except* TypeError as err: print(f”{err=}”) results = [task1.result(), task2.result(), task3.result()] task1 = g.create_task(coro_success()) task2 = g.create_task(coro_value_err()) task3 = g.create_task(coro_long()) Cancellation process in asycio.TaskGroup $ python311 6_cancelled_task_taskgroup.py err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()]) coro_long: Task was cancelled. print("DONE") DONE async def main(): """Cancel in TaskGroup""" try:

Slide 163

Slide 163 text

asycio.TaskGroup Features •Capturing Concurrent Exceptions •Cancel task when exception occurs •Simply catch exceptions in a conventional way of writing •Cancel without indication •Except* will not spare any concurrent exceptions. •No longer need to be aware of tasks when executing them

Slide 164

Slide 164 text

asycio.TaskGroup Features •Capturing Concurrent Exceptions •Cancel task when exception occurs •Simply catch exceptions in a conventional way of writing •Cancel without indication •Except* will not spare any concurrent exceptions. •No longer need to be aware of tasks when executing them •Writing asyncio code in a "reliable" and "safe" way •The new standard "Hello-ish world"

Slide 165

Slide 165 text

import asyncio async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(some_coro(...)) task2 = tg.create_task(another_coro(...)) New “Hello-ish world”

Slide 166

Slide 166 text

import asyncio async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(some_coro(...)) task2 = tg.create_task(another_coro(...)) try: except* BaseException as e: print(f"{e=}") asyncio.run(main()) New “Hello-ish world”

Slide 167

Slide 167 text

New “Hello-ish world” import asyncio async def main(): async with asyncio.TaskGroup() as tg: task1 = tg.create_task(some_coro(...)) task2 = tg.create_task(another_coro(...)) try: except* BaseException as e: print(f"{e=}") asyncio.run(main()) Python 3.11

Slide 168

Slide 168 text

•Quickstart - Example 3-2. The “Hello-ish World” of Asyncio import asyncio import time async def main(): print(f"{time.ctime()} Hello!") await asyncio.sleep(1.0) print(f"{time.ctime()} GoodBye!") loop = asyncio.get_event_loop() task = loop.create_task(main()) loop.run_until_complete(task) pending = asyncio.all_tasks(loop=loop) for task in pending: task.cancel() group = asyncio.gather(*pending, return_exceptions=True) loop.run_until_complete(group) loop.close()

Slide 169

Slide 169 text

•Quickstart - Example 3-2. import asyncio import time async def main(): print(f"{time.ctime()} Hello!") await asyncio.sleep(1.0) print(f"{time.ctime()} GoodBye!") The “Hello-ish World” of Asyncio Checking task status Waiting for cancellation process loop = asyncio.get_event_loop() task = loop.create_task(main()) loop.run_until_complete(task) pending = asyncio.all_tasks(loop=loop) for task in pending: task.cancel() group = asyncio.gather(*pending, return_exceptions=True) loop.run_until_complete(group) loop.close()

Slide 170

Slide 170 text

•Quickstart - Example 3-2. import asyncio import time async def coro(): print(f"{time.ctime()} Hello!") await asyncio.sleep(1.0) print(f"{time.ctime()} GoodBye!") The “Hello-ish World” of Asyncio Checking task status Waiting for cancellation process async def main(): async with asyncio.TaskGroup() as tg: task = tg.create_task(coro(...)) try: except* BaseException as e: print(f"{e=}") asyncio.run(main())

Slide 171

Slide 171 text

Summary •Capturing and canceling exceptions in asyncio.TaskGroup •asyncio for external I/O •Because of external dependence, exceptions are likely to occur. •Easy to write and read •Let's touch TaskGroup. •Exception/cancellation triggers a review of the existing code.

Slide 172

Slide 172 text

Summary •Capturing and canceling exceptions in asyncio.TaskGroup •asyncio for external I/O •Because of external dependence, exceptions are likely to occur. •Easy to write and read •Let's touch TaskGroup. •Exception/cancellation triggers a review of the existing code. •A more comfortable asyncio life

Slide 173

Slide 173 text

•Task Status Appendix: Task Status •PENDING •FINISHED •CANCELLED •You can check with task._state •The sample code also outputs the state. Please try to run it. Here is the code

Slide 174

Slide 174 text

Appendix: HTTPX with TaskGroup import asyncio from time import time import httpx URL = "https://pokeapi.co/api/v2/pokemon/" async def access_url_poke(client, num: int) -> str: r = await client.get(f"{URL}{num}") pokemon = r.json() # JSONύʔε return pokemon["name"] # ϙέ໊Λൈ͘ async def main_poke(): """httpxͰϙέϞϯ151ඖҾͬ͜ൈ͘""" start = time() async with httpx.AsyncClient() as client: try: async with asyncio.TaskGroup() as tg: tasks = [ tg.create_task(access_url_poke(client, number)) for number in range(1, 151) ] except* BaseException as e: print(f"{e=}") results = [task.result() for task in tasks] print(results) print("time: ", time() - start) asyncio.run(main_poke()) Here is the code

Slide 175

Slide 175 text

Appendix: New features that are still available! import asyncio async def main(): try: async with asyncio.TaskGroup() as tg: task1 = tg.create_task(some_coro(...)) task2 = tg.create_task(another_coro(...)) except* BaseException as e: print(f"{e=}") with asyncio.Runner() as runner: runner.run(main()) 3.11 What's New What is asyncio.Runner! For interactive sessions? run multiple times. Here is the code

Slide 176

Slide 176 text

Appendix: How to Cancel and Track Future States •It helps to think of Futures as being in one of four states: •Waiting •Done, holding a result •Done, holding an exception /FPQZUIPOJD3FBTPOJOHBCPVUBTZODJP4FNBQIPSF IUUQOFPQZUIPOJDCMPHTQPUDPNSFBTPOJOHBCPVUBTZODJPTFNBQIPSFIUNM •Done, but cancelled

Slide 177

Slide 177 text

reference •Using Asyncio in Python - Oreilly & Associates Inc •Python 3.11 Preview: Task and Exception Groups - realpython.com •https://realpython.com/python311-exception-groups/ •New in Python 3.11 (Part 4) PEP 654 Exception Groups and TaskGroups •https://www.python.jp/news/wnpython311/except-grp.html •The 2021 Python Language Summit: PEP 654 — Exception Groups and except* •https://pyfound.blogspot.com/2021/05/the-2021-python-language-summit-pep-654.html • How Exception Groups Will Improve Error Handling in AsyncIO - Łukasz Langa | Power IT Conference •https://www.youtube.com/watch?v=Lfe2zsGS0Js

Slide 178

Slide 178 text

advertisement 🙏

Slide 179

Slide 179 text

Who am I? 👤 •Junya Fukudaʢ@JunyaFffʣcalled “Jun” not “Junior” •Photos 📷 Tweets 🐦 👍 •Software Engineer at Groove X, inc •book •Python Practical Recipes •Expert Python Programming - Fourth Edition •Written by Tarek Ziadé, Michał Jaworski Here is today's code

Slide 180

Slide 180 text

Do you know Groove X, inc？🤔

Slide 181

Slide 181 text

Do you know Groove X, inc？🤔

Slide 182

Slide 182 text

What is Groove X

Slide 183

Slide 183 text

Groove X is ... •We manufacture and sell a home robot called LOVOT •LOVOT is “Love ✖︎ Robot”

Slide 184

Slide 184 text

What is Groove X •Do you know Groove X ❓ • Python inside 🐍🌷

Slide 185

Slide 185 text

Would you like to see them?

Slide 186

Slide 186 text

Would you like to see them? Possibly Launching in Taiwan

Slide 187

Slide 187 text

Groove X is a sponsor of PyCon Apac2023. See you in Tokyo. Oct27-28 2023

Slide 188

Slide 188 text

Groove X is a sponsor of PyCon Apac2023. See you in Tokyo. Oct27-28 2023

Slide 189

Slide 189 text

No content

Slide 190

Slide 190 text

Thank you for your attention 👋

Slide 191

Slide 191 text

Do you have a any Question? Feel free to ask questions on X 🐦. (@junya ff f)