Slide 1
Slide 1 text
Asyncio Evolved:
Enhanced Exception Handling with
TaskGroup in Python 3.11
Junya Fukuda
EuroPython 2023
Slide 2
Slide 2 text
Who am I? đ¤
â˘book
â˘Python Practical Recipes
â˘Expert Python Programming - Fourth Edition
â˘Written by Tarek ZiadĂŠ, MichaĹ Jaworski
Here is today's code
â˘Junya Fukudaʢ@JunyaFffĘŁcalled âJunâ not âJuniorâ
â˘Photos đˇ Tweets đŚ đ
â˘Software Engineer at Groove X, inc.
Slide 3
Slide 3 text
asyncio's new features and
New "Hello-ish world"
Slide 4
Slide 4 text
Today's Goal
â˘Focus on exception handling and cancellation.
â˘Introducing the new API asyncio.TaskGroup.
â˘Writing asyncio code in a "reliable" and "safe" way
â˘The new standard "Hello-ish world"
Slide 5
Slide 5 text
Why are we talking about thisĘ
There are two main reasons.
Slide 6
Slide 6 text
Motivation (why are we talking about this?)
â˘There are two main reasons.
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
async def main():
print(f"{time.ctime()} Hello!")
await asyncio.sleep(1.0)
print(f"{time.ctime()} GoodBye!")
loop = asyncio.get_event_loop()
task = loop.create_task(main())
loop.run_until_complete(task)
pending = asyncio.all_tasks(loop=loop)
for task in pending:
task.cancel()
group = asyncio.gather(*pending, return_exceptions=True)
loop.run_until_complete(group)
loop.close()
Slide 11
Slide 11 text
Motivation (why are we talking about this?)
â˘I would like to revamp this code with asyncio.TaskGroup
Slide 12
Slide 12 text
Agenda
â˘Introduction
â˘Capturing concurrent exceptions
â˘Cancelling tasks on exception
â˘The new standard "Hello-ish world"
Slide 13
Slide 13 text
Scope of Today's Talk
â˘Talk.
â˘Exception handling in asyncio (also new except* syntax)
â˘Cancel Task
â˘asyncio's New Hello World
â˘Don't talk.
â˘Speci
fi
c examples in the application
â˘What's happening at ASGI
Here is today's code
Slide 14
Slide 14 text
asyncio, are you using it? â
Slide 15
Slide 15 text
Overview of asyncio
â˘asyncio: Python's async library.
â˘Continues to evolve.
â˘Handling multiple tasks concurrently.
â˘Useful for I/O-bound tasks like database or HTTP requests.
Slide 16
Slide 16 text
đ¤
Slide 17
Slide 17 text
Why Not Using Asyncio? Reasons to Consider
â˘Synchronous processing, which we're used to, is suf
fi
cient.
â˘Primarily CPU-bound tasks:
â˘Asynchronous processing seems like it might behave unpredictably.
â˘We might choose to write in another language
Mainly focusing on machine learning and data analysis with minimal IO
(such as Go, JavaScript (nodejs/deno..), Rust, C#, Swift, etc.)
Slide 18
Slide 18 text
Why Not Using Asyncio? Reasons to Consider
â˘Synchronous processing, which we're used to, is suf
fi
cient.
â˘Primarily CPU-bound tasks:
â˘Asynchronous processing seems like it might behave unpredictably.
â˘We might choose to write in another language
Mainly focusing on machine learning and data analysis with minimal IO
(such as Go, JavaScript (nodejs/deno..), Rust, C#, Swift, etc.)
Slide 19
Slide 19 text
Why Not Using Asyncio? Reasons to Consider
â˘Synchronous processing, which we're used to, is suf
fi
cient.
â˘Primarily CPU-bound tasks:
â˘Asynchronous processing seems like it might behave unpredictably.
â˘We might choose to write in another language
Mainly focusing on machine learning and data analysis with minimal IO
(such as Go, JavaScript (nodejs/deno..), Rust, C#, Swift, etc.)
Slide 20
Slide 20 text
Why Not Using Asyncio? Reasons to Consider
â˘Synchronous processing, which we're used to, is suf
fi
cient.
â˘Primarily CPU-bound tasks:
â˘Asynchronous processing seems like it might behave unpredictably.
â˘We might choose to write in another language
Mainly focusing on machine learning and data analysis with minimal IO
(such as Go, JavaScript (nodejs/deno..), Rust, C#, Swift, etc.)
Ultimately, it depends on the requirements.
If it's for the web, it's mostly about I/O.
It might be a good idea to create it with asyncio.
Slide 21
Slide 21 text
Why Not Using Asyncio? Reasons to Consider
â˘Synchronous processing, which we're used to, is suf
fi
cient.
â˘Primarily CPU-bound tasks:
â˘Asynchronous processing seems like it might behave unpredictably.
â˘We might choose to write in another language
Mainly focusing on machine learning and data analysis with minimal IO
(such as Go, JavaScript (nodejs/deno..), Rust, C#, Swift, etc.)
Slide 22
Slide 22 text
"What happens
if an exception occurs in this process?"
Slide 23
Slide 23 text
Have you ever mentioned or been asked this during a review?
"What happens
if an exception occurs in this process?"
ʢ´-`ĘŁ.ĹoO
Slide 24
Slide 24 text
Are you using asyncio?
â˘Asynchronous processing seems to have unpredictable behavior.
â˘that's true
â˘Seems like it might behave unpredictably
â˘When an exception occurs, it's unclear what happens
â˘âWhat happens to this concurrently running task if an exception occursĘâ
â˘During an exception, what on earth happens to the other tasks...
â˘A new feature that solves these issues
Slide 25
Slide 25 text
asyncio.TaskGroup
Slide 26
Slide 26 text
â˘Let's start with the of
fi
cial documentation
â˘The basic feature of asyncio.TaskGroup
â˘multiple tasks are handled concurrently.
â˘Similar functionality was available even before Python 3.10.
â˘asyncio.gather() and asyncio.wait()
asycio.TaskGroup
Slide 27
Slide 27 text
asycio.TaskGroup
import asyncio
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(some_coro(...))
task2 = tg.create_task(another_coro(...))
print("Both tasks have completed now.")
Slide 28
Slide 28 text
asycio.TaskGroup
import asyncio
async def main():
await asyncio.gather(
some_coro(...),
another_coro(...)
)
print("Both tasks have completed now.")
Slide 29
Slide 29 text
Features of asyncio.TaskGroup
â˘Capturing concurrent exceptions
Slide 30
Slide 30 text
Features of asyncio.TaskGroup
â˘Capturing concurrent exceptions
â˘Cancelling tasks on exception
Slide 31
Slide 31 text
Features of asyncio.TaskGroup
â˘Capturing concurrent exceptions
â˘Cancelling tasks on exception
Slide 32
Slide 32 text
â˘Handle multiple tasks at once
Capturing concurrent exceptions
â˘Naturally, exceptions may occur at the same time
â˘Let's see how to capture it in Python 3.10 or earlier
â˘Using the high-level API asyncio.gather() added in Python 3.7
â˘Options for exceptions to asyncio.gather()ĘŽreturn_exceptionsĘŻ
â˘return_exceptions False
â˘return_exceptions True
â˘Simultaneous execution of speci
fi
ed tasks (asynchronous functions)
Slide 33
Slide 33 text
Here is today's code
Slide 34
Slide 34 text
â˘Prepare asynchronous functions to run concurrently
Let's get set up
Slide 35
Slide 35 text
async def coro_success():
return "DONE"
async def coro_value_err():
raise ValueError
async def coro_type_err():
raise TypeError
Slide 36
Slide 36 text
Capturing concurrent exceptions - Python 3.10 or earlier
â˘Options for exceptions to asyncio.gather() âreturn_exceptionsâ
â˘return_exceptions False
â˘return_exceptions True
Slide 37
Slide 37 text
async def main():
â""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}")
asyncio.run(main())
asyncio.gather - return_exceptions False Capturing Concurrent Exceptions
Slide 38
Slide 38 text
async def main():
â""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}")
asyncio.run(main())
asyncio.gather - return_exceptions False Capturing Concurrent Exceptions
Slide 39
Slide 39 text
async def main():
â""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}")
asyncio.run(main())
asyncio.gather - return_exceptions False Capturing Concurrent Exceptions
Slide 40
Slide 40 text
$ python310 1_asyncio_gather_except.py
async def main():
â""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}")
asyncio.run(main())
asyncio.gather - return_exceptions False Capturing Concurrent Exceptions
Slide 41
Slide 41 text
$ python310 1_asyncio_gather_except.py
err=ValueError()
$ python310
async def main():
â""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}")
asyncio.run(main())
asyncio.gather - return_exceptions False Capturing Concurrent Exceptions
Slide 42
Slide 42 text
$ python310 1_asyncio_gather_except.py
err=ValueError()
$ python310
async def main():
â""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}")
asyncio.run(main())
asyncio.gather - return_exceptions False Capturing Concurrent Exceptions
ValueError only
TypeError cannot be caught
Slide 43
Slide 43 text
$ python310 1_asyncio_gather_except.py
err=ValueError()
$ python310
ValueError only
TypeError cannot be caught
Only the
fi
rst exception is captured
The rest are suppressed.
async def main():
â""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}")
asyncio.run(main())
asyncio.gather - return_exceptions False Capturing Concurrent Exceptions
Slide 44
Slide 44 text
Capturing concurrent exceptions - Python 3.10 or earlier
â˘Options for exceptions to asyncio.gather() âreturn_exceptionsâ
â˘return_exceptions False
â˘return_exceptions True
Slide 45
Slide 45 text
Capturing concurrent exceptions - Python 3.10 or earlier
â˘Options for exceptions to asyncio.gather() âreturn_exceptionsâ
â˘return_exceptions False
â˘return_exceptions True
return_exceptions If speci
fi
ed
Get a list of results after all tasks have been executed
Need to check the return list.
Slide 46
Slide 46 text
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
asyncio.gather - return_exceptions True Capturing Concurrent Exceptions
Slide 47
Slide 47 text
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
asyncio.gather - return_exceptions True Capturing Concurrent Exceptions
Slide 48
Slide 48 text
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
asyncio.gather - return_exceptions True Capturing Concurrent Exceptions
Slide 49
Slide 49 text
$ python310 2_asyncio_gather_except_true.py
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
asyncio.gather - return_exceptions True Capturing Concurrent Exceptions
Slide 50
Slide 50 text
$ python310 2_asyncio_gather_except_true.py
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
$ python310
asyncio.gather - return_exceptions True Capturing Concurrent Exceptions
results=['success', ValueError(), TypeError()]
Slide 51
Slide 51 text
$ python310 2_asyncio_gather_except_true.py
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
results=['success', ValueError(), TypeError()]
$ python310
Get all results in a list
Need to see what exceptions were
raised
asyncio.gather - return_exceptions True Capturing Concurrent Exceptions
Slide 52
Slide 52 text
for result in results:
match result:
case ValueError():
print("ValueError")
case TypeError():
print("TypeError")
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
asyncio.gather - return_exceptions True Capturing Concurrent Exceptions
Slide 53
Slide 53 text
for result in results:
match result:
case ValueError():
print("ValueError")
case TypeError():
print("TypeError")
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
asyncio.gather - return_exceptions True Capturing Concurrent Exceptions
Slide 54
Slide 54 text
$ python310 2_asyncio_gather_except_true.py
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
for result in results:
match result:
case ValueError():
print("ValueError")
case TypeError():
print("TypeError")
asyncio.gather - return_exceptions True Capturing Concurrent Exceptions
Slide 55
Slide 55 text
$ python310 2_asyncio_gather_except_true.py
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
results=[âDONE', ValueError(), TypeError()]
ValueError()
for result in results:
match result:
case ValueError():
print("ValueError")
case TypeError():
print("TypeError")
TypeError()
asyncio.gather - return_exceptions True Capturing Concurrent Exceptions
Slide 56
Slide 56 text
asyncio.gather - return_exceptions True
$ python310 2_asyncio_gather_except_true.py
async def main():
"""return_exceptions=True"""
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_type_err(),
return_exceptions=True)
print(f"{results=}")
results=[âDONE', ValueError(), TypeError()]
ValueError()
for result in results:
match result:
case ValueError():
print("ValueError")
case TypeError():
print("TypeError")
TypeError()
Need to check results
All tasks need to be performed
Capturing Concurrent Exceptions
Slide 57
Slide 57 text
Capturing Concurrent Exceptions Python 3.10 and earlier issues
â˘Unable to capture except for the
fi
rst exception.
â˘Cannot log all exceptions that occur
â˘return_exceptions False
Slide 58
Slide 58 text
Capturing Concurrent Exceptions Python 3.10 and earlier issues
â˘Unable to capture except for the
fi
rst exception.
â˘Cannot log all exceptions that occur
â˘Need to check the return list.
â˘All tasks must be performed
â˘return_exceptions False
â˘return_exceptions True
Slide 59
Slide 59 text
Capturing Concurrent Exceptions Python 3.10 and earlier issues
The solution is asyncio.TaskGroup
and the new "except*" syntax
Slide 60
Slide 60 text
â˘I will execute the same async function as before.
Let's get set up
Slide 61
Slide 61 text
async def coro_success():
return "DONE"
async def coro_value_err():
raise ValueError
async def coro_type_err():
raise TypeError
Slide 62
Slide 62 text
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(coro_success())
task2 = tg.create_task(coro_value_err())
task3 = tg.create_task(coro_type_err())
results = [task1.result(), task2.result(), task3.result()]
Capturing Exceptions in asycio.TaskGroup
Slide 63
Slide 63 text
try:
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(coro_success())
task2 = tg.create_task(coro_value_err())
task3 = tg.create_task(coro_type_err())
results = [task1.result(), task2.result(), task3.result()]
Capturing Exceptions in asycio.TaskGroup
Slide 64
Slide 64 text
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(coro_success())
task2 = tg.create_task(coro_value_err())
task3 = tg.create_task(coro_type_err())
results = [task1.result(), task2.result(), task3.result()]
except* ValueError as err:
print(f"{err=}")
try:
Capturing Exceptions in asycio.TaskGroup
Slide 65
Slide 65 text
try:
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(coro_success())
task2 = tg.create_task(coro_value_err())
task3 = tg.create_task(coro_type_err())
results = [task1.result(), task2.result(), task3.result()]
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(f"{err=}")
Capturing Exceptions in asycio.TaskGroup
Slide 66
Slide 66 text
try:
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(coro_success())
task2 = tg.create_task(coro_value_err())
task3 = tg.create_task(coro_type_err())
results = [task1.result(), task2.result(), task3.result()]
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(f"{err=}") Using except*
Capturing Exceptions in asycio.TaskGroup
Slide 67
Slide 67 text
try:
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(coro_success())
task2 = tg.create_task(coro_value_err())
task3 = tg.create_task(coro_type_err())
results = [task1.result(), task2.result(), task3.result()]
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(f"{err=}")
Capturing Exceptions in asycio.TaskGroup
Slide 68
Slide 68 text
try:
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(coro_success())
task2 = tg.create_task(coro_value_err())
task3 = tg.create_task(coro_type_err())
results = [task1.result(), task2.result(), task3.result()]
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(f"{err=}")
$ python311 3_asyncio_taskgroup_except.py
Capturing Exceptions in asycio.TaskGroup
Slide 69
Slide 69 text
try:
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(coro_success())
task2 = tg.create_task(coro_value_err())
task3 = tg.create_task(coro_type_err())
results = [task1.result(), task2.result(), task3.result()]
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(f"{err=}")
$ python311 3_asyncio_taskgroup_except.py
err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()])
err=ExceptionGroup('unhandled errors in a TaskGroup', [TypeError()])
$ python311
Capturing Exceptions in asycio.TaskGroup
Slide 70
Slide 70 text
try:
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(coro_success())
task2 = tg.create_task(coro_value_err())
task3 = tg.create_task(coro_type_err())
results = [task1.result(), task2.result(), task3.result()]
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(f"{err=}")
$ python311 3_asyncio_taskgroup_except.py
err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()])
err=ExceptionGroup('unhandled errors in a TaskGroup', [TypeError()])
$ python311
Two Exception
are captured.
Capturing Exceptions in asycio.TaskGroup
Slide 71
Slide 71 text
asyncio.TaskGroup and new syntax except*
â˘Capturing Exceptions
â˘Easy to read
Slide 72
Slide 72 text
asyncio.TaskGroup and new syntax except*
â˘Capturing Exceptions
â˘Easy to read
This is the
fi
rst feature of asyncio.TaskGroup
Slide 73
Slide 73 text
try:
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(coro_success())
task2 = tg.create_task(coro_value_err())
task3 = tg.create_task(coro_type_err())
results = [task1.result(), task2.result(), task3.result()]
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(f"{err=}")
Capturing Exceptions in asycio.TaskGroup
Slide 74
Slide 74 text
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(f"{err=}")
except*
except*
Capturing Exceptions in asycio.TaskGroup
Slide 75
Slide 75 text
Capturing Exceptions in asycio.TaskGroup
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(f"{err=}")
except*
except*
err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()])
err=ExceptionGroup('unhandled errors in a TaskGroup', [TypeError()])
Slide 76
Slide 76 text
except*
except*
â˘PEP 654: Exception Groups and except*
â˘Python 3.11new syntax
â˘Use with ExceptionGroup
new syntax
Slide 77
Slide 77 text
new syntax except*
except*
â˘PEP654 Motivation 5 cases
â˘Concurrent errors
Born for what?
â˘There is no good way to handle exceptions sent by multiple tasks.
â˘asyncio.gather()âs issues
Slide 78
Slide 78 text
asycio.TaskGroup
IUUQTHJUIVCDPNQZUIPODQZUIPOCMPCNBJO-JCBTZODJPUBTLHSPVQTQZ
class TaskGroup:
...
async def __aexit__(self, et, exc, tb):
...
if self._errors:
# Exceptions are heavy objects that can have object
# cycles (bad for GC); let's not keep a reference to
# a bunch of them.
errors = self._errors
self._errors = None
me = BaseExceptionGroup('unhandled errors in a TaskGroup', errors)
raise me from None
ExceptionGroup
Slide 79
Slide 79 text
class TaskGroup:
...
async def __aexit__(self, et, exc, tb):
...
if self._errors:
# Exceptions are heavy objects that can have object
# cycles (bad for GC); let's not keep a reference to
# a bunch of them.
errors = self._errors
self._errors = None
me = BaseExceptionGroup('unhandled errors in a TaskGroup', errors)
raise me from None
IUUQTHJUIVCDPNQZUIPODQZUIPOCMPCNBJO-JCBTZODJPUBTLHSPVQTQZ
asycio.TaskGroup ExceptionGroup
Slide 80
Slide 80 text
IUUQTHJUIVCDPNQZUIPODQZUIPOCMPCNBJO-JCBTZODJPUBTLHSPVQTQZ
class TaskGroup:
...
async def __aexit__(self, et, exc, tb):
...
if self._errors:
# Exceptions are heavy objects that can have object
# cycles (bad for GC); let's not keep a reference to
# a bunch of them.
errors = self._errors
self._errors = None
me = BaseExceptionGroup('unhandled errors in a TaskGroup', errors)
raise me from None
asycio.TaskGroup ExceptionGroup
Slide 81
Slide 81 text
â˘Exception Groups and except: Irit Katriel at PyCon UK
â˘https://www.youtube.com/watch?v=uARIj9eAZcQ
â˘The 2021 Python Language Summit: PEP 654 â Exception Groups and except*
â˘https://pyfound.blogspot.com/2021/05/the-2021-python-language-summit-pep-654.html
⢠How Exception Groups Will Improve Error Handling in AsyncIO - Ĺukasz Langa | Power IT Conference
â˘https://www.youtube.com/watch?v=Lfe2zsGS0Js
reference
Slide 82
Slide 82 text
Features of asyncio.TaskGroup
â˘Capturing concurrent exceptions
â˘Cancelling tasks on exception
Slide 83
Slide 83 text
Cancel task when exception occurs
â˘Concurrent task behavior during exceptionsĘ
â˘Tasks may continue running.
â˘Example: asyncio.gather()
Slide 84
Slide 84 text
â˘Add one asynchronous function - wait a bit
Let's get set up
Slide 85
Slide 85 text
async def coro_success():
return "DONE"
async def coro_value_err():
raise ValueError
async def coro_type_err():
raise TypeError
Slide 86
Slide 86 text
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return âDONEĘ"
Slide 87
Slide 87 text
Cases where tasks remain - asyncio.gather - return_exceptions false
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(f"{err=}")
print("DONE")
asyncio.run(main())
Slide 88
Slide 88 text
Cases where tasks remain - asyncio.gather - return_exceptions false
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(f"{err=}")
print("DONE")
asyncio.run(main())
Slide 89
Slide 89 text
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(f"{err=}")
print("DONE")
asyncio.run(main())
Cases where tasks remain - asyncio.gather - return_exceptions false
Slide 90
Slide 90 text
Cases where tasks remain - asyncio.gather - return_exceptions false
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(f"{err=}")
print("DONE")
asyncio.run(main())
Slide 91
Slide 91 text
$ python310 4_remaining_tasks.py
Cases where tasks remain - asyncio.gather - return_exceptions false
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(f"{err=}")
print("DONE")
asyncio.run(main())
Slide 92
Slide 92 text
$ python310 4_remaining_tasks.py
Cases where tasks remain - asyncio.gather - return_exceptions false
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(f"{err=}")
print("DONE")
asyncio.run(main())
err=ValueError()
DONE
Slide 93
Slide 93 text
$ python310 4_remaining_tasks.py
err=ValueError()
DONE
coro_long: Task is still running.
Cases where tasks remain - asyncio.gather - return_exceptions false
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(f"{err=}")
print("DONE")
asyncio.run(main())
Slide 94
Slide 94 text
$ python310 4_remaining_tasks.py
err=ValueError()
DONE
coro_long: Task is still running.
If an exception occurs
Other tasks remain running Hard
to predict behavior
Cases where tasks remain - asyncio.gather - return_exceptions false
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(f"{err=}")
print("DONE")
asyncio.run(main())
Slide 95
Slide 95 text
$ python310 4_remaining_tasks.py
err=ValueError()
DONE
coro_long: Task is still running.
I want to stop.
I want to roll back
Cases where tasks remain - asyncio.gather - return_exceptions false
If an exception occurs
Other tasks remain running Hard
to predict behavior
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(f"{err=}")
print("DONE")
asyncio.run(main())
Slide 96
Slide 96 text
Cases where tasks remain - asyncio.gather - return_exceptions false
$ python310 4_remaining_tasks.py
coro_long: Task is still running.
Éžŕ˘ÎͨÍ
ɞϊĘϧĎÎżÎŤÍ Í¨Í
ྫÖÍŕľŕŠÍ ͨŕ§ßš
ŕŹÍˇÎťÎľÎŤÍ¸ŕ˛ÍͨÎÎ
â¨
ŕ˛ŕĄÎ༧ŕŹÍ ÍŽÎÍ
I want to stop.
I want to roll back
If an exception occurs
Other tasks remain running Hard
to predict behavior
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(f"{err=}")
print("DONE")
asyncio.run(main())
err=ValueError()
DONE
Slide 97
Slide 97 text
Cancel task when exception occurs
â˘If an exception occurs,
what happens to the remaining tasks that were running at the same time?
â˘In some cases, the remaining tasks are executed as they are.
Slide 98
Slide 98 text
Cancel task when exception occurs
â˘If an exception occurs,
what happens to the remaining tasks that were running at the same time?
â˘In some cases, the remaining tasks are executed as they are.
â˘Cancel a task (Cancellations were not added on 3.11)
â˘Cancellation Basics
â˘Cancel when using asyncio.gather() before 3.10
â˘Let's
fi
x the process that leaves the task
Slide 99
Slide 99 text
Cancel task when exception occurs
â˘Let's
fi
x the process that leaves the task
â˘Asynchronous function (coro_long()) to be canceled
â˘Instructions to check and cancel remaining tasks
â˘Wait for post-cancellation processing to complete.
Added post-processing for cancellations
â˘There are three perspectives for modi
fi
cation
Slide 100
Slide 100 text
â˘task.cancel()
â˘Cancel request to a task
Added post-processing for cancellations
Slide 101
Slide 101 text
â˘task.cancel()
â˘Cancel request to a task
â˘CancelledError exception is sent to the cancelled task
â˘receive asyncio.CancelledError in an asynchronous function
â˘Resend asyncio.CancelledError
Added post-processing for cancellations
Slide 102
Slide 102 text
â˘task.cancel()
â˘Cancel request to a task
â˘CancelledError exception is sent to the cancelled task
â˘receive asyncio.CancelledError in an asynchronous function
â˘Resend asyncio.CancelledError
â˘I'm a little confused. Let's try to implement it.
Added post-processing for cancellations
Slide 103
Slide 103 text
Let's get set up
â˘receive asyncio.CancelledError in an asynchronous function
â˘Resend asyncio.CancelledError
Slide 104
Slide 104 text
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
Slide 105
Slide 105 text
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
try:
Slide 106
Slide 106 text
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
Slide 107
Slide 107 text
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
Slide 108
Slide 108 text
Cancel task when exception occurs
â˘Let's
fi
x the process that leaves the task
â˘Asynchronous function (coro_long()) to be canceled
â˘Instructions to check and cancel remaining tasks
â˘Wait for post-cancellation processing to complete.
Added post-processing for cancellations
â˘There are three perspectives for modi
fi
cation
Slide 109
Slide 109 text
Cancel task when exception occurs
â˘Let's
fi
x the process that leaves the task
â˘Asynchronous function (coro_long()) to be canceled
â˘Instructions to check and cancel remaining tasks
â˘Wait for post-cancellation processing to complete.
Added post-processing for cancellations
â˘There are three perspectives for modi
fi
cation
Slide 110
Slide 110 text
â˘Request a task to cancel - Task.cancel()
Instructions to check and cancel remaining tasks
â˘Determine completion of a task - Task.done()
â˘Wait for post-cancel process to complete - asyncio.sleep()
â˘Let's modify the code.
Slide 111
Slide 111 text
async def main():
"""return_exceptions False"""
try:
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
asyncio.run(main())
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
print("DONE")
Slide 112
Slide 112 text
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
print("DONE")
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
Slide 113
Slide 113 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
Easy to check task status
The process is the same
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
print("DONE")
Slide 114
Slide 114 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False:
Identifying un
fi
nished tasks with
Task.done()
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
print("DONE")
Slide 115
Slide 115 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False:
task.cancel() # Cancel unfinished tasks
Task.cancel() for
un
fi
nished tasks
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
print("DONE")
Slide 116
Slide 116 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False: Waiting for
completion
asyncio.sleep()
await asyncio.sleep(1)
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
print("DONE")
task.cancel() # Cancel unfinished tasks
Slide 117
Slide 117 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False:
After modi
fi
cation
await asyncio.sleep(1)
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
print("DONE")
task.cancel() # Cancel unfinished tasks
Slide 118
Slide 118 text
Review a bit of the output in case a task remains
Slide 119
Slide 119 text
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
print("DONE")
asyncio.run(main())
$ python310 4_remaining_tasks.py
err=ValueError()
DONE
coro_long: Task is still running.
Cases where tasks remain - asyncio.gather - return_exceptions false
Review
Execution before modi
fi
cation
Slide 120
Slide 120 text
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
print("DONE")
asyncio.run(main())
$ python310 4_remaining_tasks.py
coro_long: Task is still running.
Cases where tasks remain - asyncio.gather - return_exceptions false
Review
Execution before modi
fi
cation
Output from remaining tasks
err=ValueError()
DONE
Slide 121
Slide 121 text
Cases where tasks remain - asyncio.gather - return_exceptions false
async def main():
"""return_exceptions False"""
try:
results = await asyncio.gather(
coro_success(), coro_value_err(), coro_long()
)
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
print("DONE")
asyncio.run(main())
$ python310 4_remaining_tasks.py
coro_long: Task is still running.
Review
Execution before modi
fi
cation
Output from remaining tasks
err=ValueError()
DONE
Slide 122
Slide 122 text
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
Slide 123
Slide 123 text
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
Slide 124
Slide 124 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False:
task.cancel() # Cancel unfinished tasks
print("DONE")
await asyncio.sleep(1)
Run it
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
Slide 125
Slide 125 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
$ python310 5_canceling_task_gather.py
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
task.cancel() # Cancel unfinished tasks
print("DONE")
Slide 126
Slide 126 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
$ python310 5_canceling_task_gather.py
err=ValueError()
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
task.cancel() # Cancel unfinished tasks
print("DONE")
Slide 127
Slide 127 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
$ python310 5_canceling_task_gather.py
err=ValueError()
coro_long: Task was cancelled.
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
task.cancel() # Cancel unfinished tasks
print("DONE")
Slide 128
Slide 128 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
$ python310 5_canceling_task_gather.py
err=ValueError()
coro_long: Task was cancelled.
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
DONE
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
task.cancel() # Cancel unfinished tasks
print("DONE")
Slide 129
Slide 129 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
If canceled On the asynchronous
function side Catch Cancel
$ python310 5_canceling_task_gather.py
err=ValueError()
coro_long: Task was cancelled.
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
task.cancel() # Cancel unfinished tasks
print("DONE")
DONE
Slide 130
Slide 130 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
asyncio.run(main())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
$ python310 5_canceling_task_gather.py
err=ValueError()
coro_long: Task was cancelled.
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
basic principle of cancellation
Cases where tasks remain â Add instructions to con
fi
rm and cancel tasks
Python 3.10 or earlier - asyncio.gather()
task.cancel() # Cancel unfinished tasks
print("DONE")
DONE
If canceled On the asynchronous
function side Catch Cancel
Slide 131
Slide 131 text
Cancel task when exception occurs
â˘Let's
fi
x the process that leaves the task
â˘Asynchronous function (coro_long()) to be canceled
â˘Instructions to check and cancel remaining tasks
â˘Wait for post-cancellation processing to complete.
Added post-processing for cancellations
â˘There are three perspectives for modi
fi
cation
Slide 132
Slide 132 text
â˘At the time of exiting the context manager,
â˘Remaining tasks are canceled.
â˘Wait for cancellation process to complete.
Cancellation process in asycio.TaskGroup
Slide 133
Slide 133 text
â˘At the time of exiting the context manager,
â˘Remaining tasks are canceled.
â˘Wait for cancellation process to complete.
â˘That's all!
â˘In short, nothing special needs to be done. It will be canceled.
Let's refactor the code we just wrote.
Cancellation process in asycio.TaskGroup
Slide 134
Slide 134 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
asyncio.run(main())
print(f"{results=}")
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
Cancellation process in asycio.TaskGroup Let's refactor it.
results = await asyncio.gather(*[task1, task2, task3])
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
print("DONE")
task.cancel() # Cancel unfinished tasks
Slide 135
Slide 135 text
async def main():
"""return_exceptions False"""
try:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
asyncio.run(main())
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
results = await asyncio.gather(*[task1, task2, task3])
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
Cancellation process in asycio.TaskGroup Let's refactor it.
print("DONE")
task.cancel() # Cancel unfinished tasks
Slide 136
Slide 136 text
async def main():
"""Cancel in TaskGroup"""
try:
asyncio.run(main())
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
async with asyncio.TaskGroup() as g:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
results = await asyncio.gather(*[task1, task2, task3])
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
Cancellation process in asycio.TaskGroup Let's refactor it.
print("DONE")
task.cancel() # Cancel unfinished tasks
Slide 137
Slide 137 text
asyncio.run(main())
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
async with asyncio.TaskGroup() as g:
task1 = asyncio.create_task(coro_success())
task2 = asyncio.create_task(coro_value_err())
task3 = asyncio.create_task(coro_long())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
Cancellation process in asycio.TaskGroup Let's refactor it.
print("DONE")
task.cancel() # Cancel unfinished tasks
async def main():
"""Cancel in TaskGroup"""
try:
Slide 138
Slide 138 text
asyncio.run(main())
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
async with asyncio.TaskGroup() as g:
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
Cancellation process in asycio.TaskGroup Let's refactor it.
print("DONE")
task.cancel() # Cancel unfinished tasks
async def main():
"""Cancel in TaskGroup"""
try:
Slide 139
Slide 139 text
asyncio.run(main())
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
Cancellation process in asycio.TaskGroup Let's refactor it.
print("DONE")
task.cancel() # Cancel unfinished tasks
async def main():
"""Cancel in TaskGroup"""
try:
Slide 140
Slide 140 text
asyncio.run(main())
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except ValueError as err:
print(f"{err=}")
except TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
Cancellation process in asycio.TaskGroup Let's refactor it.
print("DONE")
task.cancel() # Cancel unfinished tasks
async def main():
"""Cancel in TaskGroup"""
try:
Slide 141
Slide 141 text
asyncio.run(main())
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
Cancellation process in asycio.TaskGroup Let's refactor it.
print("DONE")
task.cancel() # Cancel unfinished tasks
async def main():
"""Cancel in TaskGroup"""
try:
Slide 142
Slide 142 text
asyncio.run(main())
for task in [task1, task2, task3]:
if task.done() is False:
await asyncio.sleep(1)
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
Cancellation process in asycio.TaskGroup Let's refactor it.
print("DONE")
task.cancel() # Cancel unfinished tasks
async def main():
"""Cancel in TaskGroup"""
try:
Slide 143
Slide 143 text
asyncio.run(main())
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
Cancellation process in asycio.TaskGroup Let's refactor it.
print("DONE")
async def main():
"""Cancel in TaskGroup"""
try:
Slide 144
Slide 144 text
async def main():
"""Cancel in TaskGroup"""
try:
asyncio.run(main())
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
Cancellation process in asycio.TaskGroup Let's refactor it.
print("DONE")
Slide 145
Slide 145 text
Let's run it.
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
asyncio.run(main())
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
$ python311 6_cancelled_task_taskgroup.py
Cancellation process in asycio.TaskGroup
print("DONE")
async def main():
"""Cancel in TaskGroup"""
try:
Slide 146
Slide 146 text
Let's run it.
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
asyncio.run(main())
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
$ python311 6_cancelled_task_taskgroup.py
coro_long: Task was cancelled.
Cancellation process in asycio.TaskGroup
print("DONE")
async def main():
"""Cancel in TaskGroup"""
try:
Slide 147
Slide 147 text
Let's run it.
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
asyncio.run(main())
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
$ python311 6_cancelled_task_taskgroup.py
err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()])
coro_long: Task was cancelled.
Cancellation process in asycio.TaskGroup
print("DONE")
async def main():
"""Cancel in TaskGroup"""
try:
Slide 148
Slide 148 text
Let's run it.
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
asyncio.run(main())
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
$ python311 6_cancelled_task_taskgroup.py
err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()])
coro_long: Task was cancelled.
DONE
Cancellation process in asycio.TaskGroup
print("DONE")
async def main():
"""Cancel in TaskGroup"""
try:
Slide 149
Slide 149 text
Capturing Exceptions and
Processing after canceling
The order of Capturing and
processing after cancellation is
swapped.
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
asyncio.run(main())
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
$ python311 6_cancelled_task_taskgroup.py
err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()])
coro_long: Task was cancelled.
Cancellation process in asycio.TaskGroup
print("DONE")
DONE
async def main():
"""Cancel in TaskGroup"""
try:
Slide 150
Slide 150 text
In TaskGroup
complete
Predictable behavior
async def coro_long():
await asyncio.sleep(1)
print("coro_long: Task is still running.")
return "DONE?"
except asyncio.CancelledError as err:
try:
print("coro_long: Task was cancelled.")
raise asyncio.CancelledError
asyncio.run(main())
async with asyncio.TaskGroup() as g:
print(f"{results=}")
except* ValueError as err:
print(f"{err=}")
except* TypeError as err:
print(fâ{err=}â)
results = [task1.result(), task2.result(), task3.result()]
task1 = g.create_task(coro_success())
task2 = g.create_task(coro_value_err())
task3 = g.create_task(coro_long())
Cancellation process in asycio.TaskGroup
$ python311 6_cancelled_task_taskgroup.py
err=ExceptionGroup('unhandled errors in a TaskGroup', [ValueError()])
coro_long: Task was cancelled.
print("DONE")
DONE
async def main():
"""Cancel in TaskGroup"""
try:
Slide 151
Slide 151 text
asycio.TaskGroup Features
â˘Capturing Concurrent Exceptions
â˘Cancel task when exception occurs
â˘Simply catch exceptions in a conventional way of writing
â˘Cancel without indication
â˘Except* will not spare any concurrent exceptions.
â˘No longer need to be aware of tasks when executing them
Slide 152
Slide 152 text
asycio.TaskGroup Features
â˘Capturing Concurrent Exceptions
â˘Cancel task when exception occurs
â˘Simply catch exceptions in a conventional way of writing
â˘Cancel without indication
â˘Except* will not spare any concurrent exceptions.
â˘No longer need to be aware of tasks when executing them
â˘Writing asyncio code in a "reliable" and "safe" way
â˘The new standard "Hello-ish world"
Slide 153
Slide 153 text
import asyncio
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(some_coro(...))
task2 = tg.create_task(another_coro(...))
New âHello-ish worldâ
Slide 154
Slide 154 text
import asyncio
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(some_coro(...))
task2 = tg.create_task(another_coro(...))
try:
except* BaseException as e:
print(f"{e=}")
asyncio.run(main())
New âHello-ish worldâ
Slide 155
Slide 155 text
New âHello-ish worldâ
import asyncio
async def main():
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(some_coro(...))
task2 = tg.create_task(another_coro(...))
try:
except* BaseException as e:
print(f"{e=}")
asyncio.run(main())
Python 3.11
Slide 156
Slide 156 text
â˘Quickstart - Example 3-2. The âHello-ish Worldâ of Asyncio
import asyncio
import time
async def main():
print(f"{time.ctime()} Hello!")
await asyncio.sleep(1.0)
print(f"{time.ctime()} GoodBye!")
loop = asyncio.get_event_loop()
task = loop.create_task(main())
loop.run_until_complete(task)
pending = asyncio.all_tasks(loop=loop)
for task in pending:
task.cancel()
group = asyncio.gather(*pending, return_exceptions=True)
loop.run_until_complete(group)
loop.close()
Slide 157
Slide 157 text
â˘Quickstart - Example 3-2.
import asyncio
import time
async def main():
print(f"{time.ctime()} Hello!")
await asyncio.sleep(1.0)
print(f"{time.ctime()} GoodBye!")
The âHello-ish Worldâ of Asyncio
Checking task status
Waiting for cancellation
process
loop = asyncio.get_event_loop()
task = loop.create_task(main())
loop.run_until_complete(task)
pending = asyncio.all_tasks(loop=loop)
for task in pending:
task.cancel()
group = asyncio.gather(*pending, return_exceptions=True)
loop.run_until_complete(group)
loop.close()
Slide 158
Slide 158 text
â˘Quickstart - Example 3-2.
import asyncio
import time
async def coro():
print(f"{time.ctime()} Hello!")
await asyncio.sleep(1.0)
print(f"{time.ctime()} GoodBye!")
The âHello-ish Worldâ of Asyncio
Checking task status
Waiting for cancellation
process
async def main():
async with asyncio.TaskGroup() as tg:
task = tg.create_task(coro(...))
try:
except* BaseException as e:
print(f"{e=}")
asyncio.run(main())
Slide 159
Slide 159 text
Summary
â˘Capturing and canceling exceptions in asyncio.TaskGroup
â˘asyncio for external I/O
â˘Because of external dependence, exceptions are likely to occur.
â˘Easy to write and read
â˘Let's touch TaskGroup.
â˘Exception/cancellation triggers a review of the existing code.
Slide 160
Slide 160 text
Summary
â˘Capturing and canceling exceptions in asyncio.TaskGroup
â˘asyncio for external I/O
â˘Because of external dependence, exceptions are likely to occur.
â˘Easy to write and read
â˘Let's touch TaskGroup.
â˘Exception/cancellation triggers a review of the existing code.
â˘A more comfortable asyncio life
Slide 161
Slide 161 text
â˘Task Status
Appendix: Task Status
â˘PENDING
â˘FINISHED
â˘CANCELLED
â˘You can check with task._state
â˘The sample code also outputs the state. Please try to run it.
Here is the code
Slide 162
Slide 162 text
Appendix: HTTPX with TaskGroup
import asyncio
from time import time
import httpx
URL = "https://pokeapi.co/api/v2/pokemon/"
async def access_url_poke(client, num: int) -> str:
r = await client.get(f"{URL}{num}")
pokemon = r.json() # JSONĎĘÎľ
return pokemon["name"] # ĎÎŕťÎŕľÍ
async def main_poke():
"""httpxÍ°ĎÎĎĎŻ151ŕśŇžÍŹÍŕľÍ"""
start = time()
async with httpx.AsyncClient() as client:
try:
async with asyncio.TaskGroup() as tg:
tasks = [
tg.create_task(access_url_poke(client, number))
for number in range(1, 151)
]
except* BaseException as e:
print(f"{e=}")
results = [task.result() for task in tasks]
print(results)
print("time: ", time() - start)
asyncio.run(main_poke())
Here is the code
Slide 163
Slide 163 text
Appendix: New features that are still available!
import asyncio
async def main():
try:
async with asyncio.TaskGroup() as tg:
task1 = tg.create_task(some_coro(...))
task2 = tg.create_task(another_coro(...))
except* BaseException as e:
print(f"{e=}")
with asyncio.Runner() as runner:
runner.run(main())
3.11 What's New
What is asyncio.Runner!
For interactive sessions?
run multiple times.
Here is the code
Slide 164
Slide 164 text
reference
â˘Using Asyncio in Python - Oreilly & Associates Inc
â˘Python 3.11 Preview: Task and Exception Groups - realpython.com
â˘https://realpython.com/python311-exception-groups/
â˘New in Python 3.11 (Part 4) PEP 654 Exception Groups and TaskGroups
â˘https://www.python.jp/news/wnpython311/except-grp.html
â˘The 2021 Python Language Summit: PEP 654 â Exception Groups and except*
â˘https://pyfound.blogspot.com/2021/05/the-2021-python-language-summit-pep-654.html
⢠How Exception Groups Will Improve Error Handling in AsyncIO - Ĺukasz Langa | Power IT Conference
â˘https://www.youtube.com/watch?v=Lfe2zsGS0Js
Slide 165
Slide 165 text
advertisement đ
Slide 166
Slide 166 text
Do you know Groove X, incďźđ¤
Slide 167
Slide 167 text
Do you know Groove X, incďźđ¤
Slide 168
Slide 168 text
What is Groove X
Slide 169
Slide 169 text
Groove X is ...
â˘We manufacture and sell a home robot called
LOVOT
â˘LOVOT is âLove
âď¸
Robotâ
Slide 170
Slide 170 text
What is Groove X
â˘Do you know Groove X â
â˘
Python inside đđˇ
Slide 171
Slide 171 text
Would you like to see them?
Slide 172
Slide 172 text
Groove X is a sponsor of PyCon Apac2023.
See you in Tokyo. Oct27-28 2023
Slide 173
Slide 173 text
Groove X is a sponsor of PyCon Apac2023.
See you in Tokyo. Oct27-28 2023
Slide 174
Slide 174 text
No content
Slide 175
Slide 175 text
Thank you for your attention đ
Slide 176
Slide 176 text
Do you have a any Question?
Feel free to ask questions on Twitter. (@junya
ff
f)