Demystifying Coroutines and
Asynchronous Programming in
Python
Mariano Anaya
@rmarianoa
Slide 2
Slide 2 text
History
● PEP-255: Simple generators
● PEP-342: Coroutines via enhanced
generators
● PEP-380: Syntax for delegating to a
sub-generator
● PEP-492: Coroutines with async and await
syntax
● PEP-525: Asynchronous generators
Slide 3
Slide 3 text
Generators
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
Generate elements, one at the time,
and suspend...
● Save memory
● Support iteration pattern, infinite sequences, etc.
Slide 6
Slide 6 text
Simple Generators - Example 1
LIMIT = 1_000_000
def old_range(n):
numbers = []
i = 0
while i < n:
numbers.append(i)
i += 1
return numbers
def new_range(n):
i = 0
while i < n:
yield i
i += 1
Don’t Do
Slide 7
Slide 7 text
Simple Generators - Example 2
def new_range(n):
i = 0
while i < n:
yield i
i += 1
total = sum(new_range(LIMIT))
total = 0
i = 0
while i < LIMIT:
total += i
i += 1
Don’t Do
Slide 8
Slide 8 text
Simple Generators
● next() will advance the generator until the next yield
statement.
○ Produces a value to the caller, and suspends there.
○ If there aren’t more elements, StopIteration is
raised
Slide 9
Slide 9 text
Coroutines
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
Generators as coroutines
● Suspend?
● How about sending (receiving) data to (from) a
generator?
● And exceptions?
.send()
.throw()
.close()
Slide 12
Slide 12 text
● Coroutines are syntactically like generators
● With .send(), the caller pushes data into the coroutine.
○ yield usually appears on the RHS
value = yield result
● The coroutine is suspended at the yield
Coroutines via Enhanced Generators
Slide 13
Slide 13 text
>>> c = coro()
>>> next(c)
>>> step = c.send(received)
def coro():
step = 0
while True:
received = yield step
step += 1
print("Received: ", received)
Slide 14
Slide 14 text
Advance the Generator
Before sending any value to the generator, this has to be
advanced with:
next(coroutine) | coroutine.send(None)
If not, TypeError is raised.
Slide 15
Slide 15 text
Coroutines via Enhanced Generators
.throw(exc_type[, exc_value[, ex_tb]])
Raises the exception at the point where the coroutine is
suspended.
Slide 16
Slide 16 text
More Coroutines
Slide 17
Slide 17 text
No content
Slide 18
Slide 18 text
Delegating to a Sub-Generator
● Generators can now return values!
● yield from
○ Gets all values from an iterable object
○ Produce the values from the sub-generator
○ Open a channel to the internal generator
○ Can get the value returned by the internal generator
Slide 19
Slide 19 text
Generators as Coroutines - Return values
StopIteration.value contains the result.
→ Once the return is reached, there is no more iteration.
>>> def gen():
...: yield 1
...: yield 2
...: return 42
>>> g = gen()
>>> next(g)
1
>>> next(g)
2
>>> next(g)
------------------------------------
StopIteration
Traceback (most recent call last)
StopIteration: 42
Slide 20
Slide 20 text
yield from
● Get all the elements from an iterable
● Delegate to a sub-generator
Slide 21
Slide 21 text
yield from - Basic
Something in the form...
yield from
Could be thought of as...
for e in :
yield e
Slide 22
Slide 22 text
~ itertools.chain
>>> def chain2(*iterables):
... for it in iterables:
... yield from it
>>> list(chain2([1,2,3], (4, 5, 6), "hello"))
[1, 2, 3, 4, 5, 6, 'h', 'e', 'l', 'l', 'o']
Slide 23
Slide 23 text
yield from - More
Chain generators
● .send(), and .throw() are passed along.
● Returned (yielded) values, bubble up.
yield from acts as a “channel” from the original caller, to
the internal generators.
Slide 24
Slide 24 text
yield from - Example
def internal(name, limit):
for i in range(limit):
value = yield i
print(f"{name} got: {value}")
def general():
yield from internal("first", 10)
yield from internal("second", 20)
Slide 25
Slide 25 text
yield from - Recap
● Allows delegating to a sub-generator
● Enables chaining generators and many
iterables together
● Makes it easier to refactor generators
await
Works like yield from, except that:
● Does not accept generators that aren’t coroutines
● Accepts awaitable objects
○ __await__()
Slide 30
Slide 30 text
asyncio
● An event loop drives the coroutines scheduled to run,
and updates them with .send(), next(),
.throw(), etc.
● The coroutine we write, should only delegate with
await (yield from), to some other 3rd party generator,
that will do the actual I/O.
● yield , yield from, await give the control back to the
scheduler.
Slide 31
Slide 31 text
Asynchronous
Generators
Slide 32
Slide 32 text
No content
Slide 33
Slide 33 text
Asynchronous Generators
● Before Python 3.6, it was not possible to have a yield in
a coroutine :-(
○ “async def” only allowed “return” or “await”
● “Produce elements, one at the time, asynchronously”:
○ async for x in data_producer: ...
○ Asynchronous iterables, were required
● Difference between iterator (__iter__ / __next__),
vs. a generator
○ But for asynchronous code (__aiter__ / __anext__), vs. async
generator
Slide 34
Slide 34 text
Summary
● Generators and coroutines are conceptually different,
however they share implementation details.
● yield from as a construction evolved into await, to
allow more powerful coroutines.
● Any await chain of calls ends with a yield (at the end,
there is a generator).