Demystifying coroutines and asynchronous programming in Pyhon

Exploring the internals of how coroutines and asynchronous programming work in Python, from the ground up.

Presented at Python San Sebastián 2018, on October 13th - https://pyss18.pyss.org/

1. Demystifying Coroutines and Asynchronous Programming in Python Mariano Anaya @rmarianoa
2. History • PEP-255: Simple generators • PEP-342: Coroutines via enhanced generators • PEP-380: Syntax for delegating to a sub-generator • PEP-492: Coroutines with async and await syntax • PEP-525: Asynchronous generators
3. Generators
5. Generate elements, one at the time, and suspend... • Save memory • Support iteration pattern, infinite sequences, etc.
6. Simple Generators - Example 1 LIMIT = 1_000_000 def old_range(n): numbers = [] i = 0 while i < n: numbers.append(i) i += 1 return numbers def new_range(n): i = 0 while i < n: yield i i += 1 Don’t Do
7. Simple Generators - Example 2 def new_range(n): i = 0 while i < n: yield i i += 1 total = sum(new_range(LIMIT)) total = 0 i = 0 while i < LIMIT: total += i i += 1 Don’t Do
8. Simple Generators • next() will advance the generator until the next yield statement. ◦ Produces a value to the caller, and suspends there. ◦ If there aren’t more elements, StopIteration is raised
9. Coroutines
11. Generators as coroutines • Suspend? • How about sending (receiving) data to (from) a generator? • And exceptions? <g>.send(<value>) <g>.throw(<exception>) <g>.close()
12. • Coroutines are syntactically like generators • With .send(), the caller pushes data into the coroutine. ◦ yield usually appears on the RHS value = yield result • The coroutine is suspended at the yield Coroutines via Enhanced Generators
13. >>> c = coro() >>> next(c) >>> step = c.send(received) def coro(): step = 0 while True: received = yield step step += 1 print("Received: ", received)
14. Advance the Generator Before sending any value to the generator, this has to be advanced with: next(coroutine) | coroutine.send(None) If not, TypeError is raised.
15. Coroutines via Enhanced Generators <g>.throw(exc_type[, exc_value[, ex_tb]]) Raises the exception at the point where the coroutine is suspended.
16. More Coroutines
18. Delegating to a Sub-Generator • Generators can now return values! • yield from ◦ Gets all values from an iterable object ◦ Produce the values from the sub-generator ◦ Open a channel to the internal generator ◦ Can get the value returned by the internal generator
19. Generators as Coroutines - Return values StopIteration.value contains the result. → Once the return is reached, there is no more iteration. >>> def gen(): ...: yield 1 ...: yield 2 ...: return 42 >>> g = gen() >>> next(g) 1 >>> next(g) 2 >>> next(g) ------------------------------------ StopIteration Traceback (most recent call last) StopIteration: 42
20. yield from • Get all the elements from an iterable • Delegate to a sub-generator
21. yield from - Basic Something in the form... yield from <iterable> Could be thought of as... for e in <iterable>: yield e
22. ~ itertools.chain >>> def chain2(*iterables): ... for it in iterables: ... yield from it >>> list(chain2([1,2,3], (4, 5, 6), "hello")) [1, 2, 3, 4, 5, 6, 'h', 'e', 'l', 'l', 'o']
23. yield from - More Chain generators • .send(), and .throw() are passed along. • Returned (yielded) values, bubble up. yield from acts as a “channel” from the original caller, to the internal generators.
24. yield from - Example def internal(name, limit): for i in range(limit): value = yield i print(f"{name} got: {value}") def general(): yield from internal("first", 10) yield from internal("second", 20)
25. yield from - Recap • Allows delegating to a sub-generator • Enables chaining generators and many iterables together • Makes it easier to refactor generators
26. async def / await
28. yield from / await # py 3.4 @asyncio.coroutine def coroutine(): yield from asyncio.sleep(1) # py 3.5+ async def coroutine(): await asyncio.sleep(1)
29. await Works like yield from, except that: • Does not accept generators that aren’t coroutines • Accepts awaitable objects ◦ __await__()
30. asyncio • An event loop drives the coroutines scheduled to run, and updates them with .send(), next(), .throw(), etc. • The coroutine we write, should only delegate with await (yield from), to some other 3rd party generator, that will do the actual I/O. • yield , yield from, await give the control back to the scheduler.
31. Asynchronous Generators
33. Asynchronous Generators • Before Python 3.6, it was not possible to have a yield in a coroutine :-( ◦ “async def” only allowed “return” or “await” • “Produce elements, one at the time, asynchronously”: ◦ async for x in data_producer: ... ◦ Asynchronous iterables, were required • Difference between iterator (__iter__ / __next__), vs. a generator ◦ But for asynchronous code (__aiter__ / __anext__), vs. async generator
34. Summary • Generators and coroutines are conceptually different, however they share implementation details. • yield from as a construction evolved into await, to allow more powerful coroutines. • Any await chain of calls ends with a yield (at the end, there is a generator).
35. Thank You! Mariano Anaya @rmarianoa