Demystifying coroutines and asynchronous programming in Pyhon

Demystifying coroutines and asynchronous programming in Pyhon

Exploring the internals of how coroutines and asynchronous programming work in Python, from the ground up.

Presented at Python San Sebastián 2018, on October 13th - https://pyss18.pyss.org/

Bfd20328a551defaa84acc3c205da999?s=128

Mariano Anaya

October 13, 2018
Tweet

Transcript

  1. Demystifying Coroutines and Asynchronous Programming in Python Mariano Anaya @rmarianoa

  2. History • PEP-255: Simple generators • PEP-342: Coroutines via enhanced

    generators • PEP-380: Syntax for delegating to a sub-generator • PEP-492: Coroutines with async and await syntax • PEP-525: Asynchronous generators
  3. Generators

  4. None
  5. Generate elements, one at the time, and suspend... • Save

    memory • Support iteration pattern, infinite sequences, etc.
  6. Simple Generators - Example 1 LIMIT = 1_000_000 def old_range(n):

    numbers = [] i = 0 while i < n: numbers.append(i) i += 1 return numbers def new_range(n): i = 0 while i < n: yield i i += 1 Don’t Do
  7. Simple Generators - Example 2 def new_range(n): i = 0

    while i < n: yield i i += 1 total = sum(new_range(LIMIT)) total = 0 i = 0 while i < LIMIT: total += i i += 1 Don’t Do
  8. Simple Generators • next() will advance the generator until the

    next yield statement. ◦ Produces a value to the caller, and suspends there. ◦ If there aren’t more elements, StopIteration is raised
  9. Coroutines

  10. None
  11. Generators as coroutines • Suspend? • How about sending (receiving)

    data to (from) a generator? • And exceptions? <g>.send(<value>) <g>.throw(<exception>) <g>.close()
  12. • Coroutines are syntactically like generators • With .send(), the

    caller pushes data into the coroutine. ◦ yield usually appears on the RHS value = yield result • The coroutine is suspended at the yield Coroutines via Enhanced Generators
  13. >>> c = coro() >>> next(c) >>> step = c.send(received)

    def coro(): step = 0 while True: received = yield step step += 1 print("Received: ", received)
  14. Advance the Generator Before sending any value to the generator,

    this has to be advanced with: next(coroutine) | coroutine.send(None) If not, TypeError is raised.
  15. Coroutines via Enhanced Generators <g>.throw(exc_type[, exc_value[, ex_tb]]) Raises the exception

    at the point where the coroutine is suspended.
  16. More Coroutines

  17. None
  18. Delegating to a Sub-Generator • Generators can now return values!

    • yield from ◦ Gets all values from an iterable object ◦ Produce the values from the sub-generator ◦ Open a channel to the internal generator ◦ Can get the value returned by the internal generator
  19. Generators as Coroutines - Return values StopIteration.value contains the result.

    → Once the return is reached, there is no more iteration. >>> def gen(): ...: yield 1 ...: yield 2 ...: return 42 >>> g = gen() >>> next(g) 1 >>> next(g) 2 >>> next(g) ------------------------------------ StopIteration Traceback (most recent call last) StopIteration: 42
  20. yield from • Get all the elements from an iterable

    • Delegate to a sub-generator
  21. yield from - Basic Something in the form... yield from

    <iterable> Could be thought of as... for e in <iterable>: yield e
  22. ~ itertools.chain >>> def chain2(*iterables): ... for it in iterables:

    ... yield from it >>> list(chain2([1,2,3], (4, 5, 6), "hello")) [1, 2, 3, 4, 5, 6, 'h', 'e', 'l', 'l', 'o']
  23. yield from - More Chain generators • .send(), and .throw()

    are passed along. • Returned (yielded) values, bubble up. yield from acts as a “channel” from the original caller, to the internal generators.
  24. yield from - Example def internal(name, limit): for i in

    range(limit): value = yield i print(f"{name} got: {value}") def general(): yield from internal("first", 10) yield from internal("second", 20)
  25. yield from - Recap • Allows delegating to a sub-generator

    • Enables chaining generators and many iterables together • Makes it easier to refactor generators
  26. async def / await

  27. None
  28. yield from / await # py 3.4 @asyncio.coroutine def coroutine():

    yield from asyncio.sleep(1) # py 3.5+ async def coroutine(): await asyncio.sleep(1)
  29. await Works like yield from, except that: • Does not

    accept generators that aren’t coroutines • Accepts awaitable objects ◦ __await__()
  30. asyncio • An event loop drives the coroutines scheduled to

    run, and updates them with .send(), next(), .throw(), etc. • The coroutine we write, should only delegate with await (yield from), to some other 3rd party generator, that will do the actual I/O. • yield , yield from, await give the control back to the scheduler.
  31. Asynchronous Generators

  32. None
  33. Asynchronous Generators • Before Python 3.6, it was not possible

    to have a yield in a coroutine :-( ◦ “async def” only allowed “return” or “await” • “Produce elements, one at the time, asynchronously”: ◦ async for x in data_producer: ... ◦ Asynchronous iterables, were required • Difference between iterator (__iter__ / __next__), vs. a generator ◦ But for asynchronous code (__aiter__ / __anext__), vs. async generator
  34. Summary • Generators and coroutines are conceptually different, however they

    share implementation details. • yield from as a construction evolved into await, to allow more powerful coroutines. • Any await chain of calls ends with a yield (at the end, there is a generator).
  35. Thank You! Mariano Anaya @rmarianoa