Exploring Generators & Coroutines

Transcript

1. Exploring Generators & Coroutines Mariano Anaya @rmarianoa
2. $ history • ~2001: PEP-255 - “Simple Generators” [Py 2.2] • ~2005: PEP-342 - “Coroutines via Enhanced Generators” [Py 2.5] • ~2009: PEP-380 - “Syntax for Delegating to a Subgenerator” [Py 3.3] • ~2016: PEP-525 - “Asynchronous Generators” [Py 3.6]
3. Simple Generators: Basic Idea Generate elements, one at the time → lazy computation • Save memory • Support iteration pattern, infinite sequences, etc.
4. Simple Generators • The yield statement, makes the function a generator. • .next() will advance until the next yield statement is reached. • yield produces a value to the caller, and suspends the execution.
5. Simple Generators: Example 1 “Sum all numbers up to N”. LIMIT = 1_000_000 def old_range(n): numbers = [] i = 0 while i < n: numbers.append(i) i += 1 return numbers
6. Simple Generators: Example 2 def new_range(n): i = 0 while i < n: yield i i += 1
7. Simple Generators: iteration • Given a generator g = generator_function() ◦ next(g), will advance to the next yield statement. ◦ If there are no more elements, StopIteration is raised • The idiom “for x in g:...” follows this protocol ◦ Calls iter(g) → __iter__, __next__
8. Iteration >>> g = generator() >>> next(g) 1 >>> next(g) 2 >>> next(g) Traceback (most recent call last): File "<stdin>", line 1, in ? StopIteration >>>
9. next() & StopIteration >>> g = gen() >>> next(g) Traceback (most recent call last): File "<stdin>", line 1, in ? StopIteration >>> next(g, "default value") 'default value'
10. Use Generators def new_range(n): i = 0 while i < n: yield i i += 1 total = sum(new_range(LIMIT)) total = 0 i = 0 while i < LIMIT: total += i i += 1 Alternative Pythonic way A generator gives a proper abstraction, with all the advantages of iterators for free (combine with itertools, chain, assign, pass along, etc.).
11. Coroutines
12. Coroutines via Enhanced Generators • How about sending (receiving) data to (from) a generator? • And exceptions? <g>.send(<value>) <g>.throw(<exception>) <g>.close()
13. • Coroutines are syntactically like generators • With .send(), the caller pushes data into the coroutine. ◦ yield usually appears on the RHS value = yield result • The coroutine is suspended at the yield Coroutines via Enhanced Generators
14. >>> c = coro() >>> next(c) >>> step = c.send(received) Coroutines via Enhanced Generators def coro(): step = 0 while True: received = yield step step += 1 print("Received: ", received)
15. Advance the Generator Before sending any value to the generator, this has to be advanced with: next(coroutine) | coroutine.send(None) If not, TypeError is raised.
16. Coroutines via Enhanced Generators generator.throw(exc_type[, exc_value[, ex_tb]]) • Raises the exception at the point where the coroutine is suspended. • Equivalent to: ◦ raise exc_type, exc_value, ex_tb
17. More Coroutines
18. Delegating to a Sub-Generator • Generators can now return values! • yield from ◦ Gets all values from an iterable object ◦ Produce the values from the sub-generator ◦ Open a channel to the internal generator ◦ Can get the value returned by the internal generator
19. Generators as Coroutines: return values StopIteration.value contains the result. → Once the return is reached, there is no more iteration. >>> def gen(): ...: yield 1 ...: yield 2 ...: return 42 ...: >>> g = gen() >>> next(g) 1 >>> next(g) 2 >>> next(g) ------------------------------------ StopIteration Traceback (most recent call last) StopIteration: 42
20. yield from • Get all the elements from an iterable • Delegate to a sub-generator
21. yield from: extract values Basic usage: yield from x Could be thought of as: for e in x: yield e
22. yield from: Basics Similar to itertools.chain: >>> def chain2(*iterables): ... for it in iterables: ... yield from it >>> list(chain2([1,2,3], (4, 5, 6), "hello")) [1, 2, 3, 4, 5, 6, 'h', 'e', 'l', 'l', 'o']
23. yield from: More Communicate with the internal generators • .send(), and .throw() are passed along. • Returned (yielded) values, bubble up. yield from acts as a “channel” from the original caller, to the internal generators.
24. Example: yield from def internal(name, limit): for i in range(limit): value = yield i print(f"{name} got: {value}") def general(): yield from internal("first", 10) yield from internal("second", 20)
25. yield from: recap • Allows delegating to a sub-generator • Enables chaining generators and many iterables together • Makes it easier to refactor generators
26. async def / await
27. asyncio • On asyncio, the event loop drives the coroutines scheduled to run, and updates them with .send(), next(), .throw(), etc. • The coroutine we write, should only delegate with await (yield from), to some other 3rd party generator, that will do the actual I/O. • yield , yield from, await give the control back to the scheduler.
28. yield from & await # py 3.4 @asyncio.coroutine def coroutine(): yield from asyncio.sleep(1) # py 3.5+ async def coroutine(): await asyncio.sleep(1)
29. await Works like yield from, except that: • Does not accept generators that aren’t coroutines • Accepts awaitable objects: __await__()
30. yield from vs. await Coroutine Generator Awaitable yield from Yes Yes No await Yes No Yes
32. Asynchronous Generators • Before Python 3.6, it was not possible to have a yield in a coroutine :-( ◦ “async def” only allowed “return” or “await” • “Produce elements, one at the time, asynchronously”: ◦ async for x in data_producer: ... ◦ Asynchronous iterables, were required • Difference between iterator (__iter__ / __next__), vs. a generator ◦ But for asynchronous code (__aiter__ / __anext__), vs. async generator
33. Summary • Thinking in terms of iterables, allows to build better patterns. • Generators and coroutines are conceptually different, however their implementation details are similar. • yield from is a construction that allows more powerful coroutines. • Any yield from chain of calls ends with a yield (at the end, there is a generator).
34. Thank You!