Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Generators Inside Out

Generators Inside Out

Generators Inside Out talk given at PyCon India 2016.

7e03bd91a64c190bd4e54bd87d1ab0b2?s=128

Anand Chitipothu

September 24, 2016
Tweet

Transcript

  1. Generators Inside Out PyCon India 2016 1

  2. About Me Anand Chitipothu Software Consultant & Trainer @anandology http://anandology.com/

    PyCon India 2016 2
  3. What is this talk about? • Iterators • Generators •

    Coroutines • Async • Async IO PyCon India 2016 3
  4. How does iteration work? PyCon India 2016 4

  5. Iterating over a list for x in [1, 2, 3,

    4]: print(x) ------------------------------------------------------- 1 2 3 4 PyCon India 2016 5
  6. Iterating over a string for c in "hello": print(c) -------------------------------------------------------

    h e l l o PyCon India 2016 6
  7. Iterating over a dictionary for k in {"x": 1, "y":

    2, "z": 3}: print(k) ------------------------------------------------------- y x z PyCon India 2016 7
  8. The Iteration Protocol >>> x = iter(["a", "b", "c"]) >>>

    next(x) 'a' >>> next(x) 'b' >>> next(x) 'c' >>> next(x) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration PyCon India 2016 8
  9. win! # Largest word in the dictionary >>> max(open('/usr/share/dict/words'), key=len)

    'formaldehydesulphoxylate\n' PyCon India 2016 9
  10. Generators PyCon India 2016 10

  11. What is a generator? def squares(numbers): for n in numbers:

    yield n*n ------------------------------------------------------- >>> for x in squares([1, 2, 3]): ... print(x) 1 4 9 PyCon India 2016 11
  12. Let me add some prints to understand it better. def

    squares(numbers): print("BEGIN squares") for n in numbers: print("Computing square of", n) yield n*n print("END squares") ------------------------------------------------------- >>> sq = squares([1, 2, 3]) >>> sq <generator object squares at 0xb6c73720> PyCon India 2016 12
  13. >>> next(sq) BEGIN squares Computing square of 1 1 >>>

    next(sq) Computing square of 2 4 >>> next(sq) Computing square of 3 9 >>> next(sq) END squares Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration PyCon India 2016 13
  14. >>> for x in squares([1, 2, 3]): ... print(x) BEGIN

    squares Computing square of 1 1 Computing square of 2 4 Computing square of 3 9 END squares PyCon India 2016 14
  15. Example: Fibbonacci Numbers PyCon India 2016 15

  16. Write a program to find fibbonacci number. def fibn(n): if

    n == 1 or n == 2: return 1 else: return fibn(n-1) + fibn(n-2) ------------------------------------------------------- >>> fibn(10) 55 PyCon India 2016 16
  17. Write a program to compute first n fibbonacci numbers. def

    fibs(n): result = [] a, b = 1, 1 for i in range(n): result.append(a) a, b = b, a+b return result ------------------------------------------------------- >>> fibs(10) [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] PyCon India 2016 17
  18. What is the largest fibbonacci number below one million? def

    largest_fib(upperbound): a, b = 1, 1 while b < upperbound: a, b = b, a+b return a ------------------------------------------------------- >>> largest_fib(1000000) 832040 PyCon India 2016 18
  19. Issue Three different implementations to compute fibbonacci numbers! PyCon India

    2016 19
  20. The generator-based solution def gen_fibs(): """Generates sequence of fibbonacci numbers.

    """ a, b = 1, 1 while True: yield a a, b = b, a + b PyCon India 2016 20
  21. Let's write some generic generator utilities. def first(seq): """Returns the

    first element of a sequence. """ return next(iter(seq)) def last(seq): """Returns the last element of a sequence. """ for x in seq: pass return x PyCon India 2016 21
  22. def take(n, seq): """Takes first n elements of a sequence.

    """ seq = iter(seq) return (next(seq) for i in range(n)) def nth(n, seq): """Returns n'th element of a sequence. """ return last(take(n, seq)) PyCon India 2016 22
  23. def upto(upperbound, seq): """Returns elements in the sequence until they

    are less than upper bound. """ for x in seq: if x > upperbound: break yield x def count(seq): """Counts the number of elements in a sequence.""" return sum(1 for x in seq) PyCon India 2016 23
  24. # what is 10th fibbonacci number? >>> nth(10, gen_fibs()) 55

    # find first 10 fibbinacci numbers >>> list(take(10, gen_fibs())) [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] # find all fibbonacci numbers below 100 >>> list(upto(100, gen_fibs()) [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] PyCon India 2016 24
  25. # What is the largest fibbonacci number # below one

    million? >>> last(upto(1000000, gen_fibs())) 832040 # How many fibbonacci numbes are there # below one million? >>> count(upto(1000000, gen_fibs())) 30 PyCon India 2016 25
  26. Building data pipelines PyCon India 2016 26

  27. import os def find(root): """Finds all the files in the

    given directory tree. """ for path, dirnames, filenames in os.walk(root): for f in filenames: yield os.path.join(path, f) PyCon India 2016 27
  28. def readlines(paths): """Returns a generator over lines in all the

    files specified. """ for path in paths: yield from open(path) PyCon India 2016 28
  29. def grep(pattern, lines): """Returns only the lines that contain given

    pattern. """ return (line for line in lines if pattern in line) PyCon India 2016 29
  30. def main(): # find all files in the project filenames

    = find("project") # pick only python files filenames = grep('.py', filenames) # read all the lines lines = readlines(filenames) # pick only function definitions lines = grep('def ', lines) # count the total number of functions in your project print(count(lines)) PyCon India 2016 30
  31. Coroutines PyCon India 2016 31

  32. Let's look at this strange example: def display(values): for v

    in values: print(v) yield def main(): g1 = display("ABC") g2 = display("123") next(g1); next(g2) next(g1); next(g2) next(g1); next(g2) PyCon India 2016 32
  33. >>> main() A 1 B 2 C 3 PyCon India

    2016 33
  34. Slightly generalized. def run_all(generators): # Runs all the generators concurrently

    # stop when any one of them stops try: while True: for g in generators: next(g) except StopIteration: pass def main2(): g1 = display("ABC") g2 = display("123") run_all([g1, g2]) PyCon India 2016 34
  35. >>> main2() A 1 B 2 C 3 PyCon India

    2016 35
  36. How about writing a function to print two sets of

    values? def display2(values1, values2): # WARNING: this doesn't work display(values1) display(values2) PyCon India 2016 36
  37. def display2(values1, values2): yield from display(values1) yield from display(values2) def

    main3(): g1 = display2("ABC", "XYZ") g2 = display2("...", "...") run_all([g1, g2]) PyCon India 2016 37
  38. >>> main3() A . B . C . X .

    Y . Z . PyCon India 2016 38
  39. Let's try to build a simple concurrency library based on

    coroutines. from collections import deque _tasks = deque() def run(task): _tasks.append(task) run_all() def spawn(task): _tasks.appendleft(task) yield PyCon India 2016 39
  40. def run_all(): while _tasks: task = _tasks.popleft() try: next(task) except

    StopIteration: pass else: _tasks.append(task) PyCon India 2016 40
  41. Returning a value from a generator def square(x): """Computes square

    of a number using square microservice. """ response = send_request("/api/square", number=x) # Let something else run while square is being computed. yield return response.json()['result'] PyCon India 2016 41
  42. def sum_of_squares(x, y): x2 = yield from square(x) y2 =

    yield from square(y) return x2 + y2 PyCon India 2016 42
  43. Generators are overloaded • Used to build and process data

    streams • Also used as coroutines Confusing! PyCon India 2016 43
  44. Native Coroutines PyCon India 2016 44

  45. async def square(x): return x*x async def sum_of_squares(x, y): x2

    = await square(x) y2 = await square(y) return x2+y2 PyCon India 2016 45
  46. Coroutine Protocol >>> square(4) <coroutine object square at 0xb57a6510> >>>

    x = square(4) >>> x.send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 16 PyCon India 2016 46
  47. Running Coroutines def run(coroutine): try: while True: coroutine.send(None) except StopIteration

    as e: return e.value ------------------------------------------------------- >>> run(square(4)) 16 PyCon India 2016 47
  48. Generator-based coroutines import types @types.coroutine def aprint(x): print(x) yield PyCon

    India 2016 48
  49. async def display(values): for v in values: await aprint(v) -------------------------------------------------------

    >>> run(display("ABC")) A B C PyCon India 2016 49
  50. Coroutine Library PyCon India 2016 50

  51. """coro.py - a simple coroutine conncurrency library. """ import types

    from collections import deque _tasks = deque() def run(task): _tasks.append(task) run_all() @types.coroutine def spawn(task): _tasks.appendleft(task) yield PyCon India 2016 51
  52. def run_all(): while _tasks: task = _tasks.popleft() try: task.send(None) except

    StopIteration: pass else: _tasks.append(task) PyCon India 2016 52
  53. Async Example from coro import spawn, run import types @types.coroutine

    def aprint(x): print(x) yield PyCon India 2016 53
  54. async def display(values): for v in values: await aprint(v) async

    def main(): await spawn(display("ABC")) await spawn(display("123")) if __name__ == "__main__": run(main()) PyCon India 2016 54
  55. Output: A 1 B 2 C 3 PyCon India 2016

    55
  56. Async IO? PyCon India 2016 56

  57. """asocket - simple async socket implementation. """ from socket import

    * import types import select # Rename the original socket as _socket as # we are going to write a new socket class _socket = socket PyCon India 2016 57
  58. class socket: """Simple async socket. """ def __init__(self, *args): self._sock

    = _socket(*args) self._sock.setblocking(0) def __getattr__(self, name): return getattr(self._sock, name) PyCon India 2016 58
  59. def connect(self, addr): try: self._sock.connect(addr) except BlockingIOError: pass async def

    send(self, data): await wait_for_write(self._sock) return self._sock.send(data) async def recv(self, size): await wait_for_read(self._sock) return self._sock.recv(size) PyCon India 2016 59
  60. @types.coroutine def wait_for_read(sock): while True: r, w, e = select.select([sock],

    [], [], 0) if r: break yield @types.coroutine def wait_for_write(sock): while True: r, w, e = select.select([], [sock], [], 0) if w: break yield PyCon India 2016 60
  61. Async IO Example from asocket import * from coro import

    spawn, run async def echo_client(host, port, label): sock = socket(AF_INET, SOCK_STREAM) sock.connect((host, port)) for i in range(3): await sock.send(str(i).encode('ascii')) data = await sock.recv(1024) print(label, data.decode('ascii')) PyCon India 2016 61
  62. async def main(): host, port = 'localhost', 1234 await spawn(echo_client(host,

    port, 'A')) await spawn(echo_client(host, port, 'B')) await spawn(echo_client(host, port, 'C')) await spawn(echo_client(host, port, 'D')) if __name__ == "__main__": run(main()) PyCon India 2016 62
  63. $ python echo_client.py B 0 A 0 D 0 C

    0 B 1 B 2 A 1 D 1 C 1 A 2 D 2 C 2 PyCon India 2016 63
  64. Summary Generators are awesome. Use them for everything! Coroutines are

    good vehicles for concurrency. Explore them more! PyCon India 2016 64
  65. Questions? Anand Chitipothu @anandology http://bit.ly/pygen0 PyCon India 2016 65