Generators Inside Out - Speaker Deck

Slide 1

Slide 1 text

Generators Inside Out PyCon India 2016 1

Slide 2

Slide 2 text

About Me Anand Chitipothu Software Consultant & Trainer @anandology http://anandology.com/ PyCon India 2016 2

Slide 3

Slide 3 text

What is this talk about? • Iterators • Generators • Coroutines • Async • Async IO PyCon India 2016 3

Slide 4

Slide 4 text

How does iteration work? PyCon India 2016 4

Slide 5

Slide 5 text

Iterating over a list for x in [1, 2, 3, 4]: print(x) ------------------------------------------------------- 1 2 3 4 PyCon India 2016 5

Slide 6

Slide 6 text

Iterating over a string for c in "hello": print(c) ------------------------------------------------------- h e l l o PyCon India 2016 6

Slide 7

Slide 7 text

Iterating over a dictionary for k in {"x": 1, "y": 2, "z": 3}: print(k) ------------------------------------------------------- y x z PyCon India 2016 7

Slide 8

Slide 8 text

The Iteration Protocol >>> x = iter(["a", "b", "c"]) >>> next(x) 'a' >>> next(x) 'b' >>> next(x) 'c' >>> next(x) Traceback (most recent call last): File "", line 1, in StopIteration PyCon India 2016 8

Slide 9

Slide 9 text

win! # Largest word in the dictionary >>> max(open('/usr/share/dict/words'), key=len) 'formaldehydesulphoxylate\n' PyCon India 2016 9

Slide 10

Slide 10 text

Generators PyCon India 2016 10

Slide 11

Slide 11 text

What is a generator? def squares(numbers): for n in numbers: yield n*n ------------------------------------------------------- >>> for x in squares([1, 2, 3]): ... print(x) 1 4 9 PyCon India 2016 11

Slide 12

Slide 12 text

Let me add some prints to understand it better. def squares(numbers): print("BEGIN squares") for n in numbers: print("Computing square of", n) yield n*n print("END squares") ------------------------------------------------------- >>> sq = squares([1, 2, 3]) >>> sq PyCon India 2016 12

Slide 13

Slide 13 text

>>> next(sq) BEGIN squares Computing square of 1 1 >>> next(sq) Computing square of 2 4 >>> next(sq) Computing square of 3 9 >>> next(sq) END squares Traceback (most recent call last): File "", line 1, in StopIteration PyCon India 2016 13

Slide 14

Slide 14 text

>>> for x in squares([1, 2, 3]): ... print(x) BEGIN squares Computing square of 1 1 Computing square of 2 4 Computing square of 3 9 END squares PyCon India 2016 14

Slide 15

Slide 15 text

Example: Fibbonacci Numbers PyCon India 2016 15

Slide 16

Slide 16 text

Write a program to ﬁnd ﬁbbonacci number. def fibn(n): if n == 1 or n == 2: return 1 else: return fibn(n-1) + fibn(n-2) ------------------------------------------------------- >>> fibn(10) 55 PyCon India 2016 16

Slide 17

Slide 17 text

Write a program to compute ﬁrst n ﬁbbonacci numbers. def fibs(n): result = [] a, b = 1, 1 for i in range(n): result.append(a) a, b = b, a+b return result ------------------------------------------------------- >>> fibs(10) [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] PyCon India 2016 17

Slide 18

Slide 18 text

What is the largest ﬁbbonacci number below one million? def largest_fib(upperbound): a, b = 1, 1 while b < upperbound: a, b = b, a+b return a ------------------------------------------------------- >>> largest_fib(1000000) 832040 PyCon India 2016 18

Slide 19

Slide 19 text

Issue Three different implementations to compute ﬁbbonacci numbers! PyCon India 2016 19

Slide 20

Slide 20 text

The generator-based solution def gen_fibs(): """Generates sequence of fibbonacci numbers. """ a, b = 1, 1 while True: yield a a, b = b, a + b PyCon India 2016 20

Slide 21

Slide 21 text

Let's write some generic generator utilities. def first(seq): """Returns the first element of a sequence. """ return next(iter(seq)) def last(seq): """Returns the last element of a sequence. """ for x in seq: pass return x PyCon India 2016 21

Slide 22

Slide 22 text

def take(n, seq): """Takes first n elements of a sequence. """ seq = iter(seq) return (next(seq) for i in range(n)) def nth(n, seq): """Returns n'th element of a sequence. """ return last(take(n, seq)) PyCon India 2016 22

Slide 23

Slide 23 text

def upto(upperbound, seq): """Returns elements in the sequence until they are less than upper bound. """ for x in seq: if x > upperbound: break yield x def count(seq): """Counts the number of elements in a sequence.""" return sum(1 for x in seq) PyCon India 2016 23

Slide 24

Slide 24 text

# what is 10th fibbonacci number? >>> nth(10, gen_fibs()) 55 # find first 10 fibbinacci numbers >>> list(take(10, gen_fibs())) [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] # find all fibbonacci numbers below 100 >>> list(upto(100, gen_fibs()) [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] PyCon India 2016 24

Slide 25

Slide 25 text

# What is the largest fibbonacci number # below one million? >>> last(upto(1000000, gen_fibs())) 832040 # How many fibbonacci numbes are there # below one million? >>> count(upto(1000000, gen_fibs())) 30 PyCon India 2016 25

Slide 26

Slide 26 text

Building data pipelines PyCon India 2016 26

Slide 27

Slide 27 text

import os def find(root): """Finds all the files in the given directory tree. """ for path, dirnames, filenames in os.walk(root): for f in filenames: yield os.path.join(path, f) PyCon India 2016 27

Slide 28

Slide 28 text

def readlines(paths): """Returns a generator over lines in all the files specified. """ for path in paths: yield from open(path) PyCon India 2016 28

Slide 29

Slide 29 text

def grep(pattern, lines): """Returns only the lines that contain given pattern. """ return (line for line in lines if pattern in line) PyCon India 2016 29

Slide 30

Slide 30 text

def main(): # find all files in the project filenames = find("project") # pick only python files filenames = grep('.py', filenames) # read all the lines lines = readlines(filenames) # pick only function definitions lines = grep('def ', lines) # count the total number of functions in your project print(count(lines)) PyCon India 2016 30

Slide 31

Slide 31 text

Coroutines PyCon India 2016 31

Slide 32

Slide 32 text

Let's look at this strange example: def display(values): for v in values: print(v) yield def main(): g1 = display("ABC") g2 = display("123") next(g1); next(g2) next(g1); next(g2) next(g1); next(g2) PyCon India 2016 32

Slide 33

Slide 33 text

>>> main() A 1 B 2 C 3 PyCon India 2016 33

Slide 34

Slide 34 text

Slightly generalized. def run_all(generators): # Runs all the generators concurrently # stop when any one of them stops try: while True: for g in generators: next(g) except StopIteration: pass def main2(): g1 = display("ABC") g2 = display("123") run_all([g1, g2]) PyCon India 2016 34

Slide 35

Slide 35 text

>>> main2() A 1 B 2 C 3 PyCon India 2016 35

Slide 36

Slide 36 text

How about writing a function to print two sets of values? def display2(values1, values2): # WARNING: this doesn't work display(values1) display(values2) PyCon India 2016 36

Slide 37

Slide 37 text

def display2(values1, values2): yield from display(values1) yield from display(values2) def main3(): g1 = display2("ABC", "XYZ") g2 = display2("...", "...") run_all([g1, g2]) PyCon India 2016 37

Slide 38

Slide 38 text

>>> main3() A . B . C . X . Y . Z . PyCon India 2016 38

Slide 39

Slide 39 text

Let's try to build a simple concurrency library based on coroutines. from collections import deque _tasks = deque() def run(task): _tasks.append(task) run_all() def spawn(task): _tasks.appendleft(task) yield PyCon India 2016 39

Slide 40

Slide 40 text

def run_all(): while _tasks: task = _tasks.popleft() try: next(task) except StopIteration: pass else: _tasks.append(task) PyCon India 2016 40

Slide 41

Slide 41 text

Returning a value from a generator def square(x): """Computes square of a number using square microservice. """ response = send_request("/api/square", number=x) # Let something else run while square is being computed. yield return response.json()['result'] PyCon India 2016 41

Slide 42

Slide 42 text

def sum_of_squares(x, y): x2 = yield from square(x) y2 = yield from square(y) return x2 + y2 PyCon India 2016 42

Slide 43

Slide 43 text

Generators are overloaded • Used to build and process data streams • Also used as coroutines Confusing! PyCon India 2016 43

Slide 44

Slide 44 text

Native Coroutines PyCon India 2016 44

Slide 45

Slide 45 text

async def square(x): return x*x async def sum_of_squares(x, y): x2 = await square(x) y2 = await square(y) return x2+y2 PyCon India 2016 45

Slide 46

Slide 46 text

Coroutine Protocol >>> square(4) >>> x = square(4) >>> x.send(None) Traceback (most recent call last): File "", line 1, in StopIteration: 16 PyCon India 2016 46

Slide 47

Slide 47 text

Running Coroutines def run(coroutine): try: while True: coroutine.send(None) except StopIteration as e: return e.value ------------------------------------------------------- >>> run(square(4)) 16 PyCon India 2016 47

Slide 48

Slide 48 text

Generator-based coroutines import types @types.coroutine def aprint(x): print(x) yield PyCon India 2016 48

Slide 49

Slide 49 text

async def display(values): for v in values: await aprint(v) ------------------------------------------------------- >>> run(display("ABC")) A B C PyCon India 2016 49

Slide 50

Slide 50 text

Coroutine Library PyCon India 2016 50

Slide 51

Slide 51 text

"""coro.py - a simple coroutine conncurrency library. """ import types from collections import deque _tasks = deque() def run(task): _tasks.append(task) run_all() @types.coroutine def spawn(task): _tasks.appendleft(task) yield PyCon India 2016 51

Slide 52

Slide 52 text

def run_all(): while _tasks: task = _tasks.popleft() try: task.send(None) except StopIteration: pass else: _tasks.append(task) PyCon India 2016 52

Slide 53

Slide 53 text

Async Example from coro import spawn, run import types @types.coroutine def aprint(x): print(x) yield PyCon India 2016 53

Slide 54

Slide 54 text

async def display(values): for v in values: await aprint(v) async def main(): await spawn(display("ABC")) await spawn(display("123")) if __name__ == "__main__": run(main()) PyCon India 2016 54

Slide 55

Slide 55 text

Output: A 1 B 2 C 3 PyCon India 2016 55

Slide 56

Slide 56 text

Async IO? PyCon India 2016 56

Slide 57

Slide 57 text

"""asocket - simple async socket implementation. """ from socket import * import types import select # Rename the original socket as _socket as # we are going to write a new socket class _socket = socket PyCon India 2016 57

Slide 58

Slide 58 text

class socket: """Simple async socket. """ def __init__(self, *args): self._sock = _socket(*args) self._sock.setblocking(0) def __getattr__(self, name): return getattr(self._sock, name) PyCon India 2016 58

Slide 59

Slide 59 text

def connect(self, addr): try: self._sock.connect(addr) except BlockingIOError: pass async def send(self, data): await wait_for_write(self._sock) return self._sock.send(data) async def recv(self, size): await wait_for_read(self._sock) return self._sock.recv(size) PyCon India 2016 59

Slide 60

Slide 60 text

@types.coroutine def wait_for_read(sock): while True: r, w, e = select.select([sock], [], [], 0) if r: break yield @types.coroutine def wait_for_write(sock): while True: r, w, e = select.select([], [sock], [], 0) if w: break yield PyCon India 2016 60

Slide 61

Slide 61 text

Async IO Example from asocket import * from coro import spawn, run async def echo_client(host, port, label): sock = socket(AF_INET, SOCK_STREAM) sock.connect((host, port)) for i in range(3): await sock.send(str(i).encode('ascii')) data = await sock.recv(1024) print(label, data.decode('ascii')) PyCon India 2016 61

Slide 62

Slide 62 text

async def main(): host, port = 'localhost', 1234 await spawn(echo_client(host, port, 'A')) await spawn(echo_client(host, port, 'B')) await spawn(echo_client(host, port, 'C')) await spawn(echo_client(host, port, 'D')) if __name__ == "__main__": run(main()) PyCon India 2016 62

Slide 63

Slide 63 text

$ python echo_client.py B 0 A 0 D 0 C 0 B 1 B 2 A 1 D 1 C 1 A 2 D 2 C 2 PyCon India 2016 63

Slide 64

Slide 64 text

Summary Generators are awesome. Use them for everything! Coroutines are good vehicles for concurrency. Explore them more! PyCon India 2016 64

Slide 65

Slide 65 text

Questions? Anand Chitipothu @anandology http://bit.ly/pygen0 PyCon India 2016 65