A Functional Programming Approach To Data Processing In Python

Slide 1

Slide 1 text

A FUNCTIONAL PROGRAMMING APPROACH TO LambdaConf — Boulder, Colorado — May 26, 2017 DATA PROCESSING IN PYTHON By Reuben Cummings

Slide 2

Slide 2 text

Reuben Cummings λ @reubano λ #LambdaConf Who am I? Managing Director, Nerevu Development Founder, Arusha Coders Author of several popular Python packages

Slide 3

Slide 3 text

WHAT IS DATA? I dare you, I double dare you! Image Credit: www.emaze.com SAY BIG DATA ONE MORE TIME

Slide 4

Slide 4 text

Reuben Cummings λ @reubano λ #LambdaConf Organization language presenter mercury alex scala gleb haskell michael "This session seeks to entertain and teach the developer who is already..." structured unstructured

Slide 5

Slide 5 text

Reuben Cummings λ @reubano λ #LambdaConf Storage type,duration leap,360 hop,120 de novo,60 inspire,10 00103e0 b0e6 04... 00105f0 e4e7 03... 0010600 0be8 04... 00105b0 c4e4 02... 00106e0 b0e9 04... ﬂat/text binary

Slide 6

Slide 6 text

Reuben Cummings λ @reubano λ #LambdaConf Organization vs Storage ﬂat/text binary structured unstructured

Slide 7

Slide 7 text

What is it data processing good for?

Slide 8

Slide 8 text

Spotify's Discovery Weekly playlist of new songs you like adapts to user's shifting musical tastes handles outliers and seasonality Image Credit: www.spotify.com/int/discoverweekly/

Slide 9

Slide 9 text

What is functional programming good for?

Slide 10

Slide 10 text

Om rapid UI re-renders serializable application state time travel/undo Image Credit: circleci.com

Slide 11

Slide 11 text

A BRIEF INTRO TO PYTHON Ooouuu, stickers! Image Credit: www.pythongear.com

Slide 12

Slide 12 text

Reuben Cummings λ @reubano λ #LambdaConf Presentation: GitHub repo github.com/reubano/lambdaconf- tutorial

Slide 13

Slide 13 text

Reuben Cummings λ @reubano λ #LambdaConf Presentation: Jupyter Notebook beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (presentation.ipybn)

Slide 14

Slide 14 text

Naive Image Credit: (Alika Seu) www.ﬂickr.com

Slide 15

Slide 15 text

Reading data

Slide 16

Slide 16 text

Reuben Cummings λ @reubano λ #LambdaConf Reading data (naive) from urllib.request import urlopen from json import loads BASE = 'https://api.github.com/search' _url1 = '{}/repositories?q={}' q = 'data&per_page=100' url1 = _url1.format(BASE, q) f = urlopen(url1)

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

GitHub API Image Credit: https://api.github.com/search/repositories?q=data

Slide 20

Slide 20 text

Reuben Cummings λ @reubano λ #LambdaConf >>> data = loads(f.read().decode('utf-8')) Reading data (naive)

Slide 21

Slide 21 text

Reuben Cummings λ @reubano λ #LambdaConf >>> repos = data['items'] >>> repos[0]['description'] 'Jargon from the functional programming world in simple terms!' >>> repos[0]['full_name'] 'hemanth/functional-programming-jargon' >>> data = loads(f.read().decode('utf-8')) Reading data (naive)

Slide 22

Slide 22 text

Processing data

Slide 23

Slide 23 text

Reuben Cummings λ @reubano λ #LambdaConf Processing data (naive) def rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated

Slide 24

Slide 24 text

Reuben Cummings λ @reubano λ #LambdaConf Processing data (naive) def rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated

Slide 25

Slide 25 text

Reuben Cummings λ @reubano λ #LambdaConf Processing data (naive) def rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated

Slide 26

Slide 26 text

Reuben Cummings λ @reubano λ #LambdaConf Processing data (naive) def rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated

Slide 27

Slide 27 text

Reuben Cummings λ @reubano λ #LambdaConf Processing data (naive) >>> rate(repos)[:5] [36520, 30174, 28576, 26842, 24092]

Slide 28

Slide 28 text

Reuben Cummings λ @reubano λ #LambdaConf >>> from itertools import count >>> >>> inf_repos = ( ... {'watchers': c} for c in count()) Processing infinite data (naive)

Slide 29

Slide 29 text

Reuben Cummings λ @reubano λ #LambdaConf >>> from itertools import count >>> >>> inf_repos = ( ... {'watchers': c} for c in count()) >>> >>> rate(inf_repos) Processing infinite data (naive)

Slide 30

Slide 30 text

Reuben Cummings λ @reubano λ #LambdaConf KeyboardInterrupt Traceback (most recent call last) in () >>> from itertools import count >>> >>> inf_repos = ( ... {'watchers': c} for c in count()) >>> >>> rate(inf_repos) Processing infinite data (naive)

Slide 31

Slide 31 text

Reuben Cummings λ @reubano λ #LambdaConf rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated def rate(repos): Processing expensive data (naive)

Slide 32

Slide 32 text

Reuben Cummings λ @reubano λ #LambdaConf def exp_rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated Processing expensive data (naive)

Slide 33

Slide 33 text

Reuben Cummings λ @reubano λ #LambdaConf from time import sleep def exp_rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated Processing expensive data (naive)

Slide 34

Slide 34 text

Reuben Cummings λ @reubano λ #LambdaConf from time import sleep def exp_rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated sleep(5) Processing expensive data (naive)

Slide 35

Slide 35 text

Reuben Cummings λ @reubano λ #LambdaConf >>> exp_rate(repos)[:5] Processing expensive data (naive)

Slide 36

Slide 36 text

Reuben Cummings λ @reubano λ #LambdaConf [36520, 30174, 28576, 26842, 24092] >>> exp_rate(repos)[:5] Processing expensive data (naive)

Slide 37

Slide 37 text

Lazy evaluation Image Credit: (Mark Turnauckas) www.ﬂickr.com

Slide 38

Slide 38 text

Lazy intro

Slide 39

Slide 39 text

Reuben Cummings λ @reubano λ #LambdaConf >>> next(lazy_list) 0 >>> eager_list = list(range(5)) >>> eager_list [0, 1, 2, 3, 4] >>> lazy_list = iter(eager_list) >>> lazy_list Iterators

Slide 40

Slide 40 text

Reuben Cummings λ @reubano λ #LambdaConf >>> next(lazy_list) StopIteration Traceback (most recent call last) in () ----> 1 next(lazy_list) Iterators >>> list(lazy_list) [1, 2, 3, 4]

Slide 41

Slide 41 text

Reading data

Slide 42

Slide 42 text

Reuben Cummings λ @reubano λ #LambdaConf $ pip install ijson Reading data (lazy evaluation)

Slide 43

Slide 43 text

Reuben Cummings λ @reubano λ #LambdaConf >>> from ijson import items >>> >>> f = urlopen(url1) >>> repos = items(f, 'items.item') >>> repos >>> repo = next(repos) >>> repo['full_name'] 'hemanth/functional-programming-jargon' Reading data (lazy evaluation)

Slide 44

Slide 44 text

Processing data

Slide 45

Slide 45 text

Reuben Cummings λ @reubano λ #LambdaConf rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated def rate(repos): Processing data (lazy evaluation)

Slide 46

Slide 46 text

Reuben Cummings λ @reubano λ #LambdaConf rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated def gen_rates(repos): Processing data (lazy evaluation)

Slide 47

Slide 47 text

Reuben Cummings λ @reubano λ #LambdaConf def gen_rates(repos): for repo in repos: yield repo['watchers'] * 2 Processing data (lazy evaluation)

Slide 48

Slide 48 text

Reuben Cummings λ @reubano λ #LambdaConf >>> rates = gen_rates(repos) >>> next(rates) 36520 >>> next(rates) 30174 >>> gen_rates(repos) Processing data (lazy evaluation)

Slide 49

Slide 49 text

Reuben Cummings λ @reubano λ #LambdaConf Processing infinite data (lazy evaluation) >>> rates = gen_rates(inf_repos) >>> next(rates) 42220156

Slide 50

Slide 50 text

Reuben Cummings λ @reubano λ #LambdaConf Processing expensive data (lazy evaluation) def gen_exp_rates(repos): for repo in repos: sleep(5) yield repo['watchers'] * 2

Slide 51

Slide 51 text

Reuben Cummings λ @reubano λ #LambdaConf Processing expensive data (lazy evaluation) def gen_exp_rates(repos): for repo in repos: sleep(5) yield repo['watchers'] * 2

Slide 52

Slide 52 text

Reuben Cummings λ @reubano λ #LambdaConf >>> list(result) >>> from itertools import islice >>> >>> rates = gen_exp_rates(repos) >>> result = islice(rates, 5) Processing expensive data (lazy evaluation)

Slide 53

Slide 53 text

Reuben Cummings λ @reubano λ #LambdaConf [36520, 30174, 28576, 26842, 24092] >>> list(result) >>> from itertools import islice >>> >>> rates = gen_exp_rates(repos) >>> result = islice(rates, 5) Processing expensive data (lazy evaluation)

Slide 54

Slide 54 text

Reuben Cummings λ @reubano λ #LambdaConf >>> from itertools import islice >>> >>> rates = gen_exp_rates(repos) >>> result = islice(rates, 5) >>> list(result) [36520, 30174, 28576, 26842, 24092] >>> next(rates) 648 Processing expensive data (lazy evaluation)

Slide 55

Slide 55 text

Grouping data

Slide 56

Slide 56 text

Reuben Cummings λ @reubano λ #LambdaConf Grouping data >>> f = urlopen(url1) >>> repos = items(f, 'items.item') >>> repo = next(repos) >>> repo.keys() dict_keys(['id', 'name', 'full_name', 'owner', 'private', 'html_url', 'description', 'fork', 'url', 'forks_url', 'keys_url', ...])

Slide 57

Slide 57 text

Reuben Cummings λ @reubano λ #LambdaConf Grouping data >>> repo['has_issues'] True

Slide 58

Slide 58 text

Reuben Cummings λ @reubano λ #LambdaConf Grouping data >>> import itertools as it >>> from operator import itemgetter >>> >>> keyfunc = itemgetter('has_issues') >>> sorted_repos = sorted(repos, key=keyfunc) >>> grouped = it.groupby( ... sorted_repos, keyfunc) >>> data = ( ... (k, len(list(g))) for k, g in grouped)

Slide 59

Slide 59 text

Reuben Cummings λ @reubano λ #LambdaConf Grouping data >>> next(data) (False, 3) >>> next(data) (True, 96)

Slide 60

Slide 60 text

Memoization Image Credit: (olho wodzynski) www.ﬂickr.com

Slide 61

Slide 61 text

Processing data

Slide 62

Slide 62 text

Reuben Cummings λ @reubano λ #LambdaConf def gen_exp_rates(repos): for repo in repos: sleep(5) yield repo['watchers'] * 2 Processing expensive data (memoization)

Slide 63

Slide 63 text

Reuben Cummings λ @reubano λ #LambdaConf def calc_rate(watchers): sleep(5) return watchers * 2 def gen_exp_rates(repos): for repo in repos: yield calc_rate(repo['watchers']) Processing expensive data (memoization)

Slide 64

Slide 64 text

Reuben Cummings λ @reubano λ #LambdaConf def _calc_rate(watchers): cacher = lru_cache() calc_rate = cacher(_calc_rate) from functools import lru_cache sleep(5) return watchers * 2 Processing expensive data (memoization)

Slide 65

Slide 65 text

Reuben Cummings λ @reubano λ #LambdaConf @lru_cache() from functools import lru_cache def calc_rate(watchers): sleep(5) return watchers * 2 def gen_exp_rates(repos): for repo in repos: yield calc_rate(repo['watchers']) Processing expensive data (memoization)

Slide 66

Slide 66 text

Reuben Cummings λ @reubano λ #LambdaConf [10, 10, 10, 10, 10] >>> list(result) >>> repos = it.repeat({'watchers': 5}) >>> rates = gen_exp_rates(repos) >>> result = islice(rates, 5) Processing expensive data (memoization)

Slide 67

Slide 67 text

EXERCISE #1 Mount Meru — Arusha, Tanzania Image Credit: Reuben Cummings

Slide 68

Slide 68 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #1: Problem display the total # of watchers per language (ignore repos w/o a language)

Slide 69

Slide 69 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #1: Result C# 32 C++ 63 HTML 349 JavaScript 3881 Jupyter Notebook 5481 PHP 201 Python 37007 R 18

Slide 70

Slide 70 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #1: Data source https://api.github.com/search/ repositories?q=data

Slide 71

Slide 71 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #1: Jupyter Notebook beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (exercises.ipybn)

Slide 72

Slide 72 text

Reuben Cummings λ @reubano λ #LambdaConf from urllib.request import urlopen from itertools import groupby from operator import itemgetter from ijson import items url2 = '{}/repositories?q=data'.format(BASE) f = urlopen(url2) repos = items(f, 'items.item') Exercise #1: Solution

Slide 73

Slide 73 text

Reuben Cummings λ @reubano λ #LambdaConf keyfunc = itemgetter('language') cleaned = filter(keyfunc, repos) records = sorted(cleaned, key=keyfunc) grouped = groupby(records, keyfunc) for key, group in grouped: cnt = sum(g['watchers'] for g in group) print(key, cnt) Exercise #1: Solution

Slide 74

Slide 74 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #1: Solution beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (solutions.ipybn)

Slide 75

Slide 75 text

INTRODUCING MEZA Because you might not need Pandas Image Credit: github.com/reubano/meza

Slide 76

Slide 76 text

Reuben Cummings λ @reubano λ #LambdaConf $ pip install meza Meza demo

Slide 77

Slide 77 text

Reuben Cummings λ @reubano λ #LambdaConf Meza demo: Jupyter Notebook beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (presentation.ipybn)

Slide 78

Slide 78 text

Reading data

Slide 79

Slide 79 text

Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> from urllib.request import urlopen >>> from meza.io import read_json >>> >>> f = urlopen(url2) >>> records = read_json(f, path='items.item') >>> repo = next(records) >>> repo['full_name'] 'emberjs/data'

Slide 80

Slide 80 text

Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> len(list(records)) 29

Slide 81

Slide 81 text

Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> from io import StringIO >>> from meza.io import read_csv >>> >>> f = StringIO( ... 'greeting,location\nhello,world\n') >>> >>> next(read_csv(f)) {'greeting': 'hello', 'location': 'world'}

Slide 82

Slide 82 text

Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> from os import path as p >>> from meza.io import join >>> >>> url3 = '{}&page=2'.format(url2) >>> files = map(urlopen, [url2, url3]) >>> records = join( ... *files, ext='json', path='items.item')

Slide 83

Slide 83 text

Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> repo = next(records) >>> repo['full_name'] 'emberjs/data'

Slide 84

Slide 84 text

Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> repo = next(records) >>> repo['full_name'] 'emberjs/data' >>> repo['language'] 'JavaScript'

Slide 85

Slide 85 text

Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> repo = next(records) >>> repo['full_name'] 'emberjs/data' >>> repo['language'] 'JavaScript' >>> len(list(records)) 59

Slide 86

Slide 86 text

Transforming data

Slide 87

Slide 87 text

Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> from meza.process import merge >>> >>> records = [ ... {'a': 200}, {'b': 300}, {'c': 400}] >>> >>> merge(records) {'a': 200, 'b': 300, 'c': 400}

Slide 88

Slide 88 text

Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> from meza.process import group >>> >>> records = [ ... {'item': 'a', 'amount': 200}, ... {'item': 'a', 'amount': 200}, ... {'item': 'b', 'amount': 400}] >>> >>> grouped = group(records, 'item')

Slide 89

Slide 89 text

Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> key, _group = next(grouped) >>> key 'a' >>> _group [{'amount': 200, 'item': 'a'}, {'amount': 200, 'item': 'a'}]

Slide 90

Slide 90 text

Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> from meza import process as pr >>> >>> f = urlopen(url2) >>> raw = read_json(f, path='items.item') >>> fields = [ ... 'full_name', 'language', 'watchers', ... 'score', 'has_wiki'] >>> >>> cut = pr.cut(raw, fields)

Slide 91

Slide 91 text

Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> cut . at 0x10b0410f8> >>> cut, preview = pr.peek(cut) >>> cut >>> len(preview) 5

Slide 92

Slide 92 text

Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> preview[0] {'full_name': 'substance/data', 'has_wiki': True, 'language': 'JavaScript', 'score': Decimal('72.90926'), 'watchers': 678}

Slide 93

Slide 93 text

Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> filled = pr.fillempty( ... raw, value='', fields=['language']) >>> >>> pivoted = pr.pivot( ... filled, 'score', 'language', ... rows=['has_wiki'], op=min)

Slide 94

Slide 94 text

Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> next(pivoted) {'HTML': Decimal('73.52254'), 'JavaScript': Decimal('53.48755'), 'PHP': Decimal('41.3122'), 'Python': Decimal('42.49319'), 'has_wiki': False}

Slide 95

Slide 95 text

Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> next(pivoted) {'': Decimal('44.83392'), 'C#': Decimal('47.793495'), 'HTML': Decimal('69.20008'), 'JavaScript': Decimal('70.15174'), 'PHP': Decimal('44.251198'), 'Python': Decimal('45.78215'), 'R': Decimal('46.23451'), 'has_wiki': True}

Slide 96

Slide 96 text

Reuben Cummings λ @reubano λ #LambdaConf | full_name | language | score | has_wiki | | --------- | ---------- | ------ | -------- | | 'aptnote…' | '' | 76.11… | True | | 'GSA/dat…' | 'HTML' | 73.52… | False | | 'substan…' | 'JavaScr…' | 72.83… | True | | 'GoogleT…' | 'JavaScr…' | 70.15… | True | | 'curran/…' | 'HTML' | 69.20… | True | Transforming data (before)

Slide 97

Slide 97 text

Reuben Cummings λ @reubano λ #LambdaConf | has_wiki | '' | HTML | JavaScript | | -------- | -------- | -------- | ---------- | | False | | 73.52254 | | | True | 76.11933 | 69.20008 | 70.15174 | Transforming data (after)

Slide 98

Slide 98 text

EXERCISE #2 Image Credit: Reuben Cummings Mount Kilimanjaro — Kilimanjaro Region, Tanzania

Slide 99

Slide 99 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Problem display the language with the most # of watchers per owner_type per has_pages

Slide 100

Slide 100 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Result (partial) {'has_pages': True, 'language': 'JavaScript', 'owner_type': 'Organization', 'watchers': 128605}

Slide 101

Slide 101 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Data source https://api.github.com/search/ repositories? q=data&sort=stars&order=desc

Slide 102

Slide 102 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Hint from meza.fntools import flatten # and one of the following from meza.process import normalize # this from meza.process import aggregate # or this

Slide 103

Slide 103 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Jupyter Notebook beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (exercises.ipybn)

Slide 104

Slide 104 text

Reuben Cummings λ @reubano λ #LambdaConf from urllib.request import urlopen from operator import itemgetter from functools import partial from meza import process as pr, fntools as ft from meza.io import read_json q = 'data&sort=stars&order=desc' url4 = '{}/repositories?q={}'.format(BASE, q) f = urlopen(url4) Exercise #2: Solution

Slide 105

Slide 105 text

Reuben Cummings λ @reubano λ #LambdaConf records = read_json(f, path='items.item') filled = pr.fillempty( records, value='', fields=['language']) flat = (dict(ft.flatten(r)) for r in filled) args = ('watchers', 'language') rows = ['has_pages', 'owner_type'] Exercise #2: Solution

Slide 106

Slide 106 text

Reuben Cummings λ @reubano λ #LambdaConf spun = pr.pivot( flat, *args, rows=rows, op=sum) spun, preview = pr.peek(spun) Exercise #2: Solution

Slide 107

Slide 107 text

Reuben Cummings λ @reubano λ #LambdaConf >>> preview[0] {'C#': 7675, 'C++': 55602, 'Go': 13223, 'Objective-C': 10556, … 'has_pages': False, 'owner_type': 'Organization'} Exercise #2: Solution

Slide 108

Slide 108 text

Reuben Cummings λ @reubano λ #LambdaConf >>> kw = {'rows': rows, 'invert': True} >>> normal = pr.normalize(spun, *args, **kw) >>> normal, preview = pr.peek(normal) >>> preview[0] {'has_pages': False, 'language': 'Objective-C', 'owner_type': 'Organization', 'watchers': 10556} Exercise #2: Solution

Slide 109

Slide 109 text

Reuben Cummings λ @reubano λ #LambdaConf akeyfunc = itemgetter('watchers') gkeyfunc = lambda x: tuple(x[r] for r in rows) aggregator = partial(max, key=akeyfunc) kwargs = { 'tupled': False, 'aggregator': aggregator} grouped = pr.group(normal, gkeyfunc, **kwargs) Exercise #2: Solution

Slide 110

Slide 110 text

Reuben Cummings λ @reubano λ #LambdaConf >>> grouped, preview = pr.peek(grouped) >>> preview[0] {'has_pages': False, 'language': 'C++', 'owner_type': 'Organization', 'watchers': 55602} Exercise #2: Solution

Slide 111

Slide 111 text

Reuben Cummings λ @reubano λ #LambdaConf sgrouped = sorted( grouped, key=akeyfunc, reverse=True) for record in sgrouped: print(record) Exercise #2: Solution

Slide 112

Slide 112 text

Reuben Cummings λ @reubano λ #LambdaConf | language | watchers | owner_ty… | has_pages | | -------- | -------- | --------- | --------- | | 'JavaS…' | 128605 | 'Organi…' | True | | 'C++' | 55602 | 'Organi…' | False | | 'Python' | 54269 | 'User' | False | | 'Jupyte…'| 12046 | 'User' | True | Exercise #2: Result (full)

Slide 113

Slide 113 text

Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Solution beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (solutions.ipybn)

Slide 114

Slide 114 text

Thanks! Reuben Cummings @reubano

Slide 115

Slide 115 text

Extra Slides

Slide 116

Slide 116 text

Reuben Cummings λ @reubano λ #LambdaConf def gen_rates(repos): for repo in repos: yield repo['watchers'] * 2 Processing data (lazy evaluation)

Slide 117

Slide 117 text

Reuben Cummings λ @reubano λ #LambdaConf def gen_rates(repos): return ( r['watchers'] * 2 for r in repos) Processing data (lazy evaluation)

Slide 118

Slide 118 text

Slide 119

Slide 119 text

Reuben Cummings λ @reubano λ #LambdaConf records = read_json(f, path='items.item') filled = pr.fillempty( records, value='', fields=['language']) flat = (dict(ft.flatten(r)) for r in filled) akeyfunc = itemgetter('watchers') Exercise #2: Alt. solution

Slide 120

Slide 120 text

Reuben Cummings λ @reubano λ #LambdaConf rows = [ 'has_pages', 'owner_type', 'language', 'watchers'] def grouper(records, rows, aggregator): kwargs = {'aggregator': aggregator} key = lambda x: tuple(x[r] for r in rows) _grouper = partial(pr.group, tupled=False) return _grouper(records, key, **kwargs) Exercise #2: Alt. solution

Slide 121

Slide 121 text

Reuben Cummings λ @reubano λ #LambdaConf def agg1(records): args = (records, 'watchers', sum) return pr.aggregate(*args) grouped = grouper(flat, rows[:3], agg1) agg2 = partial(max, key=akeyfunc) regrouped = grouper(grouped, rows[:2], agg2) cut = pr.cut(regrouped, rows) Exercise #2: Alt. solution

Slide 122

Slide 122 text

Reuben Cummings λ @reubano λ #LambdaConf >>> cut, preview = pr.peek(cut) >>> preview[0] {'has_pages': False, 'language': 'C++', 'owner_type': 'Organization', 'watchers': 55602} Exercise #2: Alt. solution

Slide 123

Slide 123 text

Reuben Cummings λ @reubano λ #LambdaConf sgrouped = sorted( cut, key=akeyfunc, reverse=True) for record in sgrouped: print(record) Exercise #2: Alt. solution