A Functional Programming Approach To Data Processing In Python

A FUNCTIONAL PROGRAMMING
APPROACH TO
LambdaConf — Boulder, Colorado — May 26, 2017
DATA PROCESSING IN PYTHON
By Reuben Cummings

View Slide

Reuben Cummings λ @reubano λ #LambdaConf
Who am I?
Managing Director, Nerevu Development
Founder, Arusha Coders
Author of several popular Python packages

View Slide

WHAT IS DATA?
I dare you, I double dare you!
Image Credit: www.emaze.com
SAY BIG DATA
ONE MORE TIME

View Slide

Organization
language presenter
mercury alex
scala gleb
haskell michael
"This session
seeks to
entertain and
teach the
developer who
is already..."
structured unstructured

View Slide

Storage
type,duration
leap,360
hop,120
de novo,60
inspire,10
00103e0 b0e6 04...
00105f0 e4e7 03...
0010600 0be8 04...
00105b0 c4e4 02...
00106e0 b0e9 04...
ﬂat/text binary

View Slide

Organization vs Storage
ﬂat/text
binary
structured
unstructured

View Slide

What is it
data processing
good for?

View Slide

Spotify's Discovery
Weekly
playlist of new songs you like
adapts to user's shifting
musical tastes
handles outliers and
seasonality Image Credit: www.spotify.com/int/discoverweekly/

View Slide

What is
functional
programming
good for?

View Slide

Om
rapid UI re-renders
serializable application state
time travel/undo
Image Credit: circleci.com

View Slide

A BRIEF INTRO TO PYTHON
Ooouuu, stickers!
Image Credit: www.pythongear.com

View Slide

Presentation: GitHub repo
github.com/reubano/lambdaconf-
tutorial

View Slide

Presentation: Jupyter Notebook
beta.mybinder.org/v2/gh/reubano/
lambdaconf-tutorial/master
(presentation.ipybn)

View Slide

Naive
Image Credit: (Alika Seu) www.ﬂickr.com

View Slide

Reading data

View Slide

Reading data (naive)
from urllib.request import urlopen
from json import loads
BASE = 'https://api.github.com/search'
_url1 = '{}/repositories?q={}'
q = 'data&per_page=100'
url1 = _url1.format(BASE, q)
f = urlopen(url1)

View Slide

GitHub API
Image Credit: https://api.github.com/search/repositories?q=data

View Slide

>>> data = loads(f.read().decode('utf-8'))

View Slide

>>> repos = data['items']
>>> repos[0]['description']
'Jargon from the functional programming world
in simple terms!'
>>> repos[0]['full_name']
'hemanth/functional-programming-jargon'
>>> data = loads(f.read().decode('utf-8'))

View Slide

Processing data

View Slide

Processing data (naive)
def rate(repos):
rated = []
for repo in repos:
rated.append(repo['watchers'] * 2)
return rated

View Slide

Processing data (naive)
>>> rate(repos)[:5]
[36520, 30174, 28576, 26842, 24092]

View Slide

>>> from itertools import count
>>>
>>> inf_repos = (
... {'watchers': c} for c in count())
Processing infinite data (naive)

View Slide

>>>
>>> inf_repos = (
>>>
>>> rate(inf_repos)

View Slide

KeyboardInterrupt
Traceback (most recent call last)
in ()
>>>
>>> inf_repos = (
>>>
>>> rate(inf_repos)

View Slide

rated = []
for repo in repos:
return rated
def rate(repos):
Processing expensive data (naive)

View Slide

def exp_rate(repos):
rated = []
for repo in repos:
return rated

View Slide

from time import sleep
rated = []
for repo in repos:
return rated

View Slide

from time import sleep
rated = []
for repo in repos:
return rated
sleep(5)

View Slide

>>> exp_rate(repos)[:5]

View Slide

[36520, 30174, 28576, 26842, 24092]
>>> exp_rate(repos)[:5]

View Slide

Lazy evaluation
Image Credit: (Mark Turnauckas) www.ﬂickr.com

View Slide

Lazy intro

View Slide

>>> next(lazy_list)
0
>>> eager_list = list(range(5))
>>> eager_list
[0, 1, 2, 3, 4]
>>> lazy_list = iter(eager_list)
>>> lazy_list

Iterators

View Slide

>>> next(lazy_list)
StopIteration
Traceback (most recent call last)
in ()
----> 1 next(lazy_list)
Iterators
>>> list(lazy_list)
[1, 2, 3, 4]

View Slide

Reading data

View Slide

$ pip install ijson
Reading data (lazy evaluation)

View Slide

>>> from ijson import items
>>>
>>> f = urlopen(url1)
>>> repos = items(f, 'items.item')
>>> repos

>>> repo = next(repos)
>>> repo['full_name']
'hemanth/functional-programming-jargon'
Reading data (lazy evaluation)

View Slide

Processing data

View Slide

rated = []
for repo in repos:
return rated
def rate(repos):
Processing data (lazy evaluation)

View Slide

rated = []
for repo in repos:
return rated
def gen_rates(repos):

View Slide

for repo in repos:
yield repo['watchers'] * 2

View Slide

>>> rates = gen_rates(repos)
>>> next(rates)
36520
>>> next(rates)
30174
>>> gen_rates(repos)


View Slide

Processing infinite data
(lazy evaluation)
>>> rates = gen_rates(inf_repos)
>>> next(rates)
42220156

View Slide

Processing expensive data
(lazy evaluation)
def gen_exp_rates(repos):
for repo in repos:
sleep(5)

View Slide

>>> list(result)
>>> from itertools import islice
>>>
>>> rates = gen_exp_rates(repos)
>>> result = islice(rates, 5)
(lazy evaluation)

View Slide

[36520, 30174, 28576, 26842, 24092]
>>> list(result)
>>>
(lazy evaluation)

View Slide

>>>
>>> list(result)
[36520, 30174, 28576, 26842, 24092]
>>> next(rates)
648
(lazy evaluation)

View Slide

Grouping data

View Slide

Grouping data
>>> repos = items(f, 'items.item')
>>> repo = next(repos)
>>> repo.keys()
dict_keys(['id', 'name', 'full_name', 'owner',
'private', 'html_url',
'description', 'fork', 'url',
'forks_url', 'keys_url', ...])

View Slide

Grouping data
>>> repo['has_issues']
True

View Slide

Grouping data
>>> import itertools as it
>>> from operator import itemgetter
>>>
>>> keyfunc = itemgetter('has_issues')
>>> sorted_repos = sorted(repos, key=keyfunc)
>>> grouped = it.groupby(
... sorted_repos, keyfunc)
>>> data = (
... (k, len(list(g))) for k, g in grouped)

View Slide

Grouping data
>>> next(data)
(False, 3)
>>> next(data)
(True, 96)

View Slide

Memoization
Image Credit: (olho wodzynski) www.ﬂickr.com

View Slide

Processing data

View Slide

for repo in repos:
sleep(5)
Processing expensive data (memoization)

View Slide

def calc_rate(watchers):
sleep(5)
return watchers * 2
for repo in repos:
yield calc_rate(repo['watchers'])

View Slide

def _calc_rate(watchers):
cacher = lru_cache()
calc_rate = cacher(_calc_rate)
from functools import lru_cache
sleep(5)
return watchers * 2

View Slide

@lru_cache()
from functools import lru_cache
def calc_rate(watchers):
sleep(5)
return watchers * 2
for repo in repos:
yield calc_rate(repo['watchers'])

View Slide

[10, 10, 10, 10, 10]
>>> list(result)
>>> repos = it.repeat({'watchers': 5})

View Slide

EXERCISE #1
Mount Meru — Arusha, Tanzania
Image Credit: Reuben Cummings

View Slide

Exercise #1: Problem
display the total # of
watchers per language
(ignore repos w/o a language)

View Slide

Exercise #1: Result
C# 32
C++ 63
HTML 349
JavaScript 3881
Jupyter Notebook 5481
PHP 201
Python 37007
R 18

View Slide

Exercise #1: Data source
https://api.github.com/search/
repositories?q=data

View Slide

Exercise #1: Jupyter Notebook
(exercises.ipybn)

View Slide

from itertools import groupby
from operator import itemgetter
from ijson import items
url2 = '{}/repositories?q=data'.format(BASE)
f = urlopen(url2)
repos = items(f, 'items.item')
Exercise #1: Solution

View Slide

keyfunc = itemgetter('language')
cleaned = filter(keyfunc, repos)
records = sorted(cleaned, key=keyfunc)
grouped = groupby(records, keyfunc)
for key, group in grouped:
cnt = sum(g['watchers'] for g in group)
print(key, cnt)

View Slide

(solutions.ipybn)

View Slide

INTRODUCING MEZA
Because you might not need Pandas
Image Credit: github.com/reubano/meza

View Slide

$ pip install meza
Meza demo

View Slide

Meza demo: Jupyter Notebook
(presentation.ipybn)

View Slide

Reading data

View Slide

Reading data
>>> from urllib.request import urlopen
>>> from meza.io import read_json
>>>
>>> records = read_json(f, path='items.item')
>>> repo = next(records)
'emberjs/data'

View Slide

Reading data
>>> len(list(records))
29

View Slide

Reading data
>>> from io import StringIO
>>> from meza.io import read_csv
>>>
>>> f = StringIO(
... 'greeting,location\nhello,world\n')
>>>
>>> next(read_csv(f))
{'greeting': 'hello', 'location': 'world'}

View Slide

Reading data
>>> from os import path as p
>>> from meza.io import join
>>>
>>> url3 = '{}&page=2'.format(url2)
>>> files = map(urlopen, [url2, url3])
>>> records = join(
... *files, ext='json', path='items.item')

View Slide

Reading data
'emberjs/data'

View Slide

Reading data
'emberjs/data'
>>> repo['language']
'JavaScript'

View Slide

Reading data
'emberjs/data'
>>> repo['language']
'JavaScript'
>>> len(list(records))
59

View Slide

Transforming data

View Slide

Transforming data
>>> from meza.process import merge
>>>
>>> records = [
... {'a': 200}, {'b': 300}, {'c': 400}]
>>>
>>> merge(records)
{'a': 200, 'b': 300, 'c': 400}

View Slide

Transforming data
>>> from meza.process import group
>>>
>>> records = [
... {'item': 'a', 'amount': 200},
... {'item': 'a', 'amount': 200},
... {'item': 'b', 'amount': 400}]
>>>
>>> grouped = group(records, 'item')

View Slide

Transforming data
>>> key, _group = next(grouped)
>>> key
'a'
>>> _group
[{'amount': 200, 'item': 'a'},
{'amount': 200, 'item': 'a'}]

View Slide

Transforming data
>>> from meza import process as pr
>>>
>>> raw = read_json(f, path='items.item')
>>> fields = [
... 'full_name', 'language', 'watchers',
... 'score', 'has_wiki']
>>>
>>> cut = pr.cut(raw, fields)

View Slide

Transforming data
>>> cut
. at
0x10b0410f8>
>>> cut, preview = pr.peek(cut)
>>> cut

>>> len(preview)
5

View Slide

Transforming data
>>> preview[0]
{'full_name': 'substance/data',
'has_wiki': True,
'language': 'JavaScript',
'score': Decimal('72.90926'),
'watchers': 678}

View Slide

Transforming data
>>> filled = pr.fillempty(
... raw, value='', fields=['language'])
>>>
>>> pivoted = pr.pivot(
... filled, 'score', 'language',
... rows=['has_wiki'], op=min)

View Slide

Transforming data
>>> next(pivoted)
{'HTML': Decimal('73.52254'),
'JavaScript': Decimal('53.48755'),
'PHP': Decimal('41.3122'),
'Python': Decimal('42.49319'),
'has_wiki': False}

View Slide

Transforming data
>>> next(pivoted)
{'': Decimal('44.83392'),
'C#': Decimal('47.793495'),
'HTML': Decimal('69.20008'),
'JavaScript': Decimal('70.15174'),
'PHP': Decimal('44.251198'),
'Python': Decimal('45.78215'),
'R': Decimal('46.23451'),
'has_wiki': True}

View Slide

| full_name | language | score | has_wiki |
| --------- | ---------- | ------ | -------- |
| 'aptnote…' | '' | 76.11… | True |
| 'GSA/dat…' | 'HTML' | 73.52… | False |
| 'substan…' | 'JavaScr…' | 72.83… | True |
| 'GoogleT…' | 'JavaScr…' | 70.15… | True |
| 'curran/…' | 'HTML' | 69.20… | True |
Transforming data (before)

View Slide

| has_wiki | '' | HTML | JavaScript |
| -------- | -------- | -------- | ---------- |
| False | | 73.52254 | |
| True | 76.11933 | 69.20008 | 70.15174 |
Transforming data (after)

View Slide

EXERCISE #2
Image Credit: Reuben Cummings
Mount Kilimanjaro — Kilimanjaro Region, Tanzania

View Slide

Exercise #2: Problem
display the language with the
most # of watchers per
owner_type per has_pages

View Slide

Exercise #2: Result (partial)
{'has_pages': True,
'language': 'JavaScript',
'owner_type': 'Organization',
'watchers': 128605}

View Slide

Exercise #2: Data source
https://api.github.com/search/
repositories?
q=data&sort=stars&order=desc

View Slide

Exercise #2: Hint
from meza.fntools import flatten
# and one of the following
from meza.process import normalize # this
from meza.process import aggregate # or this

View Slide

Exercise #2: Jupyter Notebook
(exercises.ipybn)

View Slide

from functools import partial
from meza import process as pr, fntools as ft
from meza.io import read_json
q = 'data&sort=stars&order=desc'
url4 = '{}/repositories?q={}'.format(BASE, q)
f = urlopen(url4)

View Slide

records = read_json(f, path='items.item')
filled = pr.fillempty(
records, value='', fields=['language'])
flat = (dict(ft.flatten(r)) for r in filled)
args = ('watchers', 'language')
rows = ['has_pages', 'owner_type']

View Slide

spun = pr.pivot(
flat, *args, rows=rows, op=sum)
spun, preview = pr.peek(spun)

View Slide

>>> preview[0]
{'C#': 7675,
'C++': 55602,
'Go': 13223,
'Objective-C': 10556,
…
'has_pages': False,
'owner_type': 'Organization'}

View Slide

>>> kw = {'rows': rows, 'invert': True}
>>> normal = pr.normalize(spun, *args, **kw)
>>> normal, preview = pr.peek(normal)
>>> preview[0]
{'has_pages': False,
'language': 'Objective-C',
'watchers': 10556}

View Slide

akeyfunc = itemgetter('watchers')
gkeyfunc = lambda x: tuple(x[r] for r in rows)
aggregator = partial(max, key=akeyfunc)
kwargs = {
'tupled': False, 'aggregator': aggregator}
grouped = pr.group(normal, gkeyfunc, **kwargs)

View Slide

>>> grouped, preview = pr.peek(grouped)
>>> preview[0]
'language': 'C++',
'watchers': 55602}

View Slide

sgrouped = sorted(
grouped, key=akeyfunc, reverse=True)
for record in sgrouped:
print(record)

View Slide

| language | watchers | owner_ty… | has_pages |
| -------- | -------- | --------- | --------- |
| 'JavaS…' | 128605 | 'Organi…' | True |
| 'C++' | 55602 | 'Organi…' | False |
| 'Python' | 54269 | 'User' | False |
| 'Jupyte…'| 12046 | 'User' | True |
Exercise #2: Result (full)

View Slide

(solutions.ipybn)

View Slide

Thanks!
Reuben Cummings
@reubano

View Slide

Extra Slides

View Slide

for repo in repos:

View Slide

return (
r['watchers'] * 2 for r in repos)

View Slide

from functools import partial
from meza import process as pr, fntools as ft
from meza.io import read_json
q = 'data&sort=stars&order=desc'
url4 = '{}/repositories?q={}'.format(BASE, q)
f = urlopen(url4)
Exercise #2: Alt. solution

View Slide

records = read_json(f, path='items.item')
filled = pr.fillempty(
records, value='', fields=['language'])
flat = (dict(ft.flatten(r)) for r in filled)
akeyfunc = itemgetter('watchers')

View Slide

rows = [
'has_pages', 'owner_type', 'language',
'watchers']
def grouper(records, rows, aggregator):
kwargs = {'aggregator': aggregator}
key = lambda x: tuple(x[r] for r in rows)
_grouper = partial(pr.group, tupled=False)
return _grouper(records, key, **kwargs)

View Slide

def agg1(records):
args = (records, 'watchers', sum)
return pr.aggregate(*args)
grouped = grouper(flat, rows[:3], agg1)
agg2 = partial(max, key=akeyfunc)
regrouped = grouper(grouped, rows[:2], agg2)
cut = pr.cut(regrouped, rows)

View Slide

>>> cut, preview = pr.peek(cut)
>>> preview[0]
'language': 'C++',
'watchers': 55602}

View Slide

sgrouped = sorted(
cut, key=akeyfunc, reverse=True)
for record in sgrouped:
print(record)

View Slide

A Functional Programming Approach To Data Processing In Python

A Functional Programming Approach To Data Processing In Python

Reuben Cummings

More Decks by Reuben Cummings

Other Decks in Programming

Featured

Transcript