Slide 1
Slide 1 text
Things I wish I Knew
Before Using Python for
Data Processing
Miguel Cabrera
@mfcabrera
20 July 2016
Slide 2
Slide 2 text
Hello!
I am Miguel!
Data Engineer / Scien6st @ TrustYou
Python ~ 2 years
Berlin
@mfcabrera
mfcabrera at gmail
hAp://mfcabrera.com
2
Slide 3
Slide 3 text
● (Rela6vely) New to Python, mostly Scien6fic
stack
● You have used things like Numpy, Scikit-‐
Learn, Gensim, etc…
● Your job 6tle includes either the word Data
or “Machine Learning”.
● Not necessarily a trained SoWware Engineer
3
Priors!
Slide 4
Slide 4 text
● Data Scien6st?
● Data Analyst?
● Data Engineer?
● Machine Learning Developer?
● SoWware Developer?
● Other?
4
Who is who?
Slide 5
Slide 5 text
● Basic Concepts and prac6ces
● Some goodies of the collec6on
module
● Iterators and Iterables
● Conclusion
5
Agenda
Slide 6
Slide 6 text
● Recent university grad
● Mostly R and Matlab
● Writes niWy code to classify
documents using Jupyter
Notebooks
● Mostly NltK and Scikit-‐Learn
6
David’s Story
Slide 7
Slide 7 text
7
Slide 8
Slide 8 text
8
Slide 9
Slide 9 text
9
Slide 10
Slide 10 text
10
Data Science Spaghetti Code
Slide 11
Slide 11 text
1.
Back to the basics
11
Slide 12
Slide 12 text
Code vs Software
12
Daniel Moisset: hAps://www.youtube.com/watch?v=4dlWg0B4ASw
Slide 13
Slide 13 text
“Code is something that
runs on a Computer”
13
Slide 14
Slide 14 text
Code does not necessarily…
● Have tests
● Follow conven6ons
● Have documenta6on
● Follow processes
14
Slide 15
Slide 15 text
“Software is the
programming text that
is part of a deliverable”
15
Slide 16
Slide 16 text
You want to build Software…
● Maintainable
● Testable
● Deployable
16
Slide 17
Slide 17 text
“
Python is an interpreted,
interac8ve, object-‐oriented
programming language. It
incorporates modules, excep8ons,
dynamic typing, very high level
dynamic data types, and classes.
17
Python is…
Source: https://docs.python.org/3/faq/general.html#what-is-python
Slide 18
Slide 18 text
“
(OOP) is a programming paradigm
based on the concept of "objects",
which may contain data, in the
form of fields, oGen known as
a0ributes; and code, in the form of
procedures, oGen known as
methods.
18
Object Oriented Programming
(OOP)
Source: https://en.wikipedia.org/wiki/Object-oriented_programming
Slide 19
Slide 19 text
How does an object
look in Python?
19
Slide 20
Slide 20 text
How does an object
look in Python?
20
Cookie Cutters & Cookies
Slide 21
Slide 21 text
How does an object
look in Python?
21
Classes & Objects
Slide 22
Slide 22 text
22
class
Cookie(object):
def
__init__(self,
sugar=5):
self.sugar
=
sugar
def
eat(self):
pass
def
split(self):
pass
Slide 23
Slide 23 text
23
class
Cookie(object):
def
__init__(self,
sugar=5):
self.sugar
=
sugar
def
eat(self):
pass
def
split(self):
pass
Slide 24
Slide 24 text
24
class
Cookie(object):
def
__init__(self,
sugar=5):
self.sugar
=
sugar
def
eat(self):
pass
def
split(self):
pass
Slide 25
Slide 25 text
25
class
Cookie(object):
def
__init__(self,
sugar=5):
self.sugar
=
sugar
def
eat(self):
pass
def
split(self):
pass
Slide 26
Slide 26 text
26
class
Cookie(object):
def
__init__(self,
sugar=5):
self.sugar
=
sugar
def
eat(self):
pass
def
split(self):
pass
c
=
Cookie(3)
Slide 27
Slide 27 text
27
class
Alfajor(Cookie):
def
__init__(self,
chocolate=10,
sugar=10):
super(Alfajor,
self).__init__(sugar=sugar)
self.chocolate
=
chocolate
a
=
Alfajor(chocolate=20,
sugar=30)
Slide 28
Slide 28 text
28
from
sklearn
import
svm
data
=
#
multiple
lines
to
load
the
data
X
=
#
multiples
lines
extract
the
features
y
=
#
...
clf
=
svm.SVC()
clf.fit(X,
y)
clf.predict(...)
#
multiples
lines
store
the
results
Slide 29
Slide 29 text
How do I write good
object oriented code?
29
Slide 30
Slide 30 text
How do I write good OO
code?
● DRY
● KISS
● SOLID
30
Slide 31
Slide 31 text
“
Every piece of knowledge must
have a single, unambiguous,
authorita8ve representa8on within
a system
31
Source: https://en.wikipedia.org/wiki/Don%27t_repeat_yourself
DRY: Don’t Repeat Yourself
Slide 32
Slide 32 text
“
Simplicity is the ul8mate
sophis8ca8on
32
Leonardo Davicni
KISS: Keep it Simple Stupid
Slide 33
Slide 33 text
● Single responsibility principle
● Open/closed principle
● Liskov subs6tu6on principle
● Interface segrega6on principle
● Dependency inversion principle
33
Be SOLID
“Principles Of OOD”, Robert C. Martin Source: https://es.wikipedia.org/wiki/SOLID
Slide 34
Slide 34 text
● Single responsibility principle
● Open/closed principle
● Liskov subs6tu6on principle
● Interface segrega6on principle
● Dependency inversion principle
34
Be SOLID
“Principles Of OOD”, Robert C. Martin Source: https://es.wikipedia.org/wiki/SOLID
Slide 35
Slide 35 text
Learn OOP in Python
35
Slide 36
Slide 36 text
36
Coding Conventions
Slide 37
Slide 37 text
● “Readability counts” (PEP20)
● Spaces vs. Tabs
● Indenta6on rules
● Code organiza6on
● PEP-‐8 is the de-‐facto style
37
Coding Conventions
Slide 38
Slide 38 text
38
PEP-8
hAp://pep8.org
Slide 39
Slide 39 text
39
Slide 40
Slide 40 text
● Project structure
● Tes6ng (Check out py.test!)
● Versioning and branching
● Code Reviews
● SoWware Development Life Cycle
40
Other Topics
Slide 41
Slide 41 text
Books!
41
Slide 42
Slide 42 text
Talks @ Europython
42
● Clean Code in Python by Mariano Anaya
(Today at 15:45 Barria 2)
● What’s the point of Object Orienta6on?
by Iwan Vosloo (Thursday 11:15 A2)
Slide 43
Slide 43 text
2.
Tips & Tricks
43
Slide 44
Slide 44 text
2.
Tips & Tricks
The Collec6on Module
44
Slide 45
Slide 45 text
45
Counting
Slide 46
Slide 46 text
First Attempt
46
Slide 47
Slide 47 text
47
items
=
["a",
"b",
"a",
"x",
"x",
"y",
"c",
"c",
"a"]
item_counts
=
{}
for
i
in
items:
if
i
in
items:
item_counts[i]
=
item_counts[i]
+
1
else:
item_counts[i]
=
1
Using dicts!
Slide 48
Slide 48 text
48
items
=
["a",
"b",
"a",
"x",
"x",
"y",
"c",
"c",
"a"]
item_counts
=
{}
for
i
in
items:
try:
item_counts[i]
=
item_counts[i]
+
1
except
KeyError:
item_counts[i]
=
1
Using dicts (EAFP version
)
Slide 49
Slide 49 text
We can do better
49
Slide 50
Slide 50 text
Let’s use defaultdict
50
Slide 51
Slide 51 text
51
from
collections
import
defaultdict
items
=
["a",
"b",
"a",
"x",
"x",
"y",
"c",
"c",
"a"]
item_counts
=
defaultdict(int)
for
i
in
items:
item_counts[i]
=
item_counts[i]
+
1
defaultdict(int,
{'a':
3,
'b':
1,
'c':
2,
'x':
2,
'y':
1})
Using defaultdict
Slide 52
Slide 52 text
Let’s use Counter
52
Slide 53
Slide 53 text
53
from
collections
import
Counter
items
=
["a",
"b",
"a",
"x",
"x",
"y",
"c",
"c",
"a"]
item_counts
=
Counter(items)
print(item_counts)
Using Counter
Counter({'a':
3,
'x':
2,
'c':
2,
'b':
1,
'y':
1})
Slide 54
Slide 54 text
Counter’s extra goodies
54
Slide 55
Slide 55 text
55
Extra goodies
>>>
c
=
Counter(a=3,
b=1)
>>>
d
=
Counter(a=1,
b=2)
>>>
c.most_common()
>>>
c.values()
>>>
c
+
d
>>>
c
-‐
d
>>>
c
&
d
#
intersection:
min(c[x],
d[x])
>>>
c
|
d
#
union:
max(c[x],
d[x])
Slide 56
Slide 56 text
Counter is a class
56
Slide 57
Slide 57 text
Classes can be extended
57
Slide 58
Slide 58 text
58
Counter based PMF
from
collections
import
Counter
class
PMF(Counter):
def
normalize(self):
total
=
float(sum(self.values()))
for
key
in
self:
self[key]
/=
total
Slide 59
Slide 59 text
59
Counter based PMF
from
collections
import
Counter
class
PMF(Counter):
def
normalize(self):
total
=
float(sum(self.values()))
for
key
in
self:
self[key]
/=
total
def
__init__(self,
*args,
**kwargs):
super(PMF,
self).__init__(…)
self.normalize()
Slide 60
Slide 60 text
● Use the and extend Counter
class
● Awesome ar6cle from
@TreyHunner on Coun6ng [1].
60
On Counting and Python
[1] http://treyhunner.com/2015/11/counting-things-in-python/
Slide 61
Slide 61 text
Named Tuples
61
Slide 62
Slide 62 text
● Code around the dict, tuples or
lists
● Never know what to expect
● Code becomes hard to read
62
Named Tuples
Slide 63
Slide 63 text
63
Example
from
math
import
sqrt
pt1
=
(1.0,
5.0)
pt2
=
(2.5,
1.5)
line_length
=
sqrt((pt1[0]
-‐
pt2[0])**2
+
(pt1[1]
-‐
pt2[1])**2)
Source: hAp://stackoverflow.com/ques6ons/2970608/what-‐are-‐named-‐tuples-‐in-‐python
Slide 64
Slide 64 text
64
Example
line_length
=
sqrt((pt1[0]
-‐
pt2[0])**2
+
(pt1[1]
-‐
pt2[1])**2)
Source: hAp://stackoverflow.com/ques6ons/2970608/what-‐are-‐named-‐tuples-‐in-‐python
Slide 65
Slide 65 text
65
Example
from
collections
import
namedtuple
from
math
import
sqrt
Point
=
namedtuple('Point',
'x
y')
pt1
=
Point(1.0,
5.0)
pt2
=
Point(2.5,
1.5)
line_length
=
sqrt((pt1.x
-‐
pt2.x)**2
+
(pt1.y
-‐
pt2.y)**2)
Source: hAp://stackoverflow.com/ques6ons/2970608/what-‐are-‐named-‐tuples-‐in-‐python
Slide 66
Slide 66 text
66
It has cool methods
_asdict Return a new OrderedDict which
maps field names to their values
_make(iterable) Class method that makes a new
instance from an exis6ng sequence
or iterable.
Slide 67
Slide 67 text
67
Extending NamedTuple
from
collections
import
namedtuple
_HotelBase
=
namedtuple(
'HotelDescriptor',
['cluster_id',
'trust_score',
'reviews_count',
'category_scores',
'intensity_factors'],
)
class
HotelDescriptor(_HotelBase):
def
compute_prior(self):
if
not
self.trust_score
or
not
self.reviews_count:
raise
NotEnoughDataForRanking("…")
return
_compute_prior(self.trust_score,…)
Slide 68
Slide 68 text
2.
Tips & Tricks
2.1 Iterators & Iterables
68
Slide 69
Slide 69 text
69
l
=
[1,
2,
3,
4]
for
i
in
x:
print(x)
Slide 70
Slide 70 text
70
an iterator
comprehension
(an) iterable
produces
typically is
iter()
always is
a generator
expression
a generator
funcCon
is
is
a generator
container
(list, dict, etc)
next()
Lazily produce
the next value
By Vincent Driessen - Source: http://nvie.com/posts/iterators-vs-generators/
Slide 71
Slide 71 text
71
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
Slide 72
Slide 72 text
72
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
Slide 73
Slide 73 text
73
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
Slide 74
Slide 74 text
74
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
container
(list, dict, etc)
Slide 75
Slide 75 text
75
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
container
(list, dict, etc)
assert
1
in
[1,
2,
3]
Slide 76
Slide 76 text
76
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
container
(list, dict, etc)
assert
1
in
{1,
2,
3}
Slide 77
Slide 77 text
77
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
typically is
container
(list, dict, etc)
Slide 78
Slide 78 text
78
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
typically is
container
(list, dict, etc)
l
=
[1,
2,
3,
4]
x
=
iter(l)
y
=
iter(l)
Slide 79
Slide 79 text
79
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
typically is
container
(list, dict, etc)
l
=
[1,
2,
3,
4]
x
=
iter(l)
y
=
iter(l)
type(l)
>>
type(x)
>>
Slide 80
Slide 80 text
80
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
typically is
container
(list, dict, etc)
l
=
[1,
2,
3,
4]
x
=
iter(l)
y
=
iter(l)
type(l)
>>
type(x)
>>
next(x)
>>
1
next(y)
>>
1
next(y)
>>
2
Slide 81
Slide 81 text
81
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
typically is
container
(list, dict, etc)
l
=
[1,
2,
3,
4]
for
e
in
l:
print(e)
Slide 82
Slide 82 text
● An iterable is any object that can
return an iterator
● Containers, files, sockets, etc.
● Implement __iter__().
● Some of them may be infinite
● The itertools contain many
helper func6ons
82
Iterables
Slide 83
Slide 83 text
83
class
InverseReader(object):
def
__init__(self):
with
open('file.txt')
as
f:
self.lines
=
f.readlines()
self.index
=
len(self.lines)
-‐
1
def
__iter__(self):
return
self
def
next(self):
#
Python
3
__next__
self.index
-‐=
1
if
self.index
<
0:
raise
StopIteration
return
self.lines[self.index]
Slide 84
Slide 84 text
84
ir
=
InverseReader()
for
line
in
ir:
print
line
Slide 85
Slide 85 text
85
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
typically is
container
is
is
a generator
always is
a generator
expression
a generator
funcCon
Slide 86
Slide 86 text
86
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
a generator
expression
Slide 87
Slide 87 text
87
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
a generator
expression
a generator
funcCon
Slide 88
Slide 88 text
88
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
a generator
expression
a generator
funcCon
Slide 89
Slide 89 text
89
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
is
is
a generator
a generator
expression
a generator
funcCon
Slide 90
Slide 90 text
90
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
is
is
a generator
always is
a generator
expression
a generator
funcCon
Slide 91
Slide 91 text
91
is
a generator
funcCon
is
a generator
expression
a generator
Slide 92
Slide 92 text
92
is
a generator
expression
numbers
=
[x
for
x
in
range(1,
10)]
squares
=
[x
*
x
for
x
in
numbers]
type(squares)
#
list
a generator
Slide 93
Slide 93 text
93
is
a generator
expression
numbers
=
[x
for
x
in
range(1,
10)]
squares
=
[x
*
x
for
x
in
numbers]
type(squares)
#
list
lazy_squares
=
(x
*
x
for
x
in
numbers)
lazy_squares
#
at
0x104c6da00>
a generator
Slide 94
Slide 94 text
94
is
a generator
expression
numbers
=
[x
for
x
in
range(1,
10)]
squares
=
[x
*
x
for
x
in
numbers]
type(squares)
#
list
lazy_squares
=
(x
*
x
for
x
in
numbers)
lazy_squares
#
at
0x104c6da00>
next(lazy_squares)
#
1
next(lazy_squares)
#
4
x
a generator
Slide 95
Slide 95 text
95
is
a generator
expression
numbers
=
[x
for
x
in
range(1,
10)]
squares
=
[x
*
x
for
x
in
numbers]
type(squares)
#
list
lazy_squares
=
(x
*
x
for
x
in
numbers)
lazy_squares
#
at
0x104c6da00>
next(lazy_squares)
#
1
next(lazy_squares)
#
4
lazy_squares
=
(x
*
x
for
x
in
numbers)
for
x
in
lazy_squares:
print
x
a generator
Slide 96
Slide 96 text
96
a generator
is
a generator
funcCon
Slide 97
Slide 97 text
97
a generator
def
fib():
prev,
curr
=
0,1
while
True:
yield
curr
prev,curr
=
curr,
prev+curr
is
a generator
funcCon
Slide 98
Slide 98 text
98
a generator
def
fib():
prev,
curr
=
0,1
while
True:
yield
curr
prev,curr
=
curr,
prev+curr
f
=
fib()
next(f)
#
1
next(f)
#
1
next(f)
#
2
is
a generator
funcCon
Slide 99
Slide 99 text
99
a generator
def
fib():
prev,
curr
=
0,1
while
True:
yield
curr
prev,curr
=
curr,
prev+curr
is
a generator
funcCon
for
x
in
islice(fib(),
0,3):
print
x
# 1
# 1
# 2
Slide 100
Slide 100 text
100
class
HdfsLineSentence(object):
def
__init__(self,
source):
self.source
=
source
def
__iter__(self):
stream
=
self.source.open('r')
for
line
in
stream:
cid,
s
=
line.split('\t')
#
decode
and
do
some
work
with
s
yield
s
sentences
=
HdfsLineSentence(...)
for
s
in
setences:
print(s)
Slide 101
Slide 101 text
101
an iterator
(an) iterable
iter()
always is
next()
Lazily produce
the next value
comprehension
produces
typically is
container
is
is
a generator
always is
a generator
expression
a generator
funcCon
Slide 102
Slide 102 text
What does it have to do with
Data Processing?
102
Slide 103
Slide 103 text
Unknown amout of data
103
Slide 104
Slide 104 text
Not enough memory
104
Slide 105
Slide 105 text
Data streaming via lazy
evaluation
105
Slide 106
Slide 106 text
Data processing pipelines
through iterables
106
Slide 107
Slide 107 text
Chaining iterables
107
Slide 108
Slide 108 text
108
class
HdfsLineSentence(object):
def
__init__(self,
source):
self.source
=
source
def
__iter__(self):
stream
=
self.source.open('r')
for
line
in
stream:
cid,
s
=
line.split('\t')
#
decode
and
do
some
work
with
s
yield
s
sentences
=
HdfsLineSentence(...)
for
s
in
sentences:
print
s
Slide 109
Slide 109 text
109
class
FilterComment(object):
def
__init__(self,
source):
self.source
=
source
def
__iter__(self):
for
s
in
self.source:
if
s[0]
!=
"#":
yield
s
Slide 110
Slide 110 text
110
class
FilterComment(object):
def
__init__(self,
source):
self.source
=
source
def
__iter__(self):
for
s
in
self.source:
if
s[0]
!=
"#":
yield
s
sents
=
FilterComment(HdfsLineSentence(source))
for
s
in
sents:
print
s
Slide 111
Slide 111 text
111
class
FilterComment(object):
def
__init__(self,
source):
self.source
=
source
def
__iter__(self):
for
s
in
self.source:
if
s[0]
!=
"#":
yield
s
sents
=
FilterComment(HdfsLineSentence(source))
for
s
in
sents:
print
s
Slide 112
Slide 112 text
112
class
FilterComment(object):
def
__init__(self,
source):
self.source
=
source
def
__iter__(self):
for
s
in
self.source:
if
s[0]
!=
"#":
yield
s
sents
=
FilterComment(HdfsLineSentence(source))
for
s
in
sents:
print
s
Slide 113
Slide 113 text
113
def
filter_comment(source):
for
s
in
source:
if
s[0]
!=
"#":
yield
s
sents
=
filter_comment(HdfsLineSentence(source))
for
s
in
sents:
print
s
Slide 114
Slide 114 text
Talks @ Europython
114
● Itera6on, itera6on, itera6on by John
Sutherland (Friday 15:45 Barria 1)
Slide 115
Slide 115 text
3.
Conclusions
115
Slide 116
Slide 116 text
Data Scientists /
Engineers / ML Developers
should learn…
116
Slide 117
Slide 117 text
collections and itertools
modules
117
Slide 118
Slide 118 text
Iterables and iterators for data
processing pipelines
118
Slide 119
Slide 119 text
Object oriented programming
119
Slide 120
Slide 120 text
Good soWware engineering
prac6ces
120
Slide 121
Slide 121 text
121
Spaguetti: https://www.flickr.com/photos/129610671@N02/16633987421/ (CC BY-
NC-ND 2.0) vision.communicate
Autovification: Credit: AV Dezign
https://www.flickr.com/photos/91345457@N07/22666878846/ (CC BY-NC-ND 2.0)
Iterators and Iterables based on work of Vincent Driessen : http://nvie.com/posts/iterators-vs-generators/
Ideas from Iterables taken from RaRe Technologies blog; http://rare-technologies.com/data-streaming-
in-python-generators-iterators-iterables/
Credits
Counter image: Dean Hochman Source:
https://www.flickr.com/photos/17997843@N02/24061690099/“ (CC BY-NC-ND 2.0)
PMF Class based on Vik Paruchuri’s https://www.dataquest.io/blog/python-counter-class/
Cookies: Source Wikipedia
https://en.wikipedia.org/wiki/File:R%C5%AFzn%C3%A9_druhy_cukrov%C3%AD_(2).jpg (CC BY 3.0)
Counting things in Python: http://treyhunner.com/2015/11/counting-things-in-python/
Slide 122
Slide 122 text
We are Hiring!
Visit our table for more info!