Wat‽ Mind-bending Edge Cases in Python

Slide 1

Slide 1 text

I'm Dustin http://github.com/di

Slide 2

Slide 2 text

I work at PromptWorks who I'd like to thank for sponsoring my work in the open source community as well as this talk, which I call

Slide 3

Slide 3 text

So we're going to talk about some wats in Python You might ask "what is a wat?" wats are not trick questions wats are not bugs in the language A 'wat' is an edge-case in a language that makes you say: wat‽ Mind-Bending Edge Cases (in Python)

Slide 4

Slide 4 text

they seem weird, but if you understand why they happen, they make sense we're going to look at ten different wats but to make things interesting, a few are actually impossible and you'll need to decide if they're real or not wat‽

Slide 5

Slide 5 text

Let's start with an example to get started wat #0

Slide 6

Slide 6 text

The first goal here is to determine if this is possible or not If it is possible, what can we replace the ellipsis with To give us the desired result The missing values in these wats are limited to the built-in primitive types and collections booleans, integers, strings as well as collections like lists and sets, and combinations of these No lambdas, partials, classes, or other tricky wat #0 >>> x = ... >>> x == x False

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

so if x is equal to wat #0 >>> x = ... >>> x == x False

Slide 9

Slide 9 text

zero times one times ten raised to the 309th power wat #0 - Possible! >>> x = 0*1e309 >>> x == x False

Slide 10

Slide 10 text

this is because 0*1e309 is interpreted by python as Not A Number wat #0 - Possible! >>> x = 0*1e309 >>> x == x False >>> x nan

Slide 11

Slide 11 text

This is the same thing as zero times infinity wat #0 - Possible! >>> x = 0*1e309 >>> x == x False >>> x nan >>> 0*float('inf') nan

Slide 12

Slide 12 text

Or just float nan wat #0 - Possible! >>> x = 0*1e309 >>> x == x False >>> x nan >>> 0*float('inf') nan >>> float('nan') nan

Slide 13

Slide 13 text

A brief word on NaN There's a good reason why nan is not equal to itself, or anything else It's because it's not a number! NaN is designed to propagate through all calculations, so if somewhere in your deep, complex calculations you hit upon a NaN, you don't bubble out a seemingly sensible answer. So because of this, NaN is definitely the source of many wats NaN

Slide 14

Slide 14 text

In fact, it's the star of Gary Bernhardt's infamous talk which is definitely inspiration for this talk A brief word on NaN This is not Python: > Array(16).join("wat" - 1) + " Batman!" "NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN Batman!" https://www.destroyallsoftware.com/talks/wat

Slide 15

Slide 15 text

There are a number of sensible ways to generate a NaN in Python None of them are equal to themselves For the purposes of this talk, none of the "solutions" to the wats involve using a NaN A brief word on NaN >>> 0*1e309 nan >>> float('nan') nan >>> from decimal import Decimal; Decimal('nan') Decimal('NaN') >>> complex('nan') (nan+0j)

Slide 16

Slide 16 text

wat #1

Slide 17

Slide 17 text

can we create a list x such that when sliced by a, b, and c it has a new max wat #1 >>> x = ... >>> a = ... >>> b = ... >>> c = ... >>> max(x) < max(x[a:b:c]) True

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

This wat is impossible This will always compare the list with some subset of the list no matter how you slice it wat #1 - Not Possible >>> x = ... >>> a = ... >>> b = ... >>> c = ... >>> max(x) < max(x[a:b:c]) True

Slide 20

Slide 20 text

wat #2

Slide 21

Slide 21 text

Can we make x and y such that min(x, y) is different than min(y, x)? This wat is possible wat #2 >>> x = ... >>> y = ... >>> min(x, y) == min(y, x) False

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

wat #2 - Possible! >>> x = ... >>> y = ... >>> min(x, y) == min(y, x) False

Slide 24

Slide 24 text

if x is the set containing zero wat #2 - Possible! >>> x = {0} >>> y = ... >>> min(x, y) == min(y, x) False

Slide 25

Slide 25 text

and y is the set containing anything but zero Only if x and y are sets, though. Why? wat #2 - Possible! >>> x = {0} >>> y = {1} >>> min(x, y) == min(y, x) False

Slide 26

Slide 26 text

If we take the min of the set containing zero, and the set containing one We get the set containing zero wat #2 - Possible! >>> min({0}, {1}) set([0])

Slide 27

Slide 27 text

If we do the opposite, we get the set containing one! Seems like min is broken and just returning the first element wat #2 - Possible! >>> min({0}, {1}) set([0]) >>> min({1}, {0}) set([1])

Slide 28

Slide 28 text

Well, maybe not. What's happening? Let's imagine what the min function might look like If we implemented it in python wat #2 - Possible! >>> min({0}, {1}) set([0]) >>> min({1}, {0}) set([1]) >>> min({0, 1}, {0}) set([0])

Slide 29

Slide 29 text

The min function can take any number of arguments It finds the min of all of them It can also take just one, an iterator, but let's ignore that for now wat #2 - Possible! >>> def min(*args): ...

Slide 30

Slide 30 text

We need two variables hasitem tells us if minitem has been set yet min_item holds the smallest item found at any point wat #2 - Possible! >>> def min(*args): ... has_item = False ... min_item = None ...

Slide 31

Slide 31 text

We'll iterate through the arguments wat #2 - Possible! >>> def min(*args): ... has_item = False ... min_item = None ... for x in args: ...

Slide 32

Slide 32 text

If we haven't set minitem yet, or if x is less than minitem wat #2 - Possible! >>> def min(*args): ... has_item = False ... min_item = None ... for x in args: ... if not has_item or x < min_item: ...

Slide 33

Slide 33 text

We set hasitem to true and set minitem to x wat #2 - Possible! >>> def min(*args): ... has_item = False ... min_item = None ... for x in args: ... if not has_item or x < min_item: ... has_item = True ... min_item = x ...

Slide 34

Slide 34 text

Finally we return the smallest item found This is a really simplified approach -- doesn't handle no args, etc. What's the key? It's that less than operator comparing x to min_item wat #2 - Possible! >>> def min(*args): ... has_item = False ... min_item = None ... for x in args: ... if not has_item or x < min_item: ... has_item = True ... min_item = x ... return min_item ...

Slide 35

Slide 35 text

The less than operator works as you'd expect for something like integers wat #2 - Possible! >>> 0 < 1 True

Slide 36

Slide 36 text

But the operator is overloaded for set comparison Meaning it behaves differently for two sets than it would for two ints wat #2 - Possible! >>> 0 < 1 True >>> {0} < {1} False

Slide 37

Slide 37 text

Specifically, it is the inclusion operator Here, it's checking if what's in the set on the left is included in the set on the right wat #2 - Possible! >>> 0 < 1 True >>> {0} < {1} False >>> {0} < {0, 1} True

Slide 38

Slide 38 text

So when we call the min on these two one-element sets We will always get the first argument back, regardless wat #2 - Possible! >>> 0 < 1 True >>> {0} < {1} False >>> {0} < {0, 1} True >>> min({0}, {1}) set([0])

Slide 39

Slide 39 text

wat #3

Slide 40

Slide 40 text

Can we create two lists x and y such that at least one element in x is true But when appended with y, none of them are true? This wat is impossible wat #3 >>> x = ... >>> y = ... >>> any(x) and not any(x + y) True

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

This wat is impossible The elements of y have no effect on those in x If anything in x is true, something in x+y will be true as well wat #3 - Not Possible >>> x = ... >>> y = ... >>> any(x) and not any(x + y) True

Slide 43

Slide 43 text

wat #4

Slide 44

Slide 44 text

This wat is possible wat #4 >>> x = ... >>> y = ... >>> x.count(y) > len(x) True

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

This wat is possible wat #4 - Possible! >>> x = ... >>> y = ... >>> x.count(y) > len(x) True

Slide 47

Slide 47 text

If x is any string wat #4 - Possible! >>> x = 'foobar' >>> y = ... >>> x.count(y) > len(x) True

Slide 48

Slide 48 text

But only if y is an empty string wat #4 - Possible! >>> x = 'foobar' >>> y = '' >>> x.count(y) > len(x) True

Slide 49

Slide 49 text

wat #4 - Possible! >>> x = 'foobar' >>> y = '' >>> x.count(y) > len(x) True >>> len('foobar') 6

Slide 50

Slide 50 text

Let's imagine what the count function might look like wat #4 - Possible! >>> x = 'foobar' >>> y = '' >>> x.count(y) > len(x) True >>> len('foobar') 6 >>> 'foobar'.count('') 7

Slide 51

Slide 51 text

To make things easy let's just make a function that takes a string s and sub wat #4 - Possible! >>> def count(s, sub): ...

Slide 52

Slide 52 text

We initialize the result to return as zero wat #4 - Possible! >>> def count(s, sub): ... result = 0 ...

Slide 53

Slide 53 text

We iterate through all the indexes at which sub could start in s wat #4 - Possible! >>> def count(s, sub): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ...

Slide 54

Slide 54 text

If the substring is larger than the string, we get an empty list of indexes wat #4 - Possible! >>> def count(s='foo', sub='foobar'): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... # range(3 + 1 - 6) ... # range(-2) ... # []

Slide 55

Slide 55 text

If the substring is the same size as string, we get one index, [0] wat #4 - Possible! >>> def count(s='foobar', sub='foobar'): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... # range(6 + 1 - 6) ... # range(1) ... # [0]

Slide 56

Slide 56 text

If the substring is smaller than the string, we get possible start indexes wat #4 - Possible! >>> def count(s='foobar', sub='foo'): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... # range(6 + 1 - 3) ... # range(4) ... # [0, 1, 2, 3]

Slide 57

Slide 57 text

But if substring is empty string, we get one more index than the length wat #4 - Possible! >>> def count(s='foobar', sub=''): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... # range(6 + 1 - 0) ... # range(7) ... # [0, 1, 2, 3, 4, 5, 6]

Slide 58

Slide 58 text

Using the index, we get a slice of the string as our possible match wat #4 - Possible! >>> def count(s, sub): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ...

Slide 59

Slide 59 text

When the substring has a length, this gives us a slice the same size wat #4 - Possible! >>> def count(s='foobar', sub='foo'): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ... # s[0:0 + 3] ... # s[0:3] ... # 'foo'

Slide 60

Slide 60 text

When it's empty though, we just get another empty string wat #4 - Possible! >>> def count(s='foobar', sub=''): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ... # s[0:0 + 0] ... # s[0:0] ... # ''

Slide 61

Slide 61 text

Slicing also won't raise an IndexError when the index is longer than the string wat #4 - Possible! >>> def count(s='foobar', sub=''): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ... # s[6:6 + 0] ... # s[6:6] ... # ''

Slide 62

Slide 62 text

Finally if the possible match is the same as the substring We increment the result counter wat #4 - Possible! >>> def count(s, sub): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ... if possible_match == sub: ... result += 1 ... return result ...

Slide 63

Slide 63 text

For an empty string, the range is the indexes 0 through 6, for seven total And the possible_match matches every time! Giving us a count of 7 wat #4 - Possible! >>> def count(s, sub): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ... if possible_match == sub: ... result += 1 ... return result ... >>> count('foobar', '') 7

Slide 64

Slide 64 text

wat #5

Slide 65

Slide 65 text

wat #5 >>> x = ... >>> y = ... >>> z = ... >>> x * (y * z) == (x * y) * z False

Slide 66

Slide 66 text

No content

Slide 67

Slide 67 text

wat #5 - Possible! >>> x = ... >>> y = ... >>> z = ... >>> x * (y * z) == (x * y) * z False

Slide 68

Slide 68 text

wat #5 - Possible! >>> x = [0] >>> y = ... >>> z = ... >>> x * (y * z) == (x * y) * z False

Slide 69

Slide 69 text

wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = ... >>> x * (y * z) == (x * y) * z False

Slide 70

Slide 70 text

wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False

Slide 71

Slide 71 text

wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1)

Slide 72

Slide 72 text

A list times one is itself wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1) == [0]*1

Slide 73

Slide 73 text

A list times one is itself This makes sense wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1) == [0]*1 == [0] True

Slide 74

Slide 74 text

What happens when we multiple a list by a negative number? wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1) == [0]*1 == [0] True >>> (x * y) * z == ([0]*-1)*-1

Slide 75

Slide 75 text

Values of n less than 0 are treated as 0 which yields an empty sequence of the same type as s. in this case, an empty list wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1) == [0]*1 == [0] True >>> (x * y) * z == ([0]*-1)*-1 == []*-1

Slide 76

Slide 76 text

Again, an empty list wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1) == [0]*1 == [0] True >>> (x * y) * z == ([0]*-1)*-1 == []*-1 == [] True

Slide 77

Slide 77 text

There are actually other ways to make this wat possible as well for example wat #5 - Possible! >>> x = ... >>> y = ... >>> z = ... >>> x * (y * z) == (x * y) * z False

Slide 78

Slide 78 text

if we set x, y and z to these mysterious values wat #5 - Possible! >>> x = 5e-234 >>> y = 3 >>> z = 9007199254740993 >>> x * (y * z) == (x * y) * z False

Slide 79

Slide 79 text

we get this value for the first half wat #5 - Possible! >>> x = 5e-234 >>> y = 3 >>> z = 9007199254740993 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) 1.335044315104321e-307

Slide 80

Slide 80 text

and this one for the second half this is due to the way floats are, by their nature, imprecise basically the two results are off by the difference of the least significant bit wat #5 - Possible! >>> x = 5e-234 >>> y = 3 >>> z = 9007199254740993 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) 1.335044315104321e-307 >>> (x * y) * z 1.3350443151043208e-307

Slide 81

Slide 81 text

wat #6

Slide 82

Slide 82 text

Can we define two lists x and y such that x < y But when we compare each individual element, every element in x > y This wat is possible wat #6 >>> x = ... >>> y = ... >>> x < y and all(a >= b for a, b in zip(x, y)) True

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

wat #6 - Possible! >>> x = ... >>> y = ... >>> x < y and all(a >= b for a, b in zip(x, y)) True

Slide 85

Slide 85 text

True for any interables where x is zero-length wat #6 - Possible! >>> x = '' >>> y = ... >>> x < y and all(a >= b for a, b in zip(x, y)) True

Slide 86

Slide 86 text

And y is any iterable wat #6 - Possible! >>> x = '' >>> y = 'foobar' >>> x < y and all(a >= b for a, b in zip(x, y)) True

Slide 87

Slide 87 text

Comparison of sequences uses lexicographical ordering Basically alphabetization. 'dog' comes before 'dogfish' Empty list comes before non empty lists wat #6 - Possible! >>> x = '' >>> y = 'foobar' >>> x < y and all(a >= b for a, b in zip(x, y)) True >>> '' < 'foobar' True

Slide 88

Slide 88 text

Zipping two lists of uneven length will give a list of the shorter length wat #6 - Possible! >>> x = '' >>> y = 'foobar' >>> x < y and all(a >= b for a, b in zip(x, y)) True >>> '' < 'foobar' True >>> zip('', 'foobar') []

Slide 89

Slide 89 text

all of an empty list is true! all short-circuits -- if anything is false return false, otherwise true wat #6 - Possible! >>> x = '' >>> y = 'foobar' >>> x < y and all(a >= b for a, b in zip(x, y)) True >>> '' < 'foobar' True >>> zip('', 'foobar') [] >>> all([]) True

Slide 90

Slide 90 text

wat #7

Slide 91

Slide 91 text

This wat is not possible. wat #7 >>> x = ... >>> len(set(list(x))) == len(list(set(x))) False

Slide 92

Slide 92 text

No content

Slide 93

Slide 93 text

This wat is not possible. Converting a list to a set might reduce the length of x But converting a set to a list will never add elementes wat #7 - Not Possible >>> x = ... >>> len(set(list(x))) == len(list(set(x))) False

Slide 94

Slide 94 text

wat #8

Slide 95

Slide 95 text

Can we make x such that min(x) is not the same as min(x unpacked) This wat is possible wat #8 >>> x = ... >>> min(x) == min(*x) False

Slide 96

Slide 96 text

No content

Slide 97

Slide 97 text

wat #8 - Possible! >>> x = ... >>> min(x) == min(*x) False

Slide 98

Slide 98 text

Remember earlier when I said min could take either a series of arguments Or just an iterable? This wat exists because of the different ways min handles its args wat #8 - Possible! >>> x = [[0]] >>> min(x) == min(*x) False

Slide 99

Slide 99 text

These are all the same wat #8 - Possible! >>> x = [[0]] >>> min(x) == min(*x) False >>> min([1, 2, 3]) == min(*[1, 2, 3]) == min(1, 2, 3) True

Slide 100

Slide 100 text

So the min of x is just the first element in it wat #8 - Possible! >>> x = [[0]] >>> min(x) == min(*x) False >>> min([1, 2, 3]) == min(*[1, 2, 3]) == min(1, 2, 3) True >>> min(x) == [0] True

Slide 101

Slide 101 text

But the min of x unpacked is the min of the list containing zero Which is just zero wat #8 - Possible! >>> x = [[0]] >>> min(x) == min(*x) False >>> min([1, 2, 3]) == min(*[1, 2, 3]) == min(1, 2, 3) True >>> min(x) == [0] True >>> min(*x) == min([0]) == 0 True

Slide 102

Slide 102 text

wat #9

Slide 103

Slide 103 text

This wat is impossible wat #9 >>> x = ... >>> y = ... >>> sum(0 * x, y) == y False

Slide 104

Slide 104 text

No content

Slide 105

Slide 105 text

This wat is impossible Anything times zero is either zero or an empty sequence The sum of zero or an empty sequence is always zero wat #9 - Not Possible >>> x = ... >>> y = ... >>> sum(0 * x, y) == y False

Slide 106

Slide 106 text

Here, y is the "start" of the sum wat #9 - Not Possible >>> x = ... >>> y = ... >>> sum(0 * x, y) == y False >>> sum([1, 1, 1], 7) 10

Slide 107

Slide 107 text

When the sequence is empty we just get the start value wat #9 - Not Possible >>> x = ... >>> y = ... >>> sum(0 * x, y) == y False >>> sum([1, 1, 1], 7) 10 >>> sum([], 7) 7

Slide 108

Slide 108 text

wat #10

Slide 109

Slide 109 text

Can we find two values x and y such that y > max(x), but y is also in x This wat is possible wat #10 >>> x = ... >>> y = ... >>> y > max(x) and y in x True

Slide 110

Slide 110 text

No content

Slide 111

Slide 111 text

wat #10 - Possible! >>> x = ... >>> y = ... >>> y > max(x) and y in x True

Slide 112

Slide 112 text

wat #10 - Possible! >>> x = 'aa' >>> y = ... >>> y > max(x) and y in x True

Slide 113

Slide 113 text

This wat is possible Only if x and y are strings! wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True

Slide 114

Slide 114 text

The max of the string 'aa' is 'a' wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True >>> max('aa') 'a'

Slide 115

Slide 115 text

Again, lexiographic ordering. 'dog' < 'dogfish' wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True >>> max('aa') 'a' >>> 'aa' > 'a' True

Slide 116

Slide 116 text

Also, python handles 'in' differently for strings For strings, it performs a substring search wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True >>> max('aa') 'a' >>> 'aa' > 'a' True >>> 'aa' in 'aa' True

Slide 117

Slide 117 text

Where with lists, it's comparing 'aa' to each individual element wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True >>> max('aa') 'a' >>> 'aa' > 'a' True >>> 'aa' in ['a', 'a'] False

Slide 118

Slide 118 text

These wats may seem really technically interesting But I can almost guarantee that you'll never encounter these in the wild In fact, I've been writing Python for a long time, and I've only seen one of these The only reason I have them to share with you is because a smart fellow named Christopher Night collected a bunch of them together in a repo somewhere Which is why I must implore you, please One last thing...

Slide 119

Slide 119 text

Again, thanks to Promptworks Thanks to Christopher Night for introducing me to these wats Thanks to PyGotham for this event And thanks to you all for listening! Thanks! https://github.com/di/talks/pygotham_2016/

Slide 120

Slide 120 text

Questions?

Slide 121

Slide 121 text

wat #0 - Possible! >>> x = 0*1e309 >>> x == x False

Slide 122

Slide 122 text

wat #1 - Not Possible >>> x = ... >>> a = ... >>> b = ... >>> c = ... >>> max(x) < max(x[a:b:c]) True

Slide 123

Slide 123 text

wat #2 - Possible! >>> x = {0} >>> y = {1} >>> min(x, y) == min(y, x) False

Slide 124

Slide 124 text

wat #3 - Not Possible >>> x = ... >>> y = ... >>> any(x) and not any(x + y) True

Slide 125

Slide 125 text

wat #4 - Possible! >>> x = 'foobar' >>> y = '' >>> x.count(y) > len(x) True

Slide 126

Slide 126 text

wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False

Slide 127

Slide 127 text

wat #5 >>> x = 5e-234 >>> y = 3 >>> z = 9007199254740993 >>> x * (y * z) == (x * y) * z False

Slide 128

Slide 128 text

wat #6 - Possible! >>> x = '' >>> y = 'foobar' >>> x < y and all(a >= b for a, b in zip(x, y)) True

Slide 129

Slide 129 text

wat #7 - Not Possible >>> x = ... >>> len(set(list(x))) == len(list(set(x))) False