Wat‽ Mind-bending Edge Cases in Python

PyGotham 2016 Talk (July 19, 2016)

https://www.youtube.com/watch?v=qjk2pWY4RP0
https://2016.pygotham.org/talks/363/wat-mind-bending-edge-cas/

Transcript

1. I'm Dustin http://github.com/di
2. I work at PromptWorks who I'd like to thank for sponsoring my work in the open source community as well as this talk, which I call
3. So we're going to talk about some wats in Python You might ask "what is a wat?" wats are not trick questions wats are not bugs in the language A 'wat' is an edge-case in a language that makes you say: wat‽ Mind-Bending Edge Cases (in Python)
4. they seem weird, but if you understand why they happen, they make sense we're going to look at ten different wats but to make things interesting, a few are actually impossible and you'll need to decide if they're real or not wat‽
5. Let's start with an example to get started wat #0
6. The first goal here is to determine if this is possible or not If it is possible, what can we replace the ellipsis with To give us the desired result The missing values in these wats are limited to the built-in primitive types and collections booleans, integers, strings as well as collections like lists and sets, and combinations of these No lambdas, partials, classes, or other tricky wat #0 >>> x = ... >>> x == x False
8. so if x is equal to wat #0 >>> x = ... >>> x == x False
9. zero times one times ten raised to the 309th power wat #0 - Possible! >>> x = 0*1e309 >>> x == x False
10. this is because 0*1e309 is interpreted by python as Not A Number wat #0 - Possible! >>> x = 0*1e309 >>> x == x False >>> x nan
11. This is the same thing as zero times infinity wat #0 - Possible! >>> x = 0*1e309 >>> x == x False >>> x nan >>> 0*float('inf') nan
12. Or just float nan wat #0 - Possible! >>> x = 0*1e309 >>> x == x False >>> x nan >>> 0*float('inf') nan >>> float('nan') nan
13. A brief word on NaN There's a good reason why nan is not equal to itself, or anything else It's because it's not a number! NaN is designed to propagate through all calculations, so if somewhere in your deep, complex calculations you hit upon a NaN, you don't bubble out a seemingly sensible answer. So because of this, NaN is definitely the source of many wats NaN
14. In fact, it's the star of Gary Bernhardt's infamous talk which is definitely inspiration for this talk A brief word on NaN This is not Python: > Array(16).join("wat" - 1) + " Batman!" "NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN Batman!" https://www.destroyallsoftware.com/talks/wat
15. There are a number of sensible ways to generate a NaN in Python None of them are equal to themselves For the purposes of this talk, none of the "solutions" to the wats involve using a NaN A brief word on NaN >>> 0*1e309 nan >>> float('nan') nan >>> from decimal import Decimal; Decimal('nan') Decimal('NaN') >>> complex('nan') (nan+0j)
16. wat #1
17. can we create a list x such that when sliced by a, b, and c it has a new max wat #1 >>> x = ... >>> a = ... >>> b = ... >>> c = ... >>> max(x) < max(x[a:b:c]) True
19. This wat is impossible This will always compare the list with some subset of the list no matter how you slice it wat #1 - Not Possible >>> x = ... >>> a = ... >>> b = ... >>> c = ... >>> max(x) < max(x[a:b:c]) True
20. wat #2
21. Can we make x and y such that min(x, y) is different than min(y, x)? This wat is possible wat #2 >>> x = ... >>> y = ... >>> min(x, y) == min(y, x) False
23. wat #2 - Possible! >>> x = ... >>> y = ... >>> min(x, y) == min(y, x) False
24. if x is the set containing zero wat #2 - Possible! >>> x = {0} >>> y = ... >>> min(x, y) == min(y, x) False
25. and y is the set containing anything but zero Only if x and y are sets, though. Why? wat #2 - Possible! >>> x = {0} >>> y = {1} >>> min(x, y) == min(y, x) False
26. If we take the min of the set containing zero, and the set containing one We get the set containing zero wat #2 - Possible! >>> min({0}, {1}) set([0])
27. If we do the opposite, we get the set containing one! Seems like min is broken and just returning the first element wat #2 - Possible! >>> min({0}, {1}) set([0]) >>> min({1}, {0}) set([1])
28. Well, maybe not. What's happening? Let's imagine what the min function might look like If we implemented it in python wat #2 - Possible! >>> min({0}, {1}) set([0]) >>> min({1}, {0}) set([1]) >>> min({0, 1}, {0}) set([0])
29. The min function can take any number of arguments It finds the min of all of them It can also take just one, an iterator, but let's ignore that for now wat #2 - Possible! >>> def min(*args): ...
30. We need two variables hasitem tells us if minitem has been set yet min_item holds the smallest item found at any point wat #2 - Possible! >>> def min(*args): ... has_item = False ... min_item = None ...
31. We'll iterate through the arguments wat #2 - Possible! >>> def min(*args): ... has_item = False ... min_item = None ... for x in args: ...
32. If we haven't set minitem yet, or if x is less than minitem wat #2 - Possible! >>> def min(*args): ... has_item = False ... min_item = None ... for x in args: ... if not has_item or x < min_item: ...
33. We set hasitem to true and set minitem to x wat #2 - Possible! >>> def min(*args): ... has_item = False ... min_item = None ... for x in args: ... if not has_item or x < min_item: ... has_item = True ... min_item = x ...
34. Finally we return the smallest item found This is a really simplified approach -- doesn't handle no args, etc. What's the key? It's that less than operator comparing x to min_item wat #2 - Possible! >>> def min(*args): ... has_item = False ... min_item = None ... for x in args: ... if not has_item or x < min_item: ... has_item = True ... min_item = x ... return min_item ...
35. The less than operator works as you'd expect for something like integers wat #2 - Possible! >>> 0 < 1 True
36. But the operator is overloaded for set comparison Meaning it behaves differently for two sets than it would for two ints wat #2 - Possible! >>> 0 < 1 True >>> {0} < {1} False
37. Specifically, it is the inclusion operator Here, it's checking if what's in the set on the left is included in the set on the right wat #2 - Possible! >>> 0 < 1 True >>> {0} < {1} False >>> {0} < {0, 1} True
38. So when we call the min on these two one-element sets We will always get the first argument back, regardless wat #2 - Possible! >>> 0 < 1 True >>> {0} < {1} False >>> {0} < {0, 1} True >>> min({0}, {1}) set([0])
39. wat #3
40. Can we create two lists x and y such that at least one element in x is true But when appended with y, none of them are true? This wat is impossible wat #3 >>> x = ... >>> y = ... >>> any(x) and not any(x + y) True
42. This wat is impossible The elements of y have no effect on those in x If anything in x is true, something in x+y will be true as well wat #3 - Not Possible >>> x = ... >>> y = ... >>> any(x) and not any(x + y) True
43. wat #4
44. This wat is possible wat #4 >>> x = ... >>> y = ... >>> x.count(y) > len(x) True
46. This wat is possible wat #4 - Possible! >>> x = ... >>> y = ... >>> x.count(y) > len(x) True
47. If x is any string wat #4 - Possible! >>> x = 'foobar' >>> y = ... >>> x.count(y) > len(x) True
48. But only if y is an empty string wat #4 - Possible! >>> x = 'foobar' >>> y = '' >>> x.count(y) > len(x) True
49. wat #4 - Possible! >>> x = 'foobar' >>> y = '' >>> x.count(y) > len(x) True >>> len('foobar') 6
50. Let's imagine what the count function might look like wat #4 - Possible! >>> x = 'foobar' >>> y = '' >>> x.count(y) > len(x) True >>> len('foobar') 6 >>> 'foobar'.count('') 7
51. To make things easy let's just make a function that takes a string s and sub wat #4 - Possible! >>> def count(s, sub): ...
52. We initialize the result to return as zero wat #4 - Possible! >>> def count(s, sub): ... result = 0 ...
53. We iterate through all the indexes at which sub could start in s wat #4 - Possible! >>> def count(s, sub): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ...
54. If the substring is larger than the string, we get an empty list of indexes wat #4 - Possible! >>> def count(s='foo', sub='foobar'): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... # range(3 + 1 - 6) ... # range(-2) ... # []
55. If the substring is the same size as string, we get one index, [0] wat #4 - Possible! >>> def count(s='foobar', sub='foobar'): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... # range(6 + 1 - 6) ... # range(1) ... # [0]
56. If the substring is smaller than the string, we get possible start indexes wat #4 - Possible! >>> def count(s='foobar', sub='foo'): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... # range(6 + 1 - 3) ... # range(4) ... # [0, 1, 2, 3]
57. But if substring is empty string, we get one more index than the length wat #4 - Possible! >>> def count(s='foobar', sub=''): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... # range(6 + 1 - 0) ... # range(7) ... # [0, 1, 2, 3, 4, 5, 6]
58. Using the index, we get a slice of the string as our possible match wat #4 - Possible! >>> def count(s, sub): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ...
59. When the substring has a length, this gives us a slice the same size wat #4 - Possible! >>> def count(s='foobar', sub='foo'): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ... # s[0:0 + 3] ... # s[0:3] ... # 'foo'
60. When it's empty though, we just get another empty string wat #4 - Possible! >>> def count(s='foobar', sub=''): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ... # s[0:0 + 0] ... # s[0:0] ... # ''
61. Slicing also won't raise an IndexError when the index is longer than the string wat #4 - Possible! >>> def count(s='foobar', sub=''): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ... # s[6:6 + 0] ... # s[6:6] ... # ''
62. Finally if the possible match is the same as the substring We increment the result counter wat #4 - Possible! >>> def count(s, sub): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ... if possible_match == sub: ... result += 1 ... return result ...
63. For an empty string, the range is the indexes 0 through 6, for seven total And the possible_match matches every time! Giving us a count of 7 wat #4 - Possible! >>> def count(s, sub): ... result = 0 ... for i in range(len(s) + 1 - len(sub)): ... possible_match = s[i:i + len(sub)] ... if possible_match == sub: ... result += 1 ... return result ... >>> count('foobar', '') 7
64. wat #5
65. wat #5 >>> x = ... >>> y = ... >>> z = ... >>> x * (y * z) == (x * y) * z False
67. wat #5 - Possible! >>> x = ... >>> y = ... >>> z = ... >>> x * (y * z) == (x * y) * z False
68. wat #5 - Possible! >>> x = [0] >>> y = ... >>> z = ... >>> x * (y * z) == (x * y) * z False
69. wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = ... >>> x * (y * z) == (x * y) * z False
70. wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False
71. wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1)
72. A list times one is itself wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1) == [0]*1
73. A list times one is itself This makes sense wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1) == [0]*1 == [0] True
74. What happens when we multiple a list by a negative number? wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1) == [0]*1 == [0] True >>> (x * y) * z == ([0]*-1)*-1
75. Values of n less than 0 are treated as 0 which yields an empty sequence of the same type as s. in this case, an empty list wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1) == [0]*1 == [0] True >>> (x * y) * z == ([0]*-1)*-1 == []*-1
76. Again, an empty list wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) == [0]*(-1*-1) == [0]*1 == [0] True >>> (x * y) * z == ([0]*-1)*-1 == []*-1 == [] True
77. There are actually other ways to make this wat possible as well for example wat #5 - Possible! >>> x = ... >>> y = ... >>> z = ... >>> x * (y * z) == (x * y) * z False
78. if we set x, y and z to these mysterious values wat #5 - Possible! >>> x = 5e-234 >>> y = 3 >>> z = 9007199254740993 >>> x * (y * z) == (x * y) * z False
79. we get this value for the first half wat #5 - Possible! >>> x = 5e-234 >>> y = 3 >>> z = 9007199254740993 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) 1.335044315104321e-307
80. and this one for the second half this is due to the way floats are, by their nature, imprecise basically the two results are off by the difference of the least significant bit wat #5 - Possible! >>> x = 5e-234 >>> y = 3 >>> z = 9007199254740993 >>> x * (y * z) == (x * y) * z False >>> x * (y * z) 1.335044315104321e-307 >>> (x * y) * z 1.3350443151043208e-307
81. wat #6
82. Can we define two lists x and y such that x < y But when we compare each individual element, every element in x > y This wat is possible wat #6 >>> x = ... >>> y = ... >>> x < y and all(a >= b for a, b in zip(x, y)) True
84. wat #6 - Possible! >>> x = ... >>> y = ... >>> x < y and all(a >= b for a, b in zip(x, y)) True
85. True for any interables where x is zero-length wat #6 - Possible! >>> x = '' >>> y = ... >>> x < y and all(a >= b for a, b in zip(x, y)) True
86. And y is any iterable wat #6 - Possible! >>> x = '' >>> y = 'foobar' >>> x < y and all(a >= b for a, b in zip(x, y)) True
87. Comparison of sequences uses lexicographical ordering Basically alphabetization. 'dog' comes before 'dogfish' Empty list comes before non empty lists wat #6 - Possible! >>> x = '' >>> y = 'foobar' >>> x < y and all(a >= b for a, b in zip(x, y)) True >>> '' < 'foobar' True
88. Zipping two lists of uneven length will give a list of the shorter length wat #6 - Possible! >>> x = '' >>> y = 'foobar' >>> x < y and all(a >= b for a, b in zip(x, y)) True >>> '' < 'foobar' True >>> zip('', 'foobar') []
89. all of an empty list is true! all short-circuits -- if anything is false return false, otherwise true wat #6 - Possible! >>> x = '' >>> y = 'foobar' >>> x < y and all(a >= b for a, b in zip(x, y)) True >>> '' < 'foobar' True >>> zip('', 'foobar') [] >>> all([]) True
90. wat #7
91. This wat is not possible. wat #7 >>> x = ... >>> len(set(list(x))) == len(list(set(x))) False
93. This wat is not possible. Converting a list to a set might reduce the length of x But converting a set to a list will never add elementes wat #7 - Not Possible >>> x = ... >>> len(set(list(x))) == len(list(set(x))) False
94. wat #8
95. Can we make x such that min(x) is not the same as min(x unpacked) This wat is possible wat #8 >>> x = ... >>> min(x) == min(*x) False
97. wat #8 - Possible! >>> x = ... >>> min(x) == min(*x) False
98. Remember earlier when I said min could take either a series of arguments Or just an iterable? This wat exists because of the different ways min handles its args wat #8 - Possible! >>> x = [[0]] >>> min(x) == min(*x) False
99. These are all the same wat #8 - Possible! >>> x = [[0]] >>> min(x) == min(*x) False >>> min([1, 2, 3]) == min(*[1, 2, 3]) == min(1, 2, 3) True
100. So the min of x is just the first element in it wat #8 - Possible! >>> x = [[0]] >>> min(x) == min(*x) False >>> min([1, 2, 3]) == min(*[1, 2, 3]) == min(1, 2, 3) True >>> min(x) == [0] True
101. But the min of x unpacked is the min of the list containing zero Which is just zero wat #8 - Possible! >>> x = [[0]] >>> min(x) == min(*x) False >>> min([1, 2, 3]) == min(*[1, 2, 3]) == min(1, 2, 3) True >>> min(x) == [0] True >>> min(*x) == min([0]) == 0 True
102. wat #9
103. This wat is impossible wat #9 >>> x = ... >>> y = ... >>> sum(0 * x, y) == y False
105. This wat is impossible Anything times zero is either zero or an empty sequence The sum of zero or an empty sequence is always zero wat #9 - Not Possible >>> x = ... >>> y = ... >>> sum(0 * x, y) == y False
106. Here, y is the "start" of the sum wat #9 - Not Possible >>> x = ... >>> y = ... >>> sum(0 * x, y) == y False >>> sum([1, 1, 1], 7) 10
107. When the sequence is empty we just get the start value wat #9 - Not Possible >>> x = ... >>> y = ... >>> sum(0 * x, y) == y False >>> sum([1, 1, 1], 7) 10 >>> sum([], 7) 7
108. wat #10
109. Can we find two values x and y such that y > max(x), but y is also in x This wat is possible wat #10 >>> x = ... >>> y = ... >>> y > max(x) and y in x True
111. wat #10 - Possible! >>> x = ... >>> y = ... >>> y > max(x) and y in x True
112. wat #10 - Possible! >>> x = 'aa' >>> y = ... >>> y > max(x) and y in x True
113. This wat is possible Only if x and y are strings! wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True
114. The max of the string 'aa' is 'a' wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True >>> max('aa') 'a'
115. Again, lexiographic ordering. 'dog' < 'dogfish' wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True >>> max('aa') 'a' >>> 'aa' > 'a' True
116. Also, python handles 'in' differently for strings For strings, it performs a substring search wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True >>> max('aa') 'a' >>> 'aa' > 'a' True >>> 'aa' in 'aa' True
117. Where with lists, it's comparing 'aa' to each individual element wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True >>> max('aa') 'a' >>> 'aa' > 'a' True >>> 'aa' in ['a', 'a'] False
118. These wats may seem really technically interesting But I can almost guarantee that you'll never encounter these in the wild In fact, I've been writing Python for a long time, and I've only seen one of these The only reason I have them to share with you is because a smart fellow named Christopher Night collected a bunch of them together in a repo somewhere Which is why I must implore you, please One last thing...
119. Again, thanks to Promptworks Thanks to Christopher Night for introducing me to these wats Thanks to PyGotham for this event And thanks to you all for listening! Thanks! https://github.com/di/talks/pygotham_2016/
120. Questions?
121. wat #0 - Possible! >>> x = 0*1e309 >>> x == x False
122. wat #1 - Not Possible >>> x = ... >>> a = ... >>> b = ... >>> c = ... >>> max(x) < max(x[a:b:c]) True
123. wat #2 - Possible! >>> x = {0} >>> y = {1} >>> min(x, y) == min(y, x) False
124. wat #3 - Not Possible >>> x = ... >>> y = ... >>> any(x) and not any(x + y) True
125. wat #4 - Possible! >>> x = 'foobar' >>> y = '' >>> x.count(y) > len(x) True
126. wat #5 - Possible! >>> x = [0] >>> y = -1 >>> z = -1 >>> x * (y * z) == (x * y) * z False
127. wat #5 >>> x = 5e-234 >>> y = 3 >>> z = 9007199254740993 >>> x * (y * z) == (x * y) * z False
128. wat #6 - Possible! >>> x = '' >>> y = 'foobar' >>> x < y and all(a >= b for a, b in zip(x, y)) True
129. wat #7 - Not Possible >>> x = ... >>> len(set(list(x))) == len(list(set(x))) False
130. wat #8 - Possible! >>> x = [[0]] >>> min(x) == min(*x) False
131. wat #9 - Not Possible >>> x = ... >>> y = ... >>> sum(0 * x, y) == y False
132. wat #10 - Possible! >>> x = 'aa' >>> y = 'aa' >>> y > max(x) and y in x True