Tutorial: Python Epiphanies by Stuart Williams

This tutorial is for software developers who've been using Python with success for a while but are looking for a deeper understanding of the language. It focuses on how Python differs from other languages in subtle but important ways that often confuse folks, and it demystifies a number of language features that are sometimes misunderstood.

The deck is a 31 page handout of exercises which includes the text of all the slides as paragraphs.

Transcript

1. Python Epiphanies Overview This tutorial, presented at PyCon 2013 in San Jose by Stuart Williams (stuart@swilliams.ca), is intended for In- termediate Python users looking for a deeper understanding of the language. It attempts to correct some common misperceptions of how Python works. Python is very similar to other programming languages, but quite different in some subtle but important ways. You’ll learn by seeing and doing. We’ll mostly use the interactive Python interpreter prompt, aka the Read Eval Print Loop (REPL). I’ll be using Python 2.7 but most of this will work identically in 3.x and I’ll point out where there are differences. Most exercise sections start out simple but increase quickly in difficulty in order to give more advanced students a challenge. Expect to be hurried along well before everyone has completed the entire section! I am only providing a printed copy of these exercises during the tutorial because by typing them yourselves you will learn more. If you’re working on this handout without having been in the tutorial, you’ll get more out of it if you watch the video of the tutorial which has many explanations that aren’t in the handout. License: This PyCon 2013 Python Epiphanies Tutorial by Stuart Williams is licensed under a Creative Commons Attribution-Share Alike 2.5 Canada License (http://creativecommons.org/licenses/by-sa/2.5/ca/). Dictionaries and Namespaces >>> month_number_to_name = [None, ’Jan’, ’Feb’, ’Mar’] 0 >>> month_number_to_name[1] # month 1 is January 1 >>> month_number_to_name[2] # month 2 is February 2 >>> month_name_to_number = {’Jan’: 1, ’Feb’: 2, ’Mar’: 3} 3 >>> month_name_to_number[’Jan’] # January is month 1 4 >>> month_name_to_number[’Feb’] # February is month 2 5 >>> my_namespace = {} 6 >>> my_namespace 7 >>> my_namespace[’a’] = 7 8 >>> my_namespace 9 >>> my_namespace[’a’] 10 >>> my_namespace[’a’] = 8 11 >>> my_namespace 12 >>> my_namespace[’a’] 13 1
2. >>> my_namespace[’s’] = ’March’ 14 >>> my_namespace 15 >>> my_namespace[’s’] 16 >>> del my_namespace[’a’] 17 >>> my_namespace 18 >>> my_namespace[’a’] 19 >>> dir() 20 >>> a = 7 21 >>> dir() 22 >>> a 23 >>> a = 8 24 >>> a 25 >>> s = ’March’ 26 >>> dir() 27 >>> s 28 >>> del a 29 >>> dir() 30 >>> a 31 >>> dir() 32 >>> my_namespace = {} 33 >>> my_namespace[’r’] = {} 34 >>> my_namespace[’r’][’x’] = 1.0 35 >>> my_namespace[’r’][’y’] = 2.0 36 >>> my_namespace[’r’][’x’] 37 >>> my_namespace[’r’] 38 >>> my_namespace 39 >>> class Record: 40 ... pass >>> r = Record 41 >>> r.x = 1.0 42 >>> r.y = 1.0 43 >>> r.x 44 >>> r.y 45 >>> r 46 >>> dir() 47 Objects Everything in Python is an object and has: ∙ a single value, ∙ a single type, ∙ some number of attributes ∙ a single id, 2
3. ∙ (zero or) one or more names (in one or more namespaces), ∙ and usually (indirectly), one or more base classes. A single value: >>> 1 48 >>> 1.0 49 >>> ’hello’ 50 >>> (1, 2, 3) 51 >>> [1, 2, 3] 52 A single type: >>> type(1) 53 >>> type(1.0) 54 >>> type(’hello’) 55 >>> type((1, 2, 3)) 56 >>> type([1, 2, 3]) 57 Some number of attributes: >>> dir(1) 58 >>> 1.__doc__ 59 >>> (1).__doc__ 60 >>> (1).__class__ 61 >>> (1.0).__class__ 62 >>> 1.0.__class__ 63 >>> ’hello’.__class__ 64 >>> ’mississippi’.count 65 >>> ’mississippi’.count(’s’) 66 >>> (1, 2, 3).index 67 >>> [1, 2, 3].pop 68 A single id: >>> id(1) 69 >>> id(1.0) 70 >>> id(’hello’) 71 >>> id((1, 2, 3)) 72 >>> id([1, 2, 3]) 73 Base classes: >>> import inspect 74 >>> inspect.getmro(type(’hello’)) 75 3
4. >>> ’hello’.__class__ 76 >>> type(’hello’) is ’hello’.__class__ is str 77 >>> ’hello’.__class__.__bases__ 78 >>> ’hello’.__class__.__bases__[0] 79 >>> ’hello’.__class__.__bases__[0].__bases__ 80 >>> inspect.getmro(type(’hello’)) 81 Exercises: Namespaces and Objects Restart Python to unclutter the local namespace. >>> dir() 82 >>> i = 1 83 >>> i 84 >>> dir() 85 >>> type(i) 86 >>> id(i) 87 >>> j = 1 88 >>> dir() 89 >>> id(j) 90 >>> id(i) == id(j) 91 >>> i is j 92 >>> m = [1, 2, 3] 93 >>> m 94 >>> n = m 95 >>> n 96 >>> dir() 97 >>> m is n 98 >>> m[1] = ’two’ 99 >>> m 100 >>> n 101 >>> s = ’hello’ 102 >>> s 103 >>> id(s) 104 >>> s += ’ there’ 105 >>> s 106 >>> id(s) 107 >>> m = [1, 2, 3] 108 >>> m 109 >>> id(m) 110 >>> m += [4] 111 >>> m 112 >>> id(m) 113 4
5. >>> int.__div__ 114 >>> int.__div__ == int.__truediv__ 115 >>> int.__div__ = int.__truediv__ 116 >>> dir(None) 117 >>> dir(None) == dir(object) 118 >>> dir(None) == dir(object()) 119 Namespaces A namespace is a mapping from valid identifier names to objects. Think of it as a dictionary. Assignment is a namespace operation, not an operation on variables or objects. A scope is a section of Python code where a namespace is directly accessible. For a directly accessible namespace you access values in the (namespace) dictionary by key alone, e.g. s instead of my_namespace[’s’]. For indirectly accessible namespace you access values via dot notation, e.g. dict.__doc__ or sys.version.major. The (direct) namespace search order is (from http://docs.python.org/tutorial): ∙ First: the innermost scope contains local names ∙ Second: the namespaces of enclosing functions, searched starting with the nearest enclosing scope; (or the module if outside any function) ∙ Third: the middle scope contains the current module’s global names ∙ Fourth: the outermost scope is the namespace containing built-in names All namespace changes happen in the local scope (i.e. in the current scope in which the namespace-changing code executes): ∙ assignment ∙ import ∙ def ∙ class ∙ del >>> len 120 >>> def f1(): 121 ... def len(): ... len = range(3) ... print("In f1’s len(), len = {}".format(len)) ... return len ... print(’In f1(), len = {}’.format(len)) ... return len() 5
6. >>> f1() 122 >>> def f2(): 123 ... def len(): ... print("In f2’s len(), len = {}".format(len)) ... return "From f2’s len()" ... print(’In f2(), len = {}’.format(len)) ... return len() >>> f2() 124 >>> len 125 >>> len = 99 126 >>> len 127 >>> del len 128 >>> len 129 >>> pass 130 >>> pass = 3 131 >>> del 132 Let’s look at some surprising behaviour: >>> x = 1 133 >>> def test1(): 134 ... print(’In test1 x == {}’.format(x)) >>> test1() 135 >>> def test2(): 136 ... x = 2 ... print(’In test2 x == {}’.format(x)) >>> x 137 >>> test2() 138 >>> x 139 >>> def test3(): 140 ... print(’In test3 == {}’.format(x)) ... x = 3 >>> x 141 >>> test3() 142 >>> x 143 >>> test1.func_code.co_varnames 144 >>> test3.func_code.co_varnames 145 6
7. >>> def test4(): 146 ... global x ... print(’In test4 before, x == {}’.format(x)) ... x = 4 ... print(’In test4 after, x == {}’.format(x)) >>> x 147 >>> test4() 148 >>> x 149 >>> test4.func_code.co_varnames 150 “If a name is declared global, then all references and assignments go directly to the middle scope containing the module’s global names. Otherwise, all variables found outside of the innermost scope are read-only (an attempt to write to such a variable will simply create a new local variable in the innermost scope, leaving the identically named outer variable unchanged).” [Python tutorial section 9.2 at http://docs.python.org/tutorial] The Local Namespace >>> help(dir) 151 >>> dir() 152 >>> __my_builtins__ = __builtins__ 153 >>> # Workaround a subtlety with code.interactive used by my demo script 154 >>> __my_builtins__ = __import__(’__builtin__’) 155 >>> from textwrap import fill 156 >>> def is_exception(s): 157 ... return any( ... [s in name for s in ... ’Error’, ’Warning’, ’Exception’]) >>> print(fill(’ ’.join( 158 ... [name for name in dir(__my_builtins__) ... if is_exception(name)]), 60)) >>> not_exceptions = [name for name in dir(__my_builtins__) 159 >>> if not is_exception(name)] 160 >>> print(fill(’ ’.join(not_exceptions), 60)) 161 >>> import collections 162 >>> bnames = collections.defaultdict(set) 163 >>> for name in not_exceptions: 164 ... bnames[type(getattr(__my_builtins__, name))].add(name) 7
8. >>> for typ in [type(type), type(all)]: 165 ... names = bnames.pop(typ) ... print(typ) ... print(fill(’ ’.join(names), 60)) ... print() >>> for typ in bnames.keys(): 166 ... names = bnames.pop(typ) ... print(typ, ’ ’.join(names)) Exercises: The Local Namespace >>> locals().keys() 167 >>> globals().keys() 168 >>> locals() == globals() 169 >>> locals() is globals() 170 >>> x 171 >>> locals()[’x’] 172 >>> locals()[’x’] = 1 173 >>> locals()[’x’] 174 >>> x 175 >>> dir() 176 Most builtins are unsurprising cases of type exception, type built-in function, or type. Explore some of the following suprising ones via introspection (e.g. type, inspect.getmro, and help) or the Python documen- tation: ∙ bytes ∙ enumerate, reversed ∙ exit, help, license, quit ∙ True, False, None, NotImplemented, Ellipsis >>> import inspect 177 >>> inspect.getmro(type(True)) 178 >>> isinstance(True, int) 179 Namespace Changes These change or modify a namespace: ∙ assignment 8
9. ∙ [globals() and locals()] ∙ import ∙ def ∙ class ∙ del Next we’ll explore the last three. >>> dir() 180 >>> import pprint 181 >>> dir() 182 >>> pprint 183 >>> dir(pprint) 184 >>> print(’\n’.join([a for a in dir(pprint) 185 >>> if not a.startswith(’_’)])) >>> pprint.pformat 187 >>> pprint.pprint 188 >>> pprint.foo 189 >>> foo 190 >>> from pprint import pprint as pprint_pprint_function 191 >>> dir() 192 >>> pprint.pprint is pprint_pprint_function 193 >>> pprint 194 >>> pprint.pformat 195 >>> del pprint 196 >>> import pprint as pprint_module 197 >>> dir() 198 >>> pprint_module.pprint is pprint_pprint_function 199 >>> module_name = ’string’ 200 >>> string_module = __import__(module_name) 201 >>> string_module.uppercase 202 >>> string 203 >>> import ’string’ 204 >>> import string 205 File structure: 9
10. folder1/ file1.py module1/ __init__.py -- zero length file1.py: attribute1 = 1 >>> dir() 206 >>> import folder1 207 >>> import folder1.file1 208 >>> import module1 209 >>> dir() 210 >>> dir(module1) 211 >>> import module1.file1 212 >>> dir() 213 >>> dir(module1) 214 >>> dir(module1.file1) 215 >>> from module1 import file1 216 >>> dir() 217 >>> dir(file1) 218 Exercise: The import statement >>> import pprint 219 >>> dir(pprint) 220 >>> pprint.__doc__ 221 >>> pprint.__file__ 222 >>> pprint.__name__ 223 >>> pprint.__package__ 224 >>> from pprint import * 225 >>> dir() 226 >>> reload(csv) 227 >>> reload(’csv’) 228 >>> import csv 229 >>> reload(’csv’) 230 >>> reload(csv) 231 >>> import sys 232 >>> sys.path 233 Functions >>> def f(arg1, arg2, kwarg1=’kw1’, kwarg2=’kw2’, 234 10
11. ... *args, **kwargs): ... """ ... A function with regular and keyword arguments. ... """ ... print(’arg1: {0}, arg2: {1}, ’ ... ’kwarg1: {2}, kwarg2: {3}’ ... .format(arg1, arg2, kwarg1, kwarg2)) ... print(’args:’, repr(args)) ... print(’kwargs:’, repr(kwargs)) >>> f.__name__ 235 >>> dir() 236 >>> f.__name__ = ’g’ 237 >>> dir() 238 >>> f.__name__ 239 >>> f 240 >>> f.func_dict 241 >>> f.foo = ’bar’ 242 >>> f.func_dict 243 >>> f.func_defaults 244 >>> f(1, 2) 245 >>> f(arg1=1, arg2=2) 246 >>> f(arg2=1, arg1=2) 247 >>> f(1, 2, 3) 248 >>> f(1, 2, kwarg2=4) 249 >>> f(1, kwarg1=3) 250 >>> f(1, 2, 3, 4, 5, 6) 251 >>> f(1, 2, 3, 4, keya=7, keyb=8) 252 >>> f(1, 2, 3, 4, 5, 6, keya=7, keyb=8) 253 Exercises: Functions >>> def f(a1, a2, kw1=’k1’, kw2=’k2’): 254 ... print(repr((a1, a2, kw1, kw2))) >>> f(1) 255 >>> f(1, 2) 256 >>> f(1, 2, 3) 257 >>> t = 1, 2 258 >>> t 259 >>> d = dict(kw1=3, kw2=4) 260 >>> d 261 >>> f(*t) 262 >>> f(**d) 263 >>> f(1, 2, **d) 264 >>> f(*t, **d) 265 11
12. >>> locals() 266 >>> name = ’Dad’ 267 >>> ’Hi {name}’.format(**locals()) 268 Lists are mutable, strings are not >>> # First with ‘‘=‘‘ and ‘‘+‘‘, then with ‘‘+=‘‘: 269 >>> s1 = s2 = ’hello’ 270 >>> s1 = s1 + ’ there’ 271 >>> s1, s2 272 >>> s1 = s2 = ’hello’ 273 >>> s1 += ’ there’ 274 >>> s1, s2 275 >>> m1 = m2 = [1, 2, 3] 276 >>> m1 = m1 + [4] 277 >>> m1, m2 278 >>> m1 = m2 = [1, 2, 3] 279 >>> m1 += [4] 280 >>> m1, m2 281 >>> # Why? 282 >>> # foo += 1 is not just syntactic sugar for foo = foo + 1 283 >>> import codeop, dis 284 >>> dis.dis(codeop.compile_command(’m = [1, 2, 3]; m += [4]’)) 285 >>> dis.dis(codeop.compile_command("s = ’hello’; s += ’ there’")) 286 >>> m = [1, 2, 3] 287 >>> m 288 >>> m.__iadd__([4]) 289 >>> m 290 The difference is because str.__iadd__ copies, but list.__iadd__ mutates. http://docs.python.org/2/reference/datamodel.html?highlight=__iadd__#object.__iadd__ “These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self). Parameters are always passed by reference. 12
13. >>> def test1(s): 291 ... print(’Before:’, s) ... s += ’ there’ ... print(’After:’, s) >>> str2 = ’hello’ 292 >>> str2 293 >>> test1(str2) 294 >>> str2 295 >>> test1(’hello’) 296 >>> def test2(m): 297 ... print(’Before:’, m) ... m += [4] ... print(’After:’, m) >>> list3 = [1, 2, 3] 298 >>> list3 299 >>> test2(list3) 300 >>> list3 301 Decorators >>> def square(n): 302 ... return n * n >>> square(2) 303 >>> square(3) 304 >>> def identity(function): 305 ... def new_function(n): ... return function(n) ... return new_function >>> def call_str_on(function): 306 ... def new_function(n): ... return str(function(n)) ... return new_function >>> def call_str_on(function): 307 ... def new_function(n): ... print(’called new_function({})’ 13
14. ... .format(n)) ... return str(function(n)) ... print(’called call_str_on({})’ ... .format(function)) ... return new_function >>> square 308 >>> square_str = call_str_on(square) 309 >>> square_str 310 >>> square_str(3) 311 >>> @call_str_on 312 >>> def cube(n): 313 ... return n * n * n >>> cube(2) 314 Exercises: changing the local namespace A decorator is a function that takes a function as a parameter and typically returns a new function, but it can return anything. The following code misuses decorators. What does it do? Restart Python to unclutter the local namespace. >>> dir() 315 >>> x 316 >>> def call(f): 317 ... value = f() ... return value >>> int 318 >>> int() 319 >>> call(int()) 320 >>> call(int) 321 >>> @call 322 >>> def x(): 323 ... return 3 >>> dir() 324 >>> x 325 >>> type(x) 326 14
15. Here’s equivalent code without using @decorator syntax: >>> del x 327 >>> x 328 >>> def x(): 329 ... return 1 >>> x 330 >>> x = call(x) 331 >>> x 332 The class statement Remember, everything in Python is an object and has: ∙ a single id, ∙ a single value, ∙ some number of attributes (part of its value), ∙ a single type, ∙ (zero or) one or more names (in one or more namespaces), ∙ and usually (indirectly), one or more base classes. Many objects are instances of classes. The type of such an object is its class. Classes are instances of metaclasses. So the type of a class is its metaclass, i.e. type(type(anObject)) is a metaclass. Are classes and metaclasses objects? 1. The class statement, which starts a block of code, creates a new namespace and all the name assignments in the block, i.e. assignment and function definitions, are made in that new namespace. It also creates a name for the class in the namespace of the module where it appears. 2. Instances of a class are created by calling the class: ClassName() or ClassName(parameters). ClassName.__init__(<new object>, ...) is called automatically, passing as first parameter an object, the new instance of the class, which was created by a call to __new__(). 3. Accessing an attribute method_name on a class instance returns a method object, if method_name references a method (in ClassName or its superclasses). A method object binds the class instance as the first parameter to the method. >>> class Number(object): 333 ... """A number class.""" ... # This comment satisfies the REPL ... __version__ = ’1.0’ ... # ... def __init__(self, amount): ... self.amount = amount 15
16. ... # ... def add(self, value): ... """Add a value to the number.""" ... return self.amount + value >>> Number 334 >>> Number.__version__ 335 >>> Number.__doc__ 336 >>> help(Number) 337 >>> Number.__init__ 338 >>> Number.add 339 >>> dir(Number) 340 >>> number2 = Number(2) 341 >>> number2.amount 342 >>> number2 343 >>> number2.__init__ 344 >>> number2.add 345 >>> dir(number2) 346 >>> number2.__dict__ 347 >>> Number.__dict__ 348 >>> number2.add 349 >>> number2.add(3) 350 >>> Number.add 351 >>> Number.add(2) 352 >>> Number.add(2, 3) 353 >>> Number.add(number2, 3) 354 >>> number2.add(3) 355 >>> def set_double_amount(self, amount): 356 ... self.amount = 2 * amount >>> Number.__init__ 357 >>> help(Number.__init__) 358 >>> Number.__init__ = set_double_amount 359 >>> Number.__init__ 360 >>> help(Number.__init__) 361 >>> number4 = Number(2) 362 >>> number4.add(5) 363 >>> number2.__init__ 364 >>> def multiply_by(num, value): 365 ... return num.amount * value 16
17. >>> # Methods live in classes, not instances. 366 >>> # Let’s make a mistake. 367 >>> number4.mul = multiply_by 368 >>> number4.mul 369 >>> number4.mul(5) 370 >>> number4.mul(number4, 5) 371 >>> number10 = Number(5) 372 >>> number10.mul 373 >>> dir(number4) 374 >>> dir(Number) 375 >>> Number.mul = multiply_by 376 >>> number4.mul(5) 377 >>> number10.mul(5) 378 >>> dir(number4) 379 >>> number4.mul 380 >>> del number4.mul 381 >>> Number.mul 382 >>> number4.mul 383 >>> number4.mul(5) 384 >>> number4 385 >>> number4.mul 386 >>> dir(number4.mul) 387 >>> number4.mul.im_class 388 >>> number4.mul.__self__ 389 >>> number4.mul.__self__ is number4 390 >>> number4.mul.__self__ is number4.mul.im_self 391 >>> number4.mul.__func__ 392 >>> number4.mul.__func__ is multiply_by 393 >>> number4.mul.__func__ is number4.mul.im_func 394 >>> help(number4.mul.__func__) 395 >>> number4.mul(5) 396 >>> number4.mul.__func__(number4.mul.__self__, 5) 397 Exercises: The class statement Type in this class statement: >>> class Prefixer(object): 398 ... pass 17
18. Now at the interactive prompt, similar to the demonstration above, add the method(s) required to make the following code work, inserting ’-> ’ at the beginning of each string in the sequence passed to the prepend method. >>> arrow = Prefixer(’-> ’) 399 >>> assert (arrow.prepend([’line 1’, ’line 2’]) == 400 ... [’-> line 1’, ’-> line 2’]) The type function for classes The class statement is just a way to call a function, take the result, and put it into a namespace. -- Glyph Lefkowitz in ”Turtles All The Way Down...“ at PyCon 2010 type(name, bases, dict) is the function that gets called when a class statement is used to create a class. >>> print(type.__doc__) 401 >>> def __init__(self, amount): 402 ... self.amount = amount >>> def set_double_amount(self, amount): 403 ... self.amount = 2 * amount >>> DoubleNum = type( 404 ... ’DoubleNum’, ... (object,), ... { ’__init__’: set_double_amount, ... ’mul’: multiply_by, ... }) >>> number6 = DoubleNum(3) 405 >>> type(number6) 406 >>> number6.__class__ 407 >>> number6.__dict__ 408 >>> number6.amount 409 >>> number6.mul(4) 410 This dynamic call to type is what the class statement actually triggers. However, ”When the class definition is read, if __metaclass__ is defined then the object it references will be called instead of type().“ __metaclass__ ”can be any callable accepting arguments for name, bases, and dict. Upon class creation, the callable is used instead of the built-in type().“ [Language Reference section 3.4.3] 18
19. Exercise: The class statement What does the following code do? Use only one of the ”2.7“ and ”3.x“ definitions of class x. >>> def one(name, bases, attrs): 411 ... return 1 >>> one(None, None, None) 412 If you’re running Python 2.x: >>> class x(object): # Python 2.7 syntax 413 ... __metaclass__ = one # call this to create the class If you’re running Python 3.x: >>> class x(metaclass=one): # Python 3.x syntax 414 ... pass >>> x 415 We saw how decorators are applied to functions. They can also be applied to classes. What does the following code do? >>> def two(klass): 416 ... return 2 >>> two(None) 417 >>> @two 418 >>> class y(object): 419 ... pass >>> y 420 Standard class methods ∙ __new__, __init__, __del__, __repr__, __str__, __format__ ∙ __getattr__, __getattribute__, __setattr__, __delattr__, __call__, __dir__ ∙ __len__, __getitem__, __missing__, __setitem__, __delitem__, __contains__, __iter__ ∙ __lt__, __le__, __gt__, __ge__, __eq__, __ne__, __cmp__, __nonzero__, __hash__ 19
20. ∙ __add__, __sub__, __mul__, __div__, __floordiv__, __mod__, __divmod__, __pow__, __and__, __xor__, __or__, __lshift__, __rshift__, __neg__, __pos__, __abs__, __invert__, __iadd__, __isub__, __imul__, __idiv__, __itruediv__, __ifloordiv__, __imod__, __ipow__, __iand__, __ixor__, __ior__, __ilshift__, __irshift__ ∙ __int__, __long__, __float__, __complex__, __oct__, __hex__, __coerce__ ∙ __radd__, __rsub__, __rmul__, __rdiv__, etc. ∙ __enter__, __exit__, __next__ >>> class UpperAttr(object): 421 ... """ ... A class that returns uppercase values ... on uppercase attribute access. ... """ ... def __getattr__(self, name): ... if name.isupper(): ... if name.lower() in self.__dict__: ... return self.__dict__[ ... name.lower()].upper() ... raise AttributeError( ... "’{}’ object has no attribute {}." ... .format(self, name)) >>> d = UpperAttr() 422 >>> d.__dict__ 423 >>> d.foo = ’bar’ 424 >>> d.foo 425 >>> d.__dict__ 426 >>> d.FOO 427 >>> d.bar 428 Exercise: Standard class methods Try the following (in a file if that’s easier): >>> class Get(object): 429 ... def __getitem__(self, key): ... print(’called __getitem__({} {})’ ... .format(type(key), repr(key))) >>> g = Get() 430 >>> g[1] 431 >>> g[-1] 432 >>> g[0:3] 433 >>> g[0:10:2] 434 >>> g[’Jan’] 435 >>> g[g] 436 20
21. >>> m = list(’abcdefghij’) 437 >>> m[0] 438 >>> m[-1] 439 >>> m[::2] 440 >>> s = slice(3) 441 >>> m[s] 442 >>> m[slice(1, 3)] 443 >>> m[slice(0, 2)] 444 >>> m[slice(0, len(m), 2)] 445 >>> m[::2] 446 Properties >>> class PropertyExample(object): 447 ... def __init__(self): ... self._x = None ... def getx(self): ... print(’called getx()’) ... return self._x ... def setx(self, value): ... print(’called setx()’) ... self._x = value ... def delx(self): ... print(’del x’) ... del self._x ... x = property(getx, setx, delx, "The ’x’ property.") >>> p = PropertyExample() 448 >>> p.setx(’foo’) 449 >>> p.getx() 450 >>> p.x = ’bar’ 451 >>> p.x 452 >>> del p.x 453 Iterators ∙ A for loop evaluates an expression to get an iterable and then calls iter() to get an iterator. ∙ The iterator’s next() method is called repeatedly until StopIteration is raised. ∙ iter(foo) – checks for foo.__iter__() and calls it if it exists – else checks for foo.__getitem__(), calls it starting at zero, and handles IndexError by raising StopIteration. Here’s some rough inferred pseudo-code for what iter() does: 21
22. >>> def iter(foo): 454 ... """Pseudo code for what iter() does.""" ... if hasattr(foo, ’__iter__’) and callable(foo.__iter__): ... yield foo.__iter__() ... elif hasattr(foo, ’__getitem__’) and callable(foo.__getitem__): ... index = 0 ... while True: ... try: ... yield foo.__getitem__(index) ... except IndexError: ... raise StopIteration ... index += 1 ... else: ... raise TypeError(’{!r} object is not iterable’.format(type(foo))) ... >>> class MyList(object): 455 ... def __init__(self, sequence): ... self.items = sequence ... # ... def __getitem__(self, key): ... print(’called __getitem__({})’ ... .format(key)) ... return self.items[key] >>> m = MyList([’a’, ’b’, ’c’]) 456 >>> m.__getitem__(0) 457 >>> m.__getitem__(1) 458 >>> m.__getitem__(2) 459 >>> m.__getitem__(3) 460 >>> m[0] 461 >>> m[1] 462 >>> m[2] 463 >>> m[3] 464 >>> hasattr(m, ’__iter__’) 465 >>> hasattr(m, ’__getitem__’) 466 >>> it = iter(m) 467 >>> it.next() 468 >>> it.next() 469 >>> it.next() 470 >>> it.next() 471 >>> list(m) 472 22
23. >>> for item in m: 473 ... print(item) >>> m = MyList({’Jan’: 1, ’Feb’: 2, ’Mar’: 3}) 474 >>> m[’Jan’] 475 >>> for item in m: 476 ... print(m) >>> list(m) 477 >>> m = [1, 2, 3] 478 >>> reversed(m) 479 >>> it = reversed(m) 480 >>> type(it) 481 >>> dir(it) 482 >>> it.next() 483 >>> it.next() 484 >>> it.next() 485 >>> it.next() 486 >>> it.next() 487 >>> it.next() 488 >>> m 489 >>> for i in m: 490 ... print(i) >>> m.__getitem__(0) 491 >>> m.__getitem__(1) 492 >>> m.__getitem__(2) 493 >>> m.__getitem__(3) 494 >>> it = reversed(m) 495 >>> it2 = it.__iter__() 496 >>> hasattr(it2, ’next’) 497 >>> m = [2 * i for i in range(3)] 498 >>> m 499 >>> type(m) 500 >>> mi = (2 * i for i in range(3)) 501 >>> mi 502 >>> type(mi) 503 >>> hasattr(mi, ’next’) 504 >>> dir(mi) 505 23
24. >>> help(mi) 506 >>> mi.next() 507 >>> mi.next() 508 >>> mi.next() 509 >>> mi.next() 510 Exercises: Iterators >>> m = [1, 2, 3] 511 >>> it = iter(m) 512 >>> it.next() 513 >>> it.next() 514 >>> it.next() 515 >>> it.next() 516 >>> for n in m: 517 ... print(n) >>> it = iter(m) 518 >>> it2 = iter(it) 519 >>> list(it2) 520 >>> list(it) 521 >>> it1 = iter(m) 522 >>> it2 = iter(m) 523 >>> list(it1) 524 >>> list(it2) 525 >>> list(it1) 526 >>> list(it2) 527 >>> d = {’one’: 1, ’two’: 2, ’three’:3} 528 >>> it = iter(d) 529 >>> list(it) 530 >>> mi = (2 * i for i in range(3)) 531 >>> list(mi) 532 >>> list(mi) 533 >>> import itertools 534 Take a look at the itertools module documentation. 24
25. Iterators continued >>> class MyIterable(object): 535 ... pass >>> myit = MyIterable() 536 >>> iter(myit) 537 >>> def mygetitem(self, key): 538 ... # Note we ignore self! ... print(’called mygetitem({})’.format(key)) ... return [0, 1, 2][key] >>> MyIterable.__getitem__ = mygetitem 539 >>> iter(myit) 540 >>> list(iter(myit)) 541 >>> 1 in myit 542 >>> x, y, z = myit 543 >>> myit2 = iter([1, 2, 2, 3]) 544 >>> 2 in myit2 545 >>> 2 in myit2 546 >>> 2 in myit2 547 >>> class ListOfThree(object): 548 ... def __iter__(self): ... self.index = 0 ... return self ... # ... def __next__(self): # Python 3.x ... if self.index < 3: ... self.index += 1 ... return self.index ... raise StopIteration ... # ... def next(self): # Python 2.x ... self.__next__(self) >>> m3 = ListOfThree() 549 >>> m3it = iter(m3) 550 >>> m3it.next() 551 >>> m3it.next() 552 >>> m3it.next() 553 >>> m3it.next() 554 25
26. >>> list(m3it) 555 >>> list(m3it) 556 Exercises: Iterators continued Design a subclass of dict whose iterator would return its keys, as does dict, but in sorted order, and without using yield. Design a class reversed to mimic Python’s built in reverse function. Assume an indexable sequence (that supports __getitem__) as parameter. Implement one or both of these designs. Generators >>> list_comprehension = [2 * i for i in range(5)] 557 >>> list_comprehension 558 >>> gen_exp = (2 * i for i in range(5)) 559 >>> gen_exp 560 >>> hasattr(gen_exp, ’next’) 561 >>> list(gen_exp) 562 >>> list(gen_exp) 563 >>> for i in (2 * i for i in range(5)): 564 ... print(i) >>> def list123(): 565 ... yield 1 ... yield 2 ... yield 3 >>> list123 566 >>> list123() 567 >>> it = list123() 568 >>> it.next() 569 >>> it.next() 570 >>> it.next() 571 >>> it.next() 572 >>> def even(limit): 573 ... for i in range(0, limit, 2): ... print(’Yielding’, i) ... yield i ... print(’done loop, falling out’) 26
27. >>> it = even(3) 574 >>> it 575 >>> it.next() 576 >>> it.next() 577 >>> it.next() 578 >>> for i in even(3): 579 ... print(i) >>> list(even(10)) 580 Compare these versions >>> def even_1(limit): 581 ... for i in range(0, limit, 2): ... yield i >>> def even_2(limit): 582 ... result = [] ... for i in range(0, limit, 2): ... result.append(i) ... return result >>> [i for i in even_1(10)] 583 >>> [i for i in even_2(10)] 584 >>> def paragraphs(lines): 585 ... result = ’’ ... for line in lines: ... if line.strip() == ’’: ... yield result ... result = ’’ ... else: ... result += line ... yield result >>> list(paragraphs(open(’eg.txt’))) 586 >>> len(list(paragraphs(open(’eg.txt’)))) 587 Exercises: Generators Write a generator sdouble(str) that takes a string a returns that string ”doubled“ 5 times. E.g. sdbouble(’s’) would yield these values: [’s’, ’ss’, ’ssss’, ’ssssssss’, ’ssssssssssssssss’]. 27
28. Re-design the earlier (iterator subclass of dict) exercise to use yield in the next method. Write a generator that returns sentences out of a paragraph. Make some simple assumptions about how sentences start and/or end. Write code which reads a file and produces a histogram of the frequency of all words in the file. Explore further the itertools module. First class objects Python exposes many language features and places almost no constraints on what types data structures can hold. Here’s an example of using a dictionary of functions to create a simple calculator. In some languages this require a case or switch statement, or a series of if statements. If you’ve been using such a language for a while, this example may help you expand the range of solutions you can imagine in Python. >>> 7+3 588 >>> import operator 589 >>> operator.add(7, 3) 590 >>> expr = ’7+3’ 591 >>> lhs, op, rhs = expr 592 >>> lhs, op, rhs 593 >>> lhs, rhs = int(lhs), int(rhs) 594 >>> lhs, op, rhs 595 >>> op, lhs, rhs 596 >>> operator.add(lhs, rhs) 597 >>> ops = { 598 ... ’+’: operator.add, ... ’-’: operator.sub, ... } >>> ops[op] (lhs, rhs) 599 >>> def calc(expr): 600 ... lhs, op, rhs = expr ... lhs, rhs = int(lhs), int(rhs) ... return ops[op] (lhs, rhs) >>> calc(’7+3’) 601 >>> calc(’9-5’) 602 >>> calc(’8/2’) 603 >>> ops[’/’] = operator.div 604 >>> calc(’8/2’) 605 28
29. >>> class Unpacker(object): 606 ... slices = { ... ’header’: slice(0, 3), ... ’trailer’: slice(12, 18), ... ’middle’: slice(6, 9) ... } ... # ... def __init__(self, record): ... self.record = record ... # ... def __getattr__(self, attr): ... if attr in self.slices: ... return self.record[self.slices[attr]] ... raise AttributeError( ... "’Unpacker’ object has no attribute ’{}’" ... .format(attr)) ... >>> u = Unpacker(’abcdefghijklmnopqrstuvwxyz’) 607 >>> u.header 608 >>> u.trailer 609 >>> u.middle 610 Closures and partial functions >>> def log(message, subsystem): 611 ... """ ... Write the contents of ’message’ ... to the specified subsystem. ... """ ... print(’LOG - {}: {}’.format(subsystem, message)) >>> log(’Initializing server’, ’server’) 612 >>> log(’Reading config file’, ’server’) 613 >>> def server_log(message): 614 ... log(message, ’server’) >>> server_log(’Initializing server’) 615 >>> server_log(’Reading config file’) 616 >>> import functools 617 >>> server_log = functools.partial(log, subsystem=’server’) 618 29
30. >>> server_log 619 >>> server_log.func is log 620 >>> server_log.keywords 621 >>> server_log(’Initializing server’) 622 >>> server_log(’Reading config file’) 623 Bound methods are a form of partials: >>> SENTENCE_ENDING = ’.?!’ 624 >>> sentence = ’This is a sentence!’ 625 >>> sentence[-1] in SENTENCE_ENDING 626 >>> ’.’ in SENTENCE_ENDING 627 >>> SENTENCE_ENDING.__contains__(’.’) 628 >>> SENTENCE_ENDING.__contains__(’,’) 629 >>> is_sentence_ending = SENTENCE_ENDING.__contains__ 630 >>> is_sentence_ending(’.’) 631 >>> is_sentence_ending(’,’) 632 Yet another way to bind some data is to create a class and give it a __call__ method: >>> class SentenceEnding(object): 633 ... def __init__(self, characters): ... self.punctuation = characters ... # ... def __call__(self, sentence): ... return sentence[-1] in self.punctuation >>> is_sentence1 = SentenceEnding(’.’) 634 >>> is_sentence1(’This is a test.’) 635 >>> is_sentence1(’This is a test!’) 636 >>> is_sentence2 = SentenceEnding(’.!?’) 637 >>> is_sentence2(’This is a test.’) 638 >>> is_sentence2(’This is a test!’) 639 Exercise: namedtuple, operator >>> import collections 640 >>> Month = collections.namedtuple( 641 ... ’Month’, ’name number days’, verbose=True) 30
31. >>> jan = Month(’January’, 1, 31) 642 >>> jan.name 643 >>> jan[0] 644 >>> apr = Month(’April’, 3, 30) 645 >>> apr.days 646 >>> apr[2] 647 >>> jul = Month(’July’, 7, 31) 648 >>> m = [jan, apr, jul] 649 >>> def month_days(month): 650 ... return month.days >>> import operator 651 >>> sorted(m, key=operator.itemgetter(0)) 652 >>> sorted(m, key=operator.attrgetter(’name’)) 653 >>> sorted(m, key=operator.attrgetter(’number’)) 654 31