Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python Speleology

Python Speleology

A deep overview about some tricky python features

Andrés J. Díaz

September 18, 2014
Tweet

More Decks by Andrés J. Díaz

Other Decks in Programming

Transcript

  1. The Zen of Python ◦ Beautiful is better than ugly.

    ◦ Explicit is better than implicit. ◦ Simple is better than complex. ◦ Complex is better than complicated. ◦ Flat is better than nested. ◦ Sparse is better than dense. ◦ Readability counts. ◦ Special cases aren't special enough to break the rules. ◦ Although practicality beats purity. ◦ Errors should never pass silently. ◦ Unless explicitly silenced. ◦ In the face of ambiguity, refuse the temptation to guess. ◦ There should be one -and preferably only one- obvious way to do it. ◦ Although that way may not be obvious at first unless you're Dutch. ◦ Now is better than never. ◦ Although never is often better than right now. ◦ If the implementation is hard to explain, it's a bad idea. ◦ If the implementation is easy to explain, it may be a good idea. ◦ Namespaces are one honking great idea -let's do more of those!
  2. Types and objects: To be or not to be a

    = 256 b = 256 print (a is b) a = 257 b = 257 print (a is b)
  3. Types and objects: To be or not to be a

    = 256 b = 256 print (a is b) a = 257 b = 257 print (a is b) True False
  4. Types and objects: To be or not to be a

    = 256 b = 256 print (a is b) a = 257 b = 257 print (a is b) True False a = 256 b = 256 print id(a), id(b) a = 257 b = 257 print id(a), id(b) 22036112 22036112 22363568 22363640
  5. Types and objects: To be or not to be a

    = 256 b = 256 print (a is b) a = 257 b = 257 print (a is b) True False a = 256 b = 256 print id(a), id(b) a = 257 b = 257 print id(a), id(b)
  6. Types and objects: To be or not to be a

    = 256 b = 256 print (a is b) a = 257 b = 257 print (a is b) True False a = 256 b = 256 print id(a), id(b) a = 257 b = 257 print id(a), id(b) 22036112 22036112 22363568 22363640
  7. Types and objects: functions are objects def x(f): return f

    def y(f): return x print x(y)(8)(0) 0
  8. Types and objects: functions are objects def x(i): if x.enabled:

    return i else: return "disabled" x.enabled = True print x(8)
  9. Types and objects: functions are objects def x(i): if x.enabled:

    return i else: return "disabled" x.enabled = True print x(8) What happened if not set x.enabled? 8
  10. Types and objects: star operator args = (3 , 6)

    print range(*args) args = { "name": "example" } def f(name): print name f(**args) def f(*args, **kwargs): print args print kwargs
  11. Types and objects: star operator args = (3 , 6)

    print range(*args) args = { "name": "example" } def f(name): print name f(**args) def f(*args, **kwargs): print args print kwargs [3, 4, 5] example (1, 2, 3) {'name': 'example'}
  12. Reflection: get & set class Example(object): num = 10 x

    = Example dir(x) hasattr(x, "num") == True getattr(x, "num", 0) == 10 setattr(x, "num", 20)
  13. Reflection: local & global globals() Return a dictionary representing the

    current global symbol table. This is always the dictionary of the current module (inside a function or method, this is the module where it is defined, not the module from which it is called). locals() Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks.
  14. Reflection: __magic__ __name__ This is the name of the function.

    This only have a meaningful value is the function is defined with “def”. __class__ This is a reference to the class a method belongs to. __code__ This is a reference to the code object used in the implementation of python.
  15. Reflection: inspector from inspect import getcomments # This is a

    comment def f(x): print x print getcomments(f)
  16. Reflection: inspector from inspect import getcomments # This is a

    comment def f(x): print x print getcomments(f) # This is a comment
  17. Reflection: inspector from inspect import getsource # This is a

    comment def f(x): print x print getsource(f)
  18. Reflection: inspector from inspect import getsource # This is a

    comment def f(x): print x print getsource(f) def f(x): print x
  19. Reflection: let's more tricky def f(x): print x print f.__code__.co_code

    'd\x01\x00GHd\x00\x00S' YEEES!!! THE BYTECODE!!!
  20. Context manager: nowadays... with Item() as item: item.do class Item(object):

    def __enter__(self): self.open() return self def __exit__(self,exc_type,exc_value,exc_t): self.close() ...
  21. Context manager: the real world with file("/tmp/test", "w") as f:

    f.write("hello world") with lock(): # do some concurrent with sudo("root"): # do some as root
  22. Decorations: bold & italic def makebold(fn): def wrapped(): return "<b>"

    + fn() + "</b>" return wrapped def makeitalic(fn): def wrapped(): return "<i>" + fn() + "</i>" return wrapped @makebold @makeitalic def hello(): return "hello world"
  23. Decorations: bold & italic def makebold(fn): def wrapped(): return "<b>"

    + fn() + "</b>" return wrapped def makeitalic(fn): def wrapped(): return "<i>" + fn() + "</i>" return wrapped @makebold @makeitalic def hello(): return "hello world" <b><i>hello world</i></b>
  24. Decorations: complex decor def makestyle(arg): def decorator(f): def wrapper(*args, **kw):

    return "<" + arg + ">" + f() + "</" + arg + ">" return wrapper return decorator @makestyle("b") def hello(): return "hello world" print hello()
  25. Decorations: syntax sugar def makebold(fn): def wrapped(): return "<b>" +

    fn() + "</b>" return wrapped def makestyle(arg): def decorator(f): def wrapper(*args, **kw): return "<" + arg + ">" + f() + "</" + arg + ">" return wrapper return decorator makebold(hello)
  26. Iterations: comprehesions squares = [] for x in range(10): squares.append(x**2)

    squares = [x**2 for x in range(10)] [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y] { (k,v) for k,v in [(1,2)] }
  27. Iterations: comprehesions squares = [] for x in range(10): squares.append(x**2)

    squares = [x**2 for x in range(10)] [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y] { (k,v) for k,v in [(1,2)] } SET NOT DICT!
  28. Iterations: comprehesions squares = [] for x in range(10): squares.append(x**2)

    squares = [x**2 for x in range(10)] [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y] { (k,v) for k,v in [(1,2)] } SET NOT DICT! { k:v for k,v in [(1,2)] }
  29. Iterations: co-routine def countdown(n): print "Counting down from", n while

    n > 0: yield n n -= 1 print "Done counting down" for i in countdown(10): print i
  30. Iterations: co-routine def countdown(n): print "Counting down from", n while

    n > 0: yield n n -= 1 print "Done counting down" for i in countdown(10): print i Counting down from 10 10 9 8 7 6 5 4 3 2 1 Done counting down
  31. Iterations: co-routine def countdown(n): print "Counting down from", n while

    n > 0: yield n n -= 1 print "Done counting down" print countdown(10) <generator object at 0x4035874c>
  32. Overloading: binary operations __add__(self, other) x + y __sub__(self, other)

    x - y __mul__(self, other) x * y __div__(self, other) x / y __pow__(self, other) x ** y
  33. Overloading: binary operations __radd__(self, other) y + x __rsub__(self, other)

    y - x __rmul__(self, other) y * x __rdiv__(self, other) y / x __rpow__(self, other) y ** x
  34. Overloading: binary operations __radd__(self, other) 1 + x __rsub__(self, other)

    1 - x __rmul__(self, other) 1 * x __rdiv__(self, other) 1 / x __rpow__(self, other) 1 ** x
  35. Overloading: binary operations __iadd__(self, other) x+=y __isub__(self, other) x-=y __imul__(self,

    other) x*=y __idiv__(self, other) x/=y __ipow__(self, other) x**=y
  36. Overloading: comparison operations __eq__(self, other) x == y __lt__(self, other)

    x < y __le__(self, other) x <= y __gt__(self, other) x > y __ge__(self, other) x >= y __ne__(self, other) x != y
  37. Overloading: containers __contains__(self, other) y in x __getitem__(self, other) x[y]

    __setitem__(self, other,value) x[y] = z __delitem__(self, other) del x[y] __len__(self) len(x) __reversed__(self) reversed(x) __iter__(self) iter(x)
  38. The Class Factory: class & objects class Example(object): attribute =

    "this is a class attribute" def __init__(self): self.attribute = "this is an obj attr override class one" self.another = "this is another obj attr, no class" print Example.attribute print Example().attribute print Example().another print Example.another
  39. The Class Factory: class & objects class Example(object): attribute =

    "this is a class attribute" def __init__(self): self.attribute = "this is an obj attr override class one" self.another = "this is another obj attr, no class" print Example.attribute print Example().attribute print Example().another print Example.another this is a class attribute this is an object attribute and override class one this is another object attribute, no class Traceback (most recent call last): Line 11, in <module> print Example.another AttributeError: type object 'Example' has no attribute 'another'
  40. The Class Factory: set & get class Example(object): def __init__(self):

    self._name = x @property def name(self): return self._name @name.setter def name(self, value): self._name = value
  41. The Class Factory: @property abuse class Example(object): def __init__(self, host,

    port): self.host = host self.port = port @property def connect(self): lib.connect(self.host, self.port)
  42. The Class Factory: @property abuse class Example(object): def __init__(self, host,

    port): self.host = host self.port = port @property def connect(self): lib.connect(self.host, self.port) NEVER TYPE METHODS AS PROPERTIES
  43. The Class Factory: methods and more methods @staticmethod Nothing more

    than a function defined inside a class. It is callable without instantiating the class first. It’s definition is immutable via inheritance. @classmethod Also callable without instantiating the class, but its definition follows Sub class, not Parent class, via inheritance. That’s because the first argument for @classmethod function must always be cls (class).
  44. The Class Factory: methods and more methods @staticmethod Nothing more

    than a function defined inside a class. It is callable without instantiating the class first. It’s definition is immutable via inheritance. @classmethod Also callable without instantiating the class, but its definition follows Sub class, not Parent class, via inheritance. That’s because the first argument for @classmethod function must always be cls (class). @staticmethod def static_method(): print "I do not receive nothing :(" @classmethod def class_method(cls): print "I'm a class %s" % str(cls)
  45. The Class Factory: methods and more methods class Example(object): def

    __init__(self, name): self.name = name @classmethod def class_method(cls, name): return cls(name) x = Example.class_method("example") print x
  46. The Class Factory: methods and more methods class Example(object): def

    __init__(self, name): self.name = name @classmethod def class_method(cls, name): return cls(name) x = Example.class_method("example") print x <__main__.Example object at 0x40358b2c>
  47. The Class Factory: children and parents class Example(object): def __init__(self,

    name): self.name = name def do_something(self): raise NotImplementedError() class ChildExample(Example): def do_something(self): print self.name
  48. The Class Factory: children and parents class ExampleA(object): def __init__(self,

    name): self.name = name def do_something(self): raise NotImplementedError() class ExampleB(object): def do_otherthing(self): raise NotImplementedError() class ChildExample(ExampleB, ExampleA): def do_something(self): print self.name def do_otherthing(self):
  49. The Class Factory: τὰ μετὰ τὰ κλάση class Example(object): pass

    x = Example() y = Example print x print y <__main__.Example object at 0x4035894c> <class '__main__.Example'>
  50. The Class Factory: τὰ μετὰ τὰ κλάση def example(): class

    Example(object): pass return Example x = example() y = x() print x print y
  51. The Class Factory: τὰ μετὰ τὰ κλάση def example(): class

    Example(object): pass return Example x = example() y = x() print x print y <class '__main__.Example'> <__main__.Example object at 0x403589ec>
  52. The Class Factory: τὰ μετὰ τὰ κλάση class Example(object): pass

    print type(Example) print type(Example()) <type 'type'> <class '__main__.Example'>
  53. The Class Factory: τὰ μετὰ τὰ κλάση class Example(object): pass

    print type(Example) print type(Example()) <type 'type'> <class '__main__.Example'>
  54. The Class Factory: τὰ μετὰ τὰ κλάση class Example(object): pass

    print type(Example) print type(Example()) <type 'type'> <class '__main__.Example'>
  55. The Class Factory: τὰ μετὰ τὰ κλάση class Example(object): pass

    x = Example y = type('Example', (object,), {}) print x print y print (x == y)
  56. The Class Factory: τὰ μετὰ τὰ κλάση class Example(object): pass

    x = Example y = type('Example', (object,), {}) print x print y print (x == y) <class '__main__.Example'> <class '__main__.Example'> False
  57. The Class Factory: __magic__ __new__(cls, *args, **kwargs) Is the first

    method to get called in an object's instantiation, is a @classmethod, and must return a new instance of type cls. __init__(self, *args, **kwargs) Is the initializer for the instance. It gets passed whatever the primary constructor was called with. __del__(self) Is the destructor of the instance, will be invoked before clean the reference to the instance.
  58. The Class Factory: __metaclass__ def upper_attr(f_class_name, f_class_parents, f_class_attr): attrs =

    ((name, value) \ for name, value in f_class_attr.items() \ if not name.startswith('__')) uppercase_attr = dict((name.upper(), value) \ for name, value in attrs) return type(f_class_name, f_class_parents, uppercase_attr) class Foo(object): __metaclass__ = upper_attr bar = 'bip' print hasattr(Foo, 'bar') print hasattr(Foo, 'BAR')
  59. The Class Factory: __metaclass__ def upper_attr(f_class_name, f_class_parents, f_class_attr): attrs =

    ((name, value) \ for name, value in f_class_attr.items() \ if not name.startswith('__')) uppercase_attr = dict((name.upper(), value) \ for name, value in attrs) return type(f_class_name, f_class_parents, uppercase_attr) class Foo(object): __metaclass__ = upper_attr bar = 'bip' print hasattr(Foo, 'bar') print hasattr(Foo, 'BAR') False True
  60. Monkey patching: first try class Person(object): def speak(self): print "hello"

    def monkey(foo): print "uh uh uh" Person.speak = monkey x = Person() x.speak()
  61. Monkey patching: be evil }:-) import os def monkey(*args, **kwargs):

    print "no no no" os.system = monkey os.system("ls /tmp")
  62. Monkey patching: be evil }:-) import os def monkey(*args, **kwargs):

    print "no no no" os.system = monkey os.system("ls /tmp") no no no
  63. Profiling: hand made import time class Timer(object): def __init__(self, verbose=False):

    self.verbose = verbose def __enter__(self): self.start = time.time() return self def __exit__(self, *args): self.end = time.time() self.secs = self.end - self.start self.msecs = self.secs * 1000 # millisecs if self.verbose: print 'elapsed time: %f ms' % self.msecs
  64. Profiling: hand made import time class Timer(object): def __init__(self, verbose=False):

    self.verbose = verbose def __enter__(self): self.start = time.time() return self def __exit__(self, *args): self.end = time.time() self.secs = self.end - self.start self.msecs = self.secs * 1000 # millisecs if self.verbose: print 'elapsed time: %f ms' % self.msecs from redis import Redis rdb = Redis() with Timer() as t: rdb.lpush("foo", "bar") print "=> elasped lpush: %s s" % t. secs with Timer as t: rdb.lpop("foo") print "=> elasped lpop: %s s" % t.secs
  65. Profiling: profile & cProfile try: import cProfile as profile except

    ImportError: import profile def fib(n): if n == 0: return 0 elif n == 1: return 1 else: return fib(n-1) + fib(n-2) def fib_seq(n): seq = [ ] if n > 0: seq.extend(fib_seq(n-1)) seq.append(fib(n)) return seq
  66. Profiling: profile & cProfile try: import cProfile as profile except

    ImportError: import profile def fib(n): if n == 0: return 0 elif n == 1: return 1 else: return fib(n-1) + fib(n-2) def fib_seq(n): seq = [ ] if n > 0: seq.extend(fib_seq(n-1)) seq.append(fib(n)) return seq [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765] 57356 function calls (66 primitive calls) in 0.746 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 21 0.000 0.000 0.000 0.000 :0(append) 20 0.000 0.000 0.000 0.000 :0(extend) 1 0.001 0.001 0.001 0.001 :0(setprofile) 1 0.000 0.000 0.744 0.744 <string>:1(<module>) 1 0.000 0.000 0.746 0.746 profile:0(print fib_seq(20); print) 0 0.000 0.000 profile:0(profiler) 57291/21 0.743 0.000 0.743 0.035 profile_fibonacci_raw.py:13(fib) 21/1 0.001 0.000 0.744 0.744 profile_fibonacci_raw.py:22(fib_seq)
  67. Documentation: Zen Don't create documentation for your code. Code your

    documentation. def elements(n): """Return a list of n numbers from 0 to n- 1. """ return range(0,n)
  68. Documentation: everything is an object def elements(n): """Return a list

    of n numbers from 0 to n- 1. """ return range(0,n) print elements.__doc__
  69. Documentation: everything is an object def elements(n): """Return a list

    of n numbers from 0 to n- 1. """ return range(0,n) print elements.__doc__ Return a list of n numbers from 0 to n-1
  70. Documentation: style is important def elements(n): """Return a list of

    n numbers from 0 to n-1. :type n: int :param n: the limit upper for the elements to be created (not included). :return: a class:`list` with the items. """ return range(0,n) class Example(object): """This is the documentation of the class. Usually you do not need it :) """ def __init__(self, param): """This is the documentation of the instance type. """
  71. Documentation: Bonus... abuse the doc class Command(object): """Undocumented command """

    class ListCommand(Command): """List command. """ def show_help(command): cmd = getattr(globals()[__name__], "%sCommand" % command, None) if cmd is None: return "Invalid command" else: return cmd.__doc__ print show_help("List")
  72. Future: python3 def hello(name: str, age: int) -> str: return

    name print hello.__anotations__ {'return':<class'int'>,'name':<class'str'>,'age':<class'int'>}