Namespaces in Python by Eric Snow

2. Eric Snow Fusion-io ericsnowcurrently@gmail.com Slides: http://goo.gl/gnhXt
3. Namespaces in Python
4. "Namespaces are one honking great idea -- let's do more of those!" - Tim Peters, The Zen of Python
5. What do we mean, "namespaces"? What do namespaces look like? How are they used? How do they work?
6. What are Namespaces?
7. A concept: a set of independently identifiable expectations + a unique (within set) label for each member
8. A concept: a set of independently identifiable expectations + a unique (within set) label for each member context + name => value
9. Nested namespaces? with nesting without nesting spam spam sys.version_info sys_version_info os.path.exists() os_path_exists()
10. "Flat is better than nested." - Tim Peters, The Zen of Python
11. Namespaces in C? PyObject Py_TYPE PyDict_SetItem PyErr_SetString Py_None
12. Namespaces in C? PyObject Py_TYPE PyDict_SetItem PyErr_SetString Py_None Py . Object Py . TYPE Py . Dict . SetItem Py . Err . SetString Py . None
13. Namespace Containers
14. Namespace Containers (Image CC http://www.flickr. com/photos/greeblie/)
15. Namespace Containers mapping-based attribute-based property-based
16. Namespace Containers mapping-based VOLATILE attribute-based STABLE property-based STATIC
17. A Type for sys.implementation • Python implementation specifics: PEP 421
18. A Type for sys.implementation • Python implementation specifics: PEP 421 • the type must communicate the nature of the namespace
19. A Type for sys.implementation • Python implementation specifics: PEP 421 • the type must communicate the nature of the namespace • dict too soft
20. A Type for sys.implementation • Python implementation specifics: PEP 421 • the type must communicate the nature of the namespace • dict too soft • namedtuple too hard
21. A Type for sys.implementation • Python implementation specifics: PEP 421 • the type must communicate the nature of the namespace • dict too soft • namedtuple too hard • object just right...
22. A Type for sys.implementation • Python implementation specifics: PEP 421 • the type must communicate the nature of the namespace • dict too soft • namedtuple too hard • object just right... • types.SimpleNamespace
23. Communicate the nature of your namespace! mapping-based VOLATILE attribute-based STABLE property-based STATIC
24. Mapping-based Namespaces
25. mapping-based containers (volatile namespaces) • a raw key is mapped to a value
26. Important Mappings: • obj.__dict__ • execution locals, globals, and builtins • sys.modules • os.environ
27. Syntax: obj[key] obj[key] = value del obj[key] key in obj
28. Some available types: • dict • collections.defaultdict • collections.OrderedDict • configparser.ConfigParser • shelve.Shelf • types.MappingProxyType • weakref.WeakKeyDictionary • weakref.WeakValueDictionary
29. How they work: obj[key] obj[key] = value del obj[key] key in obj obj.__getitem__(key) obj.__setitem__(key, val) obj.__delitem__(key) obj.__contains__(key)
30. How they work: obj[key] obj[key] = value del obj[key] key in obj collections.abc.(Mutable)Mapping obj.__getitem__(key) obj.__setitem__(key, val) obj.__delitem__(key) obj.__contains__(key)
31. class ReadOnlyProxy(Mapping): def __init__(self, mapping): self._mapping = mapping def __len__(self): return len(self._mapping) def __iter__(self): return iter(self._mapping) def __getitem__(self, key): return self._mapping[key]
32. Gems: • dict(mapping, **other_mapping) • d.update(mapping, **other_mapping) • dict.from_keys() • dict.setdefault() • str.formatmap() • operator.itemgetter() • threading.local
33. Property-based Namespaces
34. property-based containers (static namespaces) • a proper name aliases an underlying key • the key is associated with a value • the names and values are fixed in place (i.e. read-only)
35. Example: >>> sys.version_info sys.version_info(major=3, minor=4, micro=0, releaselevel='alpha', serial=0) >>> os.times() posix.times_result(user=0.09, system=0.04, children_user=0.0, children_system=0.0, elapsed=17248877.17)
36. Some available types: • collections.namedtuple • structseq (in C)
37. namedtuple: context + names Spam = namedtuple('Spam', "a b c") class Spam(tuple): def __new__(cls, a, b, c): return super().__new__(cls, a, b, c) @property def a(self): return self[0] ...
38. class ReadOnlyProxy: def __init__(self, obj): self._obj = obj @property def spam(self): return self._obj.spam @property def eggs(self): return self._obj.eggs
39. Attribute-based Namespaces
40. attribute-based containers (stable namespaces) • a proper name is associated with a value
41. How they work: obj.name obj.name = value del obj.name obj.__getattribute__(name) obj.__setattr__(name, val) obj.__delattr__(name)
42. How they work: obj.name obj.name = value del obj.name obj.__getattribute__(name) obj.__setattr__(name, val) obj.__delattr__(name)
43. How __getattribute__() works: 1. use a "data" descriptor, if there is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError
44. Descriptors obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)
45. Descriptors (non-data) obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)
46. Descriptors (data) obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)
47. Descriptors obj.__get__() obj.__set__() obj.__delete__() property classmethod staticmethod
48. How __getattribute__() works: 1. use a "data" descriptor, if there is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError
49. How __getattribute__() works: 1. use a "data" descriptor, if there is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError MRO!
50. How __getattribute__() works: 1. use a "data" descriptor, if there is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError Method Resolution Order!
51. class ReadOnlyProxy: def __init__(self, obj): self._obj = obj def __getattr__(self, name): return getattr(self._obj, name)
52. Some available types: • object • types.SimpleNamespace • argparse.Namespace • multiprocessing.Manager.Namespace • xmlrpc.client.ServerProxy • unittest.Mock
53. Gems: • __qualname__ • operator.attrgetter()
54. Communicate the nature of your namespace! mapping-based VOLATILE attribute-based STABLE property-based STATIC
55. Objects
56. Objects • are namespace containers • have namespaces (mostly) • ergo, as container, proxies a namespace
57. Objects • are namespace containers • have namespaces (mostly) • ergo, as container, proxies a namespace >>> ns = SimpleNamespace() >>> ns_ns = vars(ns) >>> ns.__dict__ is ns_ns True
58. Objects • are namespace containers • have namespaces (mostly) • ergo, as container, proxies a namespace • base object type has slots • assignment/definition: "bind" object to namespace
59. Object access helpers: • getattr() • setattr() • delattr() • hasattr() wrapper around getattr() • dir() calls obj.__dir__() • vars() exposes namespace* • inspect.getattr_static() • inspect.getmembers()
60. __dict__ • most builtins don't have one • key exception: function objects
61. __dict__ • most builtins don't have one • key exception: function objects • module namespaces exposed as __dict__
62. __dict__ • most builtins don't have one • key exception: function objects • module namespaces exposed as __dict__ • read-only proxy of class namespaces exposed as __dict__ • use metaclass __prepare__() to customize the namespace used during class definition
63. __dict__ vs. __slots__ • slots are turned into descriptors • __dict__ is disabled
64. Execution
65. Name lookup: • explicit lookup happens relative to containers • Method Resolution Order (MRO) • lookup on class vs. on objects • special method lookup
66. Name lookup: • explicit lookup happens relative to containers • Method Resolution Order (MRO) • lookup on class vs. on objects • special method lookup • implicit lookup happens relative to execution namespaces...
67. Implicit lookup: 1. f_locals 2. f_globals 3. f_builtins
68. Scope (a.k.a. implicit lookup): 1. locals • function execution namespace • use vars() or locals() 2. non-locals (closures) 3. globals • module execution namespace • use vars() or globals() 4. builtins • interpreter execution namespace • use builtins or __builtins__
69. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals: sys.modules
70. sys.modules >>> mordor = sys.modules['mordor']
71. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals: sys.modules
72. sys.modules >>> import mordor Traceback (most recent call last): File ".../mordor.py", line 14, in <module> raise ImportError(boromir_says) ImportError: One does not simply import Mordor.
73. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals: sys.modules ◦ access via import statement
74. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals: sys.modules ◦ access via import statement • id()
75. Summary
76. APIs: mapping-based • see collections.abc.Mapping property-based • use properties + "private" attributes attribute-based • __getattribute__() and __getattr__() • __setattr__() • __delattr__()
77. Key object attributes: • __dict__ • __dir__() • __slots__() • __mro__ (for classes)
78. Helpers: • globals() • locals() • vars() • dir() • getattr() • setattr() • delattr() • hasattr() • inspect.getattr_static() • id()
79. • namespaces: context + name => value • containers: ◦ mapping-based (volatile) ◦ property-based (static) ◦ attribute-based (stable) • pervasive in Python • rich APIs • rich tools for objects
80. Questions? Eric Snow Fusion-io ericsnowcurrently@gmail.com Slides: http://goo.gl/gnhXt