Namespaces in Python by Eric Snow

Afcfefa1f067d10bd021de0cc2e5e806?s=47 PyCon 2013
March 18, 2013

Namespaces in Python by Eric Snow

Afcfefa1f067d10bd021de0cc2e5e806?s=128

PyCon 2013

March 18, 2013
Tweet

Transcript

  1. None
  2. Eric Snow Fusion-io ericsnowcurrently@gmail.com Slides: http://goo.gl/gnhXt

  3. Namespaces in Python

  4. "Namespaces are one honking great idea -- let's do more

    of those!" - Tim Peters, The Zen of Python
  5. What do we mean, "namespaces"? What do namespaces look like?

    How are they used? How do they work?
  6. What are Namespaces?

  7. A concept: a set of independently identifiable expectations + a

    unique (within set) label for each member
  8. A concept: a set of independently identifiable expectations + a

    unique (within set) label for each member context + name => value
  9. Nested namespaces? with nesting without nesting spam spam sys.version_info sys_version_info

    os.path.exists() os_path_exists()
  10. "Flat is better than nested." - Tim Peters, The Zen

    of Python
  11. Namespaces in C? PyObject Py_TYPE PyDict_SetItem PyErr_SetString Py_None

  12. Namespaces in C? PyObject Py_TYPE PyDict_SetItem PyErr_SetString Py_None Py .

    Object Py . TYPE Py . Dict . SetItem Py . Err . SetString Py . None
  13. Namespace Containers

  14. Namespace Containers (Image CC http://www.flickr. com/photos/greeblie/)

  15. Namespace Containers mapping-based attribute-based property-based

  16. Namespace Containers mapping-based VOLATILE attribute-based STABLE property-based STATIC

  17. A Type for sys.implementation • Python implementation specifics: PEP 421

  18. A Type for sys.implementation • Python implementation specifics: PEP 421

    • the type must communicate the nature of the namespace
  19. A Type for sys.implementation • Python implementation specifics: PEP 421

    • the type must communicate the nature of the namespace • dict too soft
  20. A Type for sys.implementation • Python implementation specifics: PEP 421

    • the type must communicate the nature of the namespace • dict too soft • namedtuple too hard
  21. A Type for sys.implementation • Python implementation specifics: PEP 421

    • the type must communicate the nature of the namespace • dict too soft • namedtuple too hard • object just right...
  22. A Type for sys.implementation • Python implementation specifics: PEP 421

    • the type must communicate the nature of the namespace • dict too soft • namedtuple too hard • object just right... • types.SimpleNamespace
  23. Communicate the nature of your namespace! mapping-based VOLATILE attribute-based STABLE

    property-based STATIC
  24. Mapping-based Namespaces

  25. mapping-based containers (volatile namespaces) • a raw key is mapped

    to a value
  26. Important Mappings: • obj.__dict__ • execution locals, globals, and builtins

    • sys.modules • os.environ
  27. Syntax: obj[key] obj[key] = value del obj[key] key in obj

  28. Some available types: • dict • collections.defaultdict • collections.OrderedDict •

    configparser.ConfigParser • shelve.Shelf • types.MappingProxyType • weakref.WeakKeyDictionary • weakref.WeakValueDictionary
  29. How they work: obj[key] obj[key] = value del obj[key] key

    in obj obj.__getitem__(key) obj.__setitem__(key, val) obj.__delitem__(key) obj.__contains__(key)
  30. How they work: obj[key] obj[key] = value del obj[key] key

    in obj collections.abc.(Mutable)Mapping obj.__getitem__(key) obj.__setitem__(key, val) obj.__delitem__(key) obj.__contains__(key)
  31. class ReadOnlyProxy(Mapping): def __init__(self, mapping): self._mapping = mapping def __len__(self):

    return len(self._mapping) def __iter__(self): return iter(self._mapping) def __getitem__(self, key): return self._mapping[key]
  32. Gems: • dict(mapping, **other_mapping) • d.update(mapping, **other_mapping) • dict.from_keys() •

    dict.setdefault() • str.formatmap() • operator.itemgetter() • threading.local
  33. Property-based Namespaces

  34. property-based containers (static namespaces) • a proper name aliases an

    underlying key • the key is associated with a value • the names and values are fixed in place (i.e. read-only)
  35. Example: >>> sys.version_info sys.version_info(major=3, minor=4, micro=0, releaselevel='alpha', serial=0) >>> os.times()

    posix.times_result(user=0.09, system=0.04, children_user=0.0, children_system=0.0, elapsed=17248877.17)
  36. Some available types: • collections.namedtuple • structseq (in C)

  37. namedtuple: context + names Spam = namedtuple('Spam', "a b c")

    class Spam(tuple): def __new__(cls, a, b, c): return super().__new__(cls, a, b, c) @property def a(self): return self[0] ...
  38. class ReadOnlyProxy: def __init__(self, obj): self._obj = obj @property def

    spam(self): return self._obj.spam @property def eggs(self): return self._obj.eggs
  39. Attribute-based Namespaces

  40. attribute-based containers (stable namespaces) • a proper name is associated

    with a value
  41. How they work: obj.name obj.name = value del obj.name obj.__getattribute__(name)

    obj.__setattr__(name, val) obj.__delattr__(name)
  42. How they work: obj.name obj.name = value del obj.name obj.__getattribute__(name)

    obj.__setattr__(name, val) obj.__delattr__(name)
  43. How __getattribute__() works: 1. use a "data" descriptor, if there

    is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError
  44. Descriptors obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)

  45. Descriptors (non-data) obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)

  46. Descriptors (data) obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)

  47. Descriptors obj.__get__() obj.__set__() obj.__delete__() property classmethod staticmethod

  48. How __getattribute__() works: 1. use a "data" descriptor, if there

    is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError
  49. How __getattribute__() works: 1. use a "data" descriptor, if there

    is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError MRO!
  50. How __getattribute__() works: 1. use a "data" descriptor, if there

    is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError Method Resolution Order!
  51. class ReadOnlyProxy: def __init__(self, obj): self._obj = obj def __getattr__(self,

    name): return getattr(self._obj, name)
  52. Some available types: • object • types.SimpleNamespace • argparse.Namespace •

    multiprocessing.Manager.Namespace • xmlrpc.client.ServerProxy • unittest.Mock
  53. Gems: • __qualname__ • operator.attrgetter()

  54. Communicate the nature of your namespace! mapping-based VOLATILE attribute-based STABLE

    property-based STATIC
  55. Objects

  56. Objects • are namespace containers • have namespaces (mostly) •

    ergo, as container, proxies a namespace
  57. Objects • are namespace containers • have namespaces (mostly) •

    ergo, as container, proxies a namespace >>> ns = SimpleNamespace() >>> ns_ns = vars(ns) >>> ns.__dict__ is ns_ns True
  58. Objects • are namespace containers • have namespaces (mostly) •

    ergo, as container, proxies a namespace • base object type has slots • assignment/definition: "bind" object to namespace
  59. Object access helpers: • getattr() • setattr() • delattr() •

    hasattr() wrapper around getattr() • dir() calls obj.__dir__() • vars() exposes namespace* • inspect.getattr_static() • inspect.getmembers()
  60. __dict__ • most builtins don't have one • key exception:

    function objects
  61. __dict__ • most builtins don't have one • key exception:

    function objects • module namespaces exposed as __dict__
  62. __dict__ • most builtins don't have one • key exception:

    function objects • module namespaces exposed as __dict__ • read-only proxy of class namespaces exposed as __dict__ • use metaclass __prepare__() to customize the namespace used during class definition
  63. __dict__ vs. __slots__ • slots are turned into descriptors •

    __dict__ is disabled
  64. Execution

  65. Name lookup: • explicit lookup happens relative to containers •

    Method Resolution Order (MRO) • lookup on class vs. on objects • special method lookup
  66. Name lookup: • explicit lookup happens relative to containers •

    Method Resolution Order (MRO) • lookup on class vs. on objects • special method lookup • implicit lookup happens relative to execution namespaces...
  67. Implicit lookup: 1. f_locals 2. f_globals 3. f_builtins

  68. Scope (a.k.a. implicit lookup): 1. locals • function execution namespace

    • use vars() or locals() 2. non-locals (closures) 3. globals • module execution namespace • use vars() or globals() 4. builtins • interpreter execution namespace • use builtins or __builtins__
  69. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals:

    sys.modules
  70. sys.modules >>> mordor = sys.modules['mordor']

  71. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals:

    sys.modules
  72. sys.modules >>> import mordor Traceback (most recent call last): File

    ".../mordor.py", line 14, in <module> raise ImportError(boromir_says) ImportError: One does not simply import Mordor.
  73. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals:

    sys.modules ◦ access via import statement
  74. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals:

    sys.modules ◦ access via import statement • id()
  75. Summary

  76. APIs: mapping-based • see collections.abc.Mapping property-based • use properties +

    "private" attributes attribute-based • __getattribute__() and __getattr__() • __setattr__() • __delattr__()
  77. Key object attributes: • __dict__ • __dir__() • __slots__() •

    __mro__ (for classes)
  78. Helpers: • globals() • locals() • vars() • dir() •

    getattr() • setattr() • delattr() • hasattr() • inspect.getattr_static() • id()
  79. • namespaces: context + name => value • containers: ◦

    mapping-based (volatile) ◦ property-based (static) ◦ attribute-based (stable) • pervasive in Python • rich APIs • rich tools for objects
  80. Questions? Eric Snow Fusion-io ericsnowcurrently@gmail.com Slides: http://goo.gl/gnhXt