$30 off During Our Annual Pro Sale. View Details »

Namespaces in Python by Eric Snow

PyCon 2013
March 18, 2013

Namespaces in Python by Eric Snow

PyCon 2013

March 18, 2013
Tweet

More Decks by PyCon 2013

Other Decks in Technology

Transcript

  1. None
  2. Eric Snow Fusion-io ericsnowcurrently@gmail.com Slides: http://goo.gl/gnhXt

  3. Namespaces in Python

  4. "Namespaces are one honking great idea -- let's do more

    of those!" - Tim Peters, The Zen of Python
  5. What do we mean, "namespaces"? What do namespaces look like?

    How are they used? How do they work?
  6. What are Namespaces?

  7. A concept: a set of independently identifiable expectations + a

    unique (within set) label for each member
  8. A concept: a set of independently identifiable expectations + a

    unique (within set) label for each member context + name => value
  9. Nested namespaces? with nesting without nesting spam spam sys.version_info sys_version_info

    os.path.exists() os_path_exists()
  10. "Flat is better than nested." - Tim Peters, The Zen

    of Python
  11. Namespaces in C? PyObject Py_TYPE PyDict_SetItem PyErr_SetString Py_None

  12. Namespaces in C? PyObject Py_TYPE PyDict_SetItem PyErr_SetString Py_None Py .

    Object Py . TYPE Py . Dict . SetItem Py . Err . SetString Py . None
  13. Namespace Containers

  14. Namespace Containers (Image CC http://www.flickr. com/photos/greeblie/)

  15. Namespace Containers mapping-based attribute-based property-based

  16. Namespace Containers mapping-based VOLATILE attribute-based STABLE property-based STATIC

  17. A Type for sys.implementation • Python implementation specifics: PEP 421

  18. A Type for sys.implementation • Python implementation specifics: PEP 421

    • the type must communicate the nature of the namespace
  19. A Type for sys.implementation • Python implementation specifics: PEP 421

    • the type must communicate the nature of the namespace • dict too soft
  20. A Type for sys.implementation • Python implementation specifics: PEP 421

    • the type must communicate the nature of the namespace • dict too soft • namedtuple too hard
  21. A Type for sys.implementation • Python implementation specifics: PEP 421

    • the type must communicate the nature of the namespace • dict too soft • namedtuple too hard • object just right...
  22. A Type for sys.implementation • Python implementation specifics: PEP 421

    • the type must communicate the nature of the namespace • dict too soft • namedtuple too hard • object just right... • types.SimpleNamespace
  23. Communicate the nature of your namespace! mapping-based VOLATILE attribute-based STABLE

    property-based STATIC
  24. Mapping-based Namespaces

  25. mapping-based containers (volatile namespaces) • a raw key is mapped

    to a value
  26. Important Mappings: • obj.__dict__ • execution locals, globals, and builtins

    • sys.modules • os.environ
  27. Syntax: obj[key] obj[key] = value del obj[key] key in obj

  28. Some available types: • dict • collections.defaultdict • collections.OrderedDict •

    configparser.ConfigParser • shelve.Shelf • types.MappingProxyType • weakref.WeakKeyDictionary • weakref.WeakValueDictionary
  29. How they work: obj[key] obj[key] = value del obj[key] key

    in obj obj.__getitem__(key) obj.__setitem__(key, val) obj.__delitem__(key) obj.__contains__(key)
  30. How they work: obj[key] obj[key] = value del obj[key] key

    in obj collections.abc.(Mutable)Mapping obj.__getitem__(key) obj.__setitem__(key, val) obj.__delitem__(key) obj.__contains__(key)
  31. class ReadOnlyProxy(Mapping): def __init__(self, mapping): self._mapping = mapping def __len__(self):

    return len(self._mapping) def __iter__(self): return iter(self._mapping) def __getitem__(self, key): return self._mapping[key]
  32. Gems: • dict(mapping, **other_mapping) • d.update(mapping, **other_mapping) • dict.from_keys() •

    dict.setdefault() • str.formatmap() • operator.itemgetter() • threading.local
  33. Property-based Namespaces

  34. property-based containers (static namespaces) • a proper name aliases an

    underlying key • the key is associated with a value • the names and values are fixed in place (i.e. read-only)
  35. Example: >>> sys.version_info sys.version_info(major=3, minor=4, micro=0, releaselevel='alpha', serial=0) >>> os.times()

    posix.times_result(user=0.09, system=0.04, children_user=0.0, children_system=0.0, elapsed=17248877.17)
  36. Some available types: • collections.namedtuple • structseq (in C)

  37. namedtuple: context + names Spam = namedtuple('Spam', "a b c")

    class Spam(tuple): def __new__(cls, a, b, c): return super().__new__(cls, a, b, c) @property def a(self): return self[0] ...
  38. class ReadOnlyProxy: def __init__(self, obj): self._obj = obj @property def

    spam(self): return self._obj.spam @property def eggs(self): return self._obj.eggs
  39. Attribute-based Namespaces

  40. attribute-based containers (stable namespaces) • a proper name is associated

    with a value
  41. How they work: obj.name obj.name = value del obj.name obj.__getattribute__(name)

    obj.__setattr__(name, val) obj.__delattr__(name)
  42. How they work: obj.name obj.name = value del obj.name obj.__getattribute__(name)

    obj.__setattr__(name, val) obj.__delattr__(name)
  43. How __getattribute__() works: 1. use a "data" descriptor, if there

    is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError
  44. Descriptors obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)

  45. Descriptors (non-data) obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)

  46. Descriptors (data) obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)

  47. Descriptors obj.__get__() obj.__set__() obj.__delete__() property classmethod staticmethod

  48. How __getattribute__() works: 1. use a "data" descriptor, if there

    is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError
  49. How __getattribute__() works: 1. use a "data" descriptor, if there

    is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError MRO!
  50. How __getattribute__() works: 1. use a "data" descriptor, if there

    is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError Method Resolution Order!
  51. class ReadOnlyProxy: def __init__(self, obj): self._obj = obj def __getattr__(self,

    name): return getattr(self._obj, name)
  52. Some available types: • object • types.SimpleNamespace • argparse.Namespace •

    multiprocessing.Manager.Namespace • xmlrpc.client.ServerProxy • unittest.Mock
  53. Gems: • __qualname__ • operator.attrgetter()

  54. Communicate the nature of your namespace! mapping-based VOLATILE attribute-based STABLE

    property-based STATIC
  55. Objects

  56. Objects • are namespace containers • have namespaces (mostly) •

    ergo, as container, proxies a namespace
  57. Objects • are namespace containers • have namespaces (mostly) •

    ergo, as container, proxies a namespace >>> ns = SimpleNamespace() >>> ns_ns = vars(ns) >>> ns.__dict__ is ns_ns True
  58. Objects • are namespace containers • have namespaces (mostly) •

    ergo, as container, proxies a namespace • base object type has slots • assignment/definition: "bind" object to namespace
  59. Object access helpers: • getattr() • setattr() • delattr() •

    hasattr() wrapper around getattr() • dir() calls obj.__dir__() • vars() exposes namespace* • inspect.getattr_static() • inspect.getmembers()
  60. __dict__ • most builtins don't have one • key exception:

    function objects
  61. __dict__ • most builtins don't have one • key exception:

    function objects • module namespaces exposed as __dict__
  62. __dict__ • most builtins don't have one • key exception:

    function objects • module namespaces exposed as __dict__ • read-only proxy of class namespaces exposed as __dict__ • use metaclass __prepare__() to customize the namespace used during class definition
  63. __dict__ vs. __slots__ • slots are turned into descriptors •

    __dict__ is disabled
  64. Execution

  65. Name lookup: • explicit lookup happens relative to containers •

    Method Resolution Order (MRO) • lookup on class vs. on objects • special method lookup
  66. Name lookup: • explicit lookup happens relative to containers •

    Method Resolution Order (MRO) • lookup on class vs. on objects • special method lookup • implicit lookup happens relative to execution namespaces...
  67. Implicit lookup: 1. f_locals 2. f_globals 3. f_builtins

  68. Scope (a.k.a. implicit lookup): 1. locals • function execution namespace

    • use vars() or locals() 2. non-locals (closures) 3. globals • module execution namespace • use vars() or globals() 4. builtins • interpreter execution namespace • use builtins or __builtins__
  69. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals:

    sys.modules
  70. sys.modules >>> mordor = sys.modules['mordor']

  71. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals:

    sys.modules
  72. sys.modules >>> import mordor Traceback (most recent call last): File

    ".../mordor.py", line 14, in <module> raise ImportError(boromir_says) ImportError: One does not simply import Mordor.
  73. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals:

    sys.modules ◦ access via import statement
  74. Interpreter "top-level" namespaces: • implicit execution-globals: builtins • explicit execution-globals:

    sys.modules ◦ access via import statement • id()
  75. Summary

  76. APIs: mapping-based • see collections.abc.Mapping property-based • use properties +

    "private" attributes attribute-based • __getattribute__() and __getattr__() • __setattr__() • __delattr__()
  77. Key object attributes: • __dict__ • __dir__() • __slots__() •

    __mro__ (for classes)
  78. Helpers: • globals() • locals() • vars() • dir() •

    getattr() • setattr() • delattr() • hasattr() • inspect.getattr_static() • id()
  79. • namespaces: context + name => value • containers: ◦

    mapping-based (volatile) ◦ property-based (static) ◦ attribute-based (stable) • pervasive in Python • rich APIs • rich tools for objects
  80. Questions? Eric Snow Fusion-io ericsnowcurrently@gmail.com Slides: http://goo.gl/gnhXt