Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Namespaces in Python by Eric Snow

PyCon 2013
March 18, 2013

Namespaces in Python by Eric Snow

PyCon 2013

March 18, 2013
Tweet

More Decks by PyCon 2013

Other Decks in Technology

Transcript

  1. View Slide

  2. Eric Snow
    Fusion-io
    [email protected]
    Slides:
    http://goo.gl/gnhXt

    View Slide

  3. Namespaces in Python

    View Slide

  4. "Namespaces are one honking great idea --
    let's do more of those!"
    - Tim Peters, The Zen of Python

    View Slide

  5. What do we mean, "namespaces"?
    What do namespaces look like?
    How are they used?
    How do they work?

    View Slide

  6. What are Namespaces?

    View Slide

  7. A concept:
    a set of independently identifiable expectations
    +
    a unique (within set) label for each member

    View Slide

  8. A concept:
    a set of independently identifiable expectations
    +
    a unique (within set) label for each member
    context + name => value

    View Slide

  9. Nested namespaces?
    with nesting without nesting
    spam spam
    sys.version_info sys_version_info
    os.path.exists() os_path_exists()

    View Slide

  10. "Flat is better than nested."
    - Tim Peters, The Zen of Python

    View Slide

  11. Namespaces in C?
    PyObject
    Py_TYPE
    PyDict_SetItem
    PyErr_SetString
    Py_None

    View Slide

  12. Namespaces in C?
    PyObject
    Py_TYPE
    PyDict_SetItem
    PyErr_SetString
    Py_None
    Py . Object
    Py . TYPE
    Py . Dict . SetItem
    Py . Err . SetString
    Py . None

    View Slide

  13. Namespace Containers

    View Slide

  14. Namespace Containers
    (Image CC http://www.flickr.
    com/photos/greeblie/)

    View Slide

  15. Namespace Containers
    mapping-based
    attribute-based
    property-based

    View Slide

  16. Namespace Containers
    mapping-based
    VOLATILE
    attribute-based
    STABLE
    property-based
    STATIC

    View Slide

  17. A Type for sys.implementation
    ● Python implementation specifics: PEP 421

    View Slide

  18. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    ● the type must communicate the nature of the
    namespace

    View Slide

  19. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    ● the type must communicate the nature of the
    namespace
    ● dict too soft

    View Slide

  20. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    ● the type must communicate the nature of the
    namespace
    ● dict too soft
    ● namedtuple too hard

    View Slide

  21. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    ● the type must communicate the nature of the
    namespace
    ● dict too soft
    ● namedtuple too hard
    ● object just right...

    View Slide

  22. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    ● the type must communicate the nature of the
    namespace
    ● dict too soft
    ● namedtuple too hard
    ● object just right...
    ● types.SimpleNamespace

    View Slide

  23. Communicate the nature of your namespace!
    mapping-based
    VOLATILE
    attribute-based
    STABLE
    property-based
    STATIC

    View Slide

  24. Mapping-based
    Namespaces

    View Slide

  25. mapping-based containers
    (volatile namespaces)
    ● a raw key is mapped to a value

    View Slide

  26. Important Mappings:
    ● obj.__dict__
    ● execution locals, globals, and builtins
    ● sys.modules
    ● os.environ

    View Slide

  27. Syntax:
    obj[key]
    obj[key] = value
    del obj[key]
    key in obj

    View Slide

  28. Some available types:
    ● dict
    ● collections.defaultdict
    ● collections.OrderedDict
    ● configparser.ConfigParser
    ● shelve.Shelf
    ● types.MappingProxyType
    ● weakref.WeakKeyDictionary
    ● weakref.WeakValueDictionary

    View Slide

  29. How they work:
    obj[key]
    obj[key] = value
    del obj[key]
    key in obj
    obj.__getitem__(key)
    obj.__setitem__(key, val)
    obj.__delitem__(key)
    obj.__contains__(key)

    View Slide

  30. How they work:
    obj[key]
    obj[key] = value
    del obj[key]
    key in obj
    collections.abc.(Mutable)Mapping
    obj.__getitem__(key)
    obj.__setitem__(key, val)
    obj.__delitem__(key)
    obj.__contains__(key)

    View Slide

  31. class ReadOnlyProxy(Mapping):
    def __init__(self, mapping):
    self._mapping = mapping
    def __len__(self):
    return len(self._mapping)
    def __iter__(self):
    return iter(self._mapping)
    def __getitem__(self, key):
    return self._mapping[key]

    View Slide

  32. Gems:
    ● dict(mapping, **other_mapping)
    ● d.update(mapping, **other_mapping)
    ● dict.from_keys()
    ● dict.setdefault()
    ● str.formatmap()
    ● operator.itemgetter()
    ● threading.local

    View Slide

  33. Property-based
    Namespaces

    View Slide

  34. property-based containers
    (static namespaces)
    ● a proper name aliases an underlying key
    ● the key is associated with a value
    ● the names and values are fixed in place
    (i.e. read-only)

    View Slide

  35. Example:
    >>> sys.version_info
    sys.version_info(major=3, minor=4, micro=0,
    releaselevel='alpha', serial=0)
    >>> os.times()
    posix.times_result(user=0.09, system=0.04,
    children_user=0.0, children_system=0.0,
    elapsed=17248877.17)

    View Slide

  36. Some available types:
    ● collections.namedtuple
    ● structseq (in C)

    View Slide

  37. namedtuple: context + names
    Spam = namedtuple('Spam', "a b c")
    class Spam(tuple):
    def __new__(cls, a, b, c):
    return super().__new__(cls, a, b, c)
    @property
    def a(self):
    return self[0]
    ...

    View Slide

  38. class ReadOnlyProxy:
    def __init__(self, obj):
    self._obj = obj
    @property
    def spam(self):
    return self._obj.spam
    @property
    def eggs(self):
    return self._obj.eggs

    View Slide

  39. Attribute-based
    Namespaces

    View Slide

  40. attribute-based containers
    (stable namespaces)
    ● a proper name is associated with a value

    View Slide

  41. How they work:
    obj.name
    obj.name = value
    del obj.name
    obj.__getattribute__(name)
    obj.__setattr__(name, val)
    obj.__delattr__(name)

    View Slide

  42. How they work:
    obj.name
    obj.name = value
    del obj.name
    obj.__getattribute__(name)
    obj.__setattr__(name, val)
    obj.__delattr__(name)

    View Slide

  43. How __getattribute__() works:
    1. use a "data" descriptor, if there is one
    2. pull from the instance, if it has __dict__
    3. use a "non-data" descriptor, if there is one
    4. use __getattr__(), if there is one
    5. raise AttributeError

    View Slide

  44. Descriptors
    obj.__get__()
    obj.__set__()
    obj.__delete__()
    (http://docs.python.org/3/howto/descriptor.html)

    View Slide

  45. Descriptors (non-data)
    obj.__get__()
    obj.__set__()
    obj.__delete__()
    (http://docs.python.org/3/howto/descriptor.html)

    View Slide

  46. Descriptors (data)
    obj.__get__()
    obj.__set__()
    obj.__delete__()
    (http://docs.python.org/3/howto/descriptor.html)

    View Slide

  47. Descriptors
    obj.__get__()
    obj.__set__()
    obj.__delete__()
    property
    classmethod
    staticmethod

    View Slide

  48. How __getattribute__() works:
    1. use a "data" descriptor, if there is one
    2. pull from the instance, if it has __dict__
    3. use a "non-data" descriptor, if there is one
    4. use __getattr__(), if there is one
    5. raise AttributeError

    View Slide

  49. How __getattribute__() works:
    1. use a "data" descriptor, if there is one
    2. pull from the instance, if it has __dict__
    3. use a "non-data" descriptor, if there is one
    4. use __getattr__(), if there is one
    5. raise AttributeError
    MRO!

    View Slide

  50. How __getattribute__() works:
    1. use a "data" descriptor, if there is one
    2. pull from the instance, if it has __dict__
    3. use a "non-data" descriptor, if there is one
    4. use __getattr__(), if there is one
    5. raise AttributeError
    Method Resolution Order!

    View Slide

  51. class ReadOnlyProxy:
    def __init__(self, obj):
    self._obj = obj
    def __getattr__(self, name):
    return getattr(self._obj, name)

    View Slide

  52. Some available types:
    ● object
    ● types.SimpleNamespace
    ● argparse.Namespace
    ● multiprocessing.Manager.Namespace
    ● xmlrpc.client.ServerProxy
    ● unittest.Mock

    View Slide

  53. Gems:
    ● __qualname__
    ● operator.attrgetter()

    View Slide

  54. Communicate the nature of your namespace!
    mapping-based
    VOLATILE
    attribute-based
    STABLE
    property-based
    STATIC

    View Slide

  55. Objects

    View Slide

  56. Objects
    ● are namespace containers
    ● have namespaces (mostly)
    ● ergo, as container, proxies a namespace

    View Slide

  57. Objects
    ● are namespace containers
    ● have namespaces (mostly)
    ● ergo, as container, proxies a namespace
    >>> ns = SimpleNamespace()
    >>> ns_ns = vars(ns)
    >>> ns.__dict__ is ns_ns
    True

    View Slide

  58. Objects
    ● are namespace containers
    ● have namespaces (mostly)
    ● ergo, as container, proxies a namespace
    ● base object type has slots
    ● assignment/definition: "bind" object to
    namespace

    View Slide

  59. Object access helpers:
    ● getattr()
    ● setattr()
    ● delattr()
    ● hasattr() wrapper around getattr()
    ● dir() calls obj.__dir__()
    ● vars() exposes namespace*
    ● inspect.getattr_static()
    ● inspect.getmembers()

    View Slide

  60. __dict__
    ● most builtins don't have one
    ● key exception: function objects

    View Slide

  61. __dict__
    ● most builtins don't have one
    ● key exception: function objects
    ● module namespaces exposed as __dict__

    View Slide

  62. __dict__
    ● most builtins don't have one
    ● key exception: function objects
    ● module namespaces exposed as __dict__
    ● read-only proxy of class namespaces
    exposed as __dict__
    ● use metaclass __prepare__() to customize
    the namespace used during class definition

    View Slide

  63. __dict__ vs. __slots__
    ● slots are turned into descriptors
    ● __dict__ is disabled

    View Slide

  64. Execution

    View Slide

  65. Name lookup:
    ● explicit lookup happens relative to containers
    ● Method Resolution Order (MRO)
    ● lookup on class vs. on objects
    ● special method lookup

    View Slide

  66. Name lookup:
    ● explicit lookup happens relative to containers
    ● Method Resolution Order (MRO)
    ● lookup on class vs. on objects
    ● special method lookup
    ● implicit lookup happens relative to execution
    namespaces...

    View Slide

  67. Implicit lookup:
    1. f_locals
    2. f_globals
    3. f_builtins

    View Slide

  68. Scope (a.k.a. implicit lookup):
    1. locals
    ● function execution namespace
    ● use vars() or locals()
    2. non-locals (closures)
    3. globals
    ● module execution namespace
    ● use vars() or globals()
    4. builtins
    ● interpreter execution namespace
    ● use builtins or __builtins__

    View Slide

  69. Interpreter "top-level" namespaces:
    ● implicit execution-globals: builtins
    ● explicit execution-globals: sys.modules

    View Slide

  70. sys.modules
    >>> mordor = sys.modules['mordor']

    View Slide

  71. Interpreter "top-level" namespaces:
    ● implicit execution-globals: builtins
    ● explicit execution-globals: sys.modules

    View Slide

  72. sys.modules
    >>> import mordor
    Traceback (most recent call last):
    File ".../mordor.py", line 14, in
    raise ImportError(boromir_says)
    ImportError: One does not simply import Mordor.

    View Slide

  73. Interpreter "top-level" namespaces:
    ● implicit execution-globals: builtins
    ● explicit execution-globals: sys.modules
    ○ access via import statement

    View Slide

  74. Interpreter "top-level" namespaces:
    ● implicit execution-globals: builtins
    ● explicit execution-globals: sys.modules
    ○ access via import statement
    ● id()

    View Slide

  75. Summary

    View Slide

  76. APIs:
    mapping-based
    ● see collections.abc.Mapping
    property-based
    ● use properties + "private" attributes
    attribute-based
    ● __getattribute__() and __getattr__()
    ● __setattr__()
    ● __delattr__()

    View Slide

  77. Key object attributes:
    ● __dict__
    ● __dir__()
    ● __slots__()
    ● __mro__ (for classes)

    View Slide

  78. Helpers:
    ● globals()
    ● locals()
    ● vars()
    ● dir()
    ● getattr()
    ● setattr()
    ● delattr()
    ● hasattr()
    ● inspect.getattr_static()
    ● id()

    View Slide

  79. ● namespaces: context + name => value
    ● containers:
    ○ mapping-based (volatile)
    ○ property-based (static)
    ○ attribute-based (stable)
    ● pervasive in Python
    ● rich APIs
    ● rich tools for objects

    View Slide

  80. Questions?
    Eric Snow
    Fusion-io
    [email protected]
    Slides:
    http://goo.gl/gnhXt

    View Slide