Namespaces in Python by Eric Snow

Transcript

1. View Slide

2. Eric Snow
   Fusion-io
   ericsnowcurrently@gmail.com
   Slides:
   http://goo.gl/gnhXt
   

   View Slide

3. Namespaces in Python
   

   View Slide

4. "Namespaces are one honking great idea --
   let's do more of those!"
   - Tim Peters, The Zen of Python
   

   View Slide

5. What do we mean, "namespaces"?
   What do namespaces look like?
   How are they used?
   How do they work?
   

   View Slide

6. What are Namespaces?
   

   View Slide

7. A concept:
   a set of independently identifiable expectations
   +
   a unique (within set) label for each member
   

   View Slide

8. A concept:
   a set of independently identifiable expectations
   +
   a unique (within set) label for each member
   context + name => value
   

   View Slide

9. Nested namespaces?
   with nesting without nesting
   spam spam
   sys.version_info sys_version_info
   os.path.exists() os_path_exists()
   

   View Slide

10. "Flat is better than nested."
    - Tim Peters, The Zen of Python
    

    View Slide

11. Namespaces in C?
    PyObject
    Py_TYPE
    PyDict_SetItem
    PyErr_SetString
    Py_None
    

    View Slide

12. Namespaces in C?
    PyObject
    Py_TYPE
    PyDict_SetItem
    PyErr_SetString
    Py_None
    Py . Object
    Py . TYPE
    Py . Dict . SetItem
    Py . Err . SetString
    Py . None
    

    View Slide

13. Namespace Containers
    

    View Slide

14. Namespace Containers
    (Image CC http://www.flickr.
    com/photos/greeblie/)
    

    View Slide

15. Namespace Containers
    mapping-based
    attribute-based
    property-based
    

    View Slide

16. Namespace Containers
    mapping-based
    VOLATILE
    attribute-based
    STABLE
    property-based
    STATIC
    

    View Slide

17. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    

    View Slide

18. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    ● the type must communicate the nature of the
    namespace
    

    View Slide

19. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    ● the type must communicate the nature of the
    namespace
    ● dict too soft
    

    View Slide

20. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    ● the type must communicate the nature of the
    namespace
    ● dict too soft
    ● namedtuple too hard
    

    View Slide

21. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    ● the type must communicate the nature of the
    namespace
    ● dict too soft
    ● namedtuple too hard
    ● object just right...
    

    View Slide

22. A Type for sys.implementation
    ● Python implementation specifics: PEP 421
    ● the type must communicate the nature of the
    namespace
    ● dict too soft
    ● namedtuple too hard
    ● object just right...
    ● types.SimpleNamespace
    

    View Slide

23. Communicate the nature of your namespace!
    mapping-based
    VOLATILE
    attribute-based
    STABLE
    property-based
    STATIC
    

    View Slide

24. Mapping-based
    Namespaces
    

    View Slide

25. mapping-based containers
    (volatile namespaces)
    ● a raw key is mapped to a value
    

    View Slide

26. Important Mappings:
    ● obj.__dict__
    ● execution locals, globals, and builtins
    ● sys.modules
    ● os.environ
    

    View Slide

27. Syntax:
    obj[key]
    obj[key] = value
    del obj[key]
    key in obj
    

    View Slide

28. Some available types:
    ● dict
    ● collections.defaultdict
    ● collections.OrderedDict
    ● configparser.ConfigParser
    ● shelve.Shelf
    ● types.MappingProxyType
    ● weakref.WeakKeyDictionary
    ● weakref.WeakValueDictionary
    

    View Slide

29. How they work:
    obj[key]
    obj[key] = value
    del obj[key]
    key in obj
    obj.__getitem__(key)
    obj.__setitem__(key, val)
    obj.__delitem__(key)
    obj.__contains__(key)
    

    View Slide

30. How they work:
    obj[key]
    obj[key] = value
    del obj[key]
    key in obj
    collections.abc.(Mutable)Mapping
    obj.__getitem__(key)
    obj.__setitem__(key, val)
    obj.__delitem__(key)
    obj.__contains__(key)
    

    View Slide

31. class ReadOnlyProxy(Mapping):
    def __init__(self, mapping):
    self._mapping = mapping
    def __len__(self):
    return len(self._mapping)
    def __iter__(self):
    return iter(self._mapping)
    def __getitem__(self, key):
    return self._mapping[key]
    

    View Slide

32. Gems:
    ● dict(mapping, **other_mapping)
    ● d.update(mapping, **other_mapping)
    ● dict.from_keys()
    ● dict.setdefault()
    ● str.formatmap()
    ● operator.itemgetter()
    ● threading.local
    

    View Slide

33. Property-based
    Namespaces
    

    View Slide

34. property-based containers
    (static namespaces)
    ● a proper name aliases an underlying key
    ● the key is associated with a value
    ● the names and values are fixed in place
    (i.e. read-only)
    

    View Slide

35. Example:
    >>> sys.version_info
    sys.version_info(major=3, minor=4, micro=0,
    releaselevel='alpha', serial=0)
    >>> os.times()
    posix.times_result(user=0.09, system=0.04,
    children_user=0.0, children_system=0.0,
    elapsed=17248877.17)
    

    View Slide

36. Some available types:
    ● collections.namedtuple
    ● structseq (in C)
    

    View Slide

37. namedtuple: context + names
    Spam = namedtuple('Spam', "a b c")
    class Spam(tuple):
    def __new__(cls, a, b, c):
    return super().__new__(cls, a, b, c)
    @property
    def a(self):
    return self[0]
    ...
    

    View Slide

38. class ReadOnlyProxy:
    def __init__(self, obj):
    self._obj = obj
    @property
    def spam(self):
    return self._obj.spam
    @property
    def eggs(self):
    return self._obj.eggs
    

    View Slide

39. Attribute-based
    Namespaces
    

    View Slide

40. attribute-based containers
    (stable namespaces)
    ● a proper name is associated with a value
    

    View Slide

41. How they work:
    obj.name
    obj.name = value
    del obj.name
    obj.__getattribute__(name)
    obj.__setattr__(name, val)
    obj.__delattr__(name)
    

    View Slide

42. How they work:
    obj.name
    obj.name = value
    del obj.name
    obj.__getattribute__(name)
    obj.__setattr__(name, val)
    obj.__delattr__(name)
    

    View Slide

43. How __getattribute__() works:
    1. use a "data" descriptor, if there is one
    2. pull from the instance, if it has __dict__
    3. use a "non-data" descriptor, if there is one
    4. use __getattr__(), if there is one
    5. raise AttributeError
    

    View Slide

44. Descriptors
    obj.__get__()
    obj.__set__()
    obj.__delete__()
    (http://docs.python.org/3/howto/descriptor.html)
    

    View Slide

45. Descriptors (non-data)
    obj.__get__()
    obj.__set__()
    obj.__delete__()
    (http://docs.python.org/3/howto/descriptor.html)
    

    View Slide

46. Descriptors (data)
    obj.__get__()
    obj.__set__()
    obj.__delete__()
    (http://docs.python.org/3/howto/descriptor.html)
    

    View Slide

47. Descriptors
    obj.__get__()
    obj.__set__()
    obj.__delete__()
    property
    classmethod
    staticmethod
    

    View Slide

48. How __getattribute__() works:
    1. use a "data" descriptor, if there is one
    2. pull from the instance, if it has __dict__
    3. use a "non-data" descriptor, if there is one
    4. use __getattr__(), if there is one
    5. raise AttributeError
    

    View Slide

49. How __getattribute__() works:
    1. use a "data" descriptor, if there is one
    2. pull from the instance, if it has __dict__
    3. use a "non-data" descriptor, if there is one
    4. use __getattr__(), if there is one
    5. raise AttributeError
    MRO!
    

    View Slide

50. How __getattribute__() works:
    1. use a "data" descriptor, if there is one
    2. pull from the instance, if it has __dict__
    3. use a "non-data" descriptor, if there is one
    4. use __getattr__(), if there is one
    5. raise AttributeError
    Method Resolution Order!
    

    View Slide

51. class ReadOnlyProxy:
    def __init__(self, obj):
    self._obj = obj
    def __getattr__(self, name):
    return getattr(self._obj, name)
    

    View Slide

52. Some available types:
    ● object
    ● types.SimpleNamespace
    ● argparse.Namespace
    ● multiprocessing.Manager.Namespace
    ● xmlrpc.client.ServerProxy
    ● unittest.Mock
    

    View Slide

53. Gems:
    ● __qualname__
    ● operator.attrgetter()
    

    View Slide

54. Communicate the nature of your namespace!
    mapping-based
    VOLATILE
    attribute-based
    STABLE
    property-based
    STATIC
    

    View Slide

55. Objects
    

    View Slide

56. Objects
    ● are namespace containers
    ● have namespaces (mostly)
    ● ergo, as container, proxies a namespace
    

    View Slide

57. Objects
    ● are namespace containers
    ● have namespaces (mostly)
    ● ergo, as container, proxies a namespace
    >>> ns = SimpleNamespace()
    >>> ns_ns = vars(ns)
    >>> ns.__dict__ is ns_ns
    True
    

    View Slide

58. Objects
    ● are namespace containers
    ● have namespaces (mostly)
    ● ergo, as container, proxies a namespace
    ● base object type has slots
    ● assignment/definition: "bind" object to
    namespace
    

    View Slide

59. Object access helpers:
    ● getattr()
    ● setattr()
    ● delattr()
    ● hasattr() wrapper around getattr()
    ● dir() calls obj.__dir__()
    ● vars() exposes namespace*
    ● inspect.getattr_static()
    ● inspect.getmembers()
    

    View Slide

60. __dict__
    ● most builtins don't have one
    ● key exception: function objects
    

    View Slide

61. __dict__
    ● most builtins don't have one
    ● key exception: function objects
    ● module namespaces exposed as __dict__
    

    View Slide

62. __dict__
    ● most builtins don't have one
    ● key exception: function objects
    ● module namespaces exposed as __dict__
    ● read-only proxy of class namespaces
    exposed as __dict__
    ● use metaclass __prepare__() to customize
    the namespace used during class definition
    

    View Slide

63. __dict__ vs. __slots__
    ● slots are turned into descriptors
    ● __dict__ is disabled
    

    View Slide

64. Execution
    

    View Slide

65. Name lookup:
    ● explicit lookup happens relative to containers
    ● Method Resolution Order (MRO)
    ● lookup on class vs. on objects
    ● special method lookup
    

    View Slide

66. Name lookup:
    ● explicit lookup happens relative to containers
    ● Method Resolution Order (MRO)
    ● lookup on class vs. on objects
    ● special method lookup
    ● implicit lookup happens relative to execution
    namespaces...
    

    View Slide

67. Implicit lookup:
    1. f_locals
    2. f_globals
    3. f_builtins
    

    View Slide

68. Scope (a.k.a. implicit lookup):
    1. locals
    ● function execution namespace
    ● use vars() or locals()
    2. non-locals (closures)
    3. globals
    ● module execution namespace
    ● use vars() or globals()
    4. builtins
    ● interpreter execution namespace
    ● use builtins or __builtins__
    

    View Slide

69. Interpreter "top-level" namespaces:
    ● implicit execution-globals: builtins
    ● explicit execution-globals: sys.modules
    

    View Slide

70. sys.modules
    >>> mordor = sys.modules['mordor']
    

    View Slide

71. Interpreter "top-level" namespaces:
    ● implicit execution-globals: builtins
    ● explicit execution-globals: sys.modules
    

    View Slide

72. sys.modules
    >>> import mordor
    Traceback (most recent call last):
    File ".../mordor.py", line 14, in
    raise ImportError(boromir_says)
    ImportError: One does not simply import Mordor.
    

    View Slide

73. Interpreter "top-level" namespaces:
    ● implicit execution-globals: builtins
    ● explicit execution-globals: sys.modules
    ○ access via import statement
    

    View Slide

74. Interpreter "top-level" namespaces:
    ● implicit execution-globals: builtins
    ● explicit execution-globals: sys.modules
    ○ access via import statement
    ● id()
    

    View Slide

75. Summary
    

    View Slide

76. APIs:
    mapping-based
    ● see collections.abc.Mapping
    property-based
    ● use properties + "private" attributes
    attribute-based
    ● __getattribute__() and __getattr__()
    ● __setattr__()
    ● __delattr__()
    

    View Slide

77. Key object attributes:
    ● __dict__
    ● __dir__()
    ● __slots__()
    ● __mro__ (for classes)
    

    View Slide

78. Helpers:
    ● globals()
    ● locals()
    ● vars()
    ● dir()
    ● getattr()
    ● setattr()
    ● delattr()
    ● hasattr()
    ● inspect.getattr_static()
    ● id()
    

    View Slide

79. ● namespaces: context + name => value
    ● containers:
    ○ mapping-based (volatile)
    ○ property-based (static)
    ○ attribute-based (stable)
    ● pervasive in Python
    ● rich APIs
    ● rich tools for objects
    

    View Slide

80. Questions?
    Eric Snow
    Fusion-io
    ericsnowcurrently@gmail.com
    Slides:
    http://goo.gl/gnhXt
    

    View Slide