Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Eric Snow Fusion-io ericsnowcurrently@gmail.com Slides: http://goo.gl/gnhXt

Slide 3

Slide 3 text

Namespaces in Python

Slide 4

Slide 4 text

"Namespaces are one honking great idea -- let's do more of those!" - Tim Peters, The Zen of Python

Slide 5

Slide 5 text

What do we mean, "namespaces"? What do namespaces look like? How are they used? How do they work?

Slide 6

Slide 6 text

What are Namespaces?

Slide 7

Slide 7 text

A concept: a set of independently identifiable expectations + a unique (within set) label for each member

Slide 8

Slide 8 text

A concept: a set of independently identifiable expectations + a unique (within set) label for each member context + name => value

Slide 9

Slide 9 text

Nested namespaces? with nesting without nesting spam spam sys.version_info sys_version_info os.path.exists() os_path_exists()

Slide 10

Slide 10 text

"Flat is better than nested." - Tim Peters, The Zen of Python

Slide 11

Slide 11 text

Namespaces in C? PyObject Py_TYPE PyDict_SetItem PyErr_SetString Py_None

Slide 12

Slide 12 text

Namespaces in C? PyObject Py_TYPE PyDict_SetItem PyErr_SetString Py_None Py . Object Py . TYPE Py . Dict . SetItem Py . Err . SetString Py . None

Slide 13

Slide 13 text

Namespace Containers

Slide 14

Slide 14 text

Namespace Containers (Image CC http://www.flickr. com/photos/greeblie/)

Slide 15

Slide 15 text

Namespace Containers mapping-based attribute-based property-based

Slide 16

Slide 16 text

Namespace Containers mapping-based VOLATILE attribute-based STABLE property-based STATIC

Slide 17

Slide 17 text

A Type for sys.implementation ● Python implementation specifics: PEP 421

Slide 18

Slide 18 text

A Type for sys.implementation ● Python implementation specifics: PEP 421 ● the type must communicate the nature of the namespace

Slide 19

Slide 19 text

A Type for sys.implementation ● Python implementation specifics: PEP 421 ● the type must communicate the nature of the namespace ● dict too soft

Slide 20

Slide 20 text

A Type for sys.implementation ● Python implementation specifics: PEP 421 ● the type must communicate the nature of the namespace ● dict too soft ● namedtuple too hard

Slide 21

Slide 21 text

A Type for sys.implementation ● Python implementation specifics: PEP 421 ● the type must communicate the nature of the namespace ● dict too soft ● namedtuple too hard ● object just right...

Slide 22

Slide 22 text

A Type for sys.implementation ● Python implementation specifics: PEP 421 ● the type must communicate the nature of the namespace ● dict too soft ● namedtuple too hard ● object just right... ● types.SimpleNamespace

Slide 23

Slide 23 text

Communicate the nature of your namespace! mapping-based VOLATILE attribute-based STABLE property-based STATIC

Slide 24

Slide 24 text

Mapping-based Namespaces

Slide 25

Slide 25 text

mapping-based containers (volatile namespaces) ● a raw key is mapped to a value

Slide 26

Slide 26 text

Important Mappings: ● obj.__dict__ ● execution locals, globals, and builtins ● sys.modules ● os.environ

Slide 27

Slide 27 text

Syntax: obj[key] obj[key] = value del obj[key] key in obj

Slide 28

Slide 28 text

Some available types: ● dict ● collections.defaultdict ● collections.OrderedDict ● configparser.ConfigParser ● shelve.Shelf ● types.MappingProxyType ● weakref.WeakKeyDictionary ● weakref.WeakValueDictionary

Slide 29

Slide 29 text

How they work: obj[key] obj[key] = value del obj[key] key in obj obj.__getitem__(key) obj.__setitem__(key, val) obj.__delitem__(key) obj.__contains__(key)

Slide 30

Slide 30 text

How they work: obj[key] obj[key] = value del obj[key] key in obj collections.abc.(Mutable)Mapping obj.__getitem__(key) obj.__setitem__(key, val) obj.__delitem__(key) obj.__contains__(key)

Slide 31

Slide 31 text

class ReadOnlyProxy(Mapping): def __init__(self, mapping): self._mapping = mapping def __len__(self): return len(self._mapping) def __iter__(self): return iter(self._mapping) def __getitem__(self, key): return self._mapping[key]

Slide 32

Slide 32 text

Gems: ● dict(mapping, **other_mapping) ● d.update(mapping, **other_mapping) ● dict.from_keys() ● dict.setdefault() ● str.formatmap() ● operator.itemgetter() ● threading.local

Slide 33

Slide 33 text

Property-based Namespaces

Slide 34

Slide 34 text

property-based containers (static namespaces) ● a proper name aliases an underlying key ● the key is associated with a value ● the names and values are fixed in place (i.e. read-only)

Slide 35

Slide 35 text

Example: >>> sys.version_info sys.version_info(major=3, minor=4, micro=0, releaselevel='alpha', serial=0) >>> os.times() posix.times_result(user=0.09, system=0.04, children_user=0.0, children_system=0.0, elapsed=17248877.17)

Slide 36

Slide 36 text

Some available types: ● collections.namedtuple ● structseq (in C)

Slide 37

Slide 37 text

namedtuple: context + names Spam = namedtuple('Spam', "a b c") class Spam(tuple): def __new__(cls, a, b, c): return super().__new__(cls, a, b, c) @property def a(self): return self[0] ...

Slide 38

Slide 38 text

class ReadOnlyProxy: def __init__(self, obj): self._obj = obj @property def spam(self): return self._obj.spam @property def eggs(self): return self._obj.eggs

Slide 39

Slide 39 text

Attribute-based Namespaces

Slide 40

Slide 40 text

attribute-based containers (stable namespaces) ● a proper name is associated with a value

Slide 41

Slide 41 text

How they work: obj.name obj.name = value del obj.name obj.__getattribute__(name) obj.__setattr__(name, val) obj.__delattr__(name)

Slide 42

Slide 42 text

How they work: obj.name obj.name = value del obj.name obj.__getattribute__(name) obj.__setattr__(name, val) obj.__delattr__(name)

Slide 43

Slide 43 text

How __getattribute__() works: 1. use a "data" descriptor, if there is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError

Slide 44

Slide 44 text

Descriptors obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)

Slide 45

Slide 45 text

Descriptors (non-data) obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)

Slide 46

Slide 46 text

Descriptors (data) obj.__get__() obj.__set__() obj.__delete__() (http://docs.python.org/3/howto/descriptor.html)

Slide 47

Slide 47 text

Descriptors obj.__get__() obj.__set__() obj.__delete__() property classmethod staticmethod

Slide 48

Slide 48 text

How __getattribute__() works: 1. use a "data" descriptor, if there is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError

Slide 49

Slide 49 text

How __getattribute__() works: 1. use a "data" descriptor, if there is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError MRO!

Slide 50

Slide 50 text

How __getattribute__() works: 1. use a "data" descriptor, if there is one 2. pull from the instance, if it has __dict__ 3. use a "non-data" descriptor, if there is one 4. use __getattr__(), if there is one 5. raise AttributeError Method Resolution Order!

Slide 51

Slide 51 text

class ReadOnlyProxy: def __init__(self, obj): self._obj = obj def __getattr__(self, name): return getattr(self._obj, name)

Slide 52

Slide 52 text

Some available types: ● object ● types.SimpleNamespace ● argparse.Namespace ● multiprocessing.Manager.Namespace ● xmlrpc.client.ServerProxy ● unittest.Mock

Slide 53

Slide 53 text

Gems: ● __qualname__ ● operator.attrgetter()

Slide 54

Slide 54 text

Communicate the nature of your namespace! mapping-based VOLATILE attribute-based STABLE property-based STATIC

Slide 55

Slide 55 text

Objects

Slide 56

Slide 56 text

Objects ● are namespace containers ● have namespaces (mostly) ● ergo, as container, proxies a namespace

Slide 57

Slide 57 text

Objects ● are namespace containers ● have namespaces (mostly) ● ergo, as container, proxies a namespace >>> ns = SimpleNamespace() >>> ns_ns = vars(ns) >>> ns.__dict__ is ns_ns True

Slide 58

Slide 58 text

Objects ● are namespace containers ● have namespaces (mostly) ● ergo, as container, proxies a namespace ● base object type has slots ● assignment/definition: "bind" object to namespace

Slide 59

Slide 59 text

Object access helpers: ● getattr() ● setattr() ● delattr() ● hasattr() wrapper around getattr() ● dir() calls obj.__dir__() ● vars() exposes namespace* ● inspect.getattr_static() ● inspect.getmembers()

Slide 60

Slide 60 text

__dict__ ● most builtins don't have one ● key exception: function objects

Slide 61

Slide 61 text

__dict__ ● most builtins don't have one ● key exception: function objects ● module namespaces exposed as __dict__

Slide 62

Slide 62 text

__dict__ ● most builtins don't have one ● key exception: function objects ● module namespaces exposed as __dict__ ● read-only proxy of class namespaces exposed as __dict__ ● use metaclass __prepare__() to customize the namespace used during class definition

Slide 63

Slide 63 text

__dict__ vs. __slots__ ● slots are turned into descriptors ● __dict__ is disabled

Slide 64

Slide 64 text

Execution

Slide 65

Slide 65 text

Name lookup: ● explicit lookup happens relative to containers ● Method Resolution Order (MRO) ● lookup on class vs. on objects ● special method lookup

Slide 66

Slide 66 text

Name lookup: ● explicit lookup happens relative to containers ● Method Resolution Order (MRO) ● lookup on class vs. on objects ● special method lookup ● implicit lookup happens relative to execution namespaces...

Slide 67

Slide 67 text

Implicit lookup: 1. f_locals 2. f_globals 3. f_builtins

Slide 68

Slide 68 text

Scope (a.k.a. implicit lookup): 1. locals ● function execution namespace ● use vars() or locals() 2. non-locals (closures) 3. globals ● module execution namespace ● use vars() or globals() 4. builtins ● interpreter execution namespace ● use builtins or __builtins__

Slide 69

Slide 69 text

Interpreter "top-level" namespaces: ● implicit execution-globals: builtins ● explicit execution-globals: sys.modules

Slide 70

Slide 70 text

sys.modules >>> mordor = sys.modules['mordor']

Slide 71

Slide 71 text

Interpreter "top-level" namespaces: ● implicit execution-globals: builtins ● explicit execution-globals: sys.modules

Slide 72

Slide 72 text

sys.modules >>> import mordor Traceback (most recent call last): File ".../mordor.py", line 14, in raise ImportError(boromir_says) ImportError: One does not simply import Mordor.

Slide 73

Slide 73 text

Interpreter "top-level" namespaces: ● implicit execution-globals: builtins ● explicit execution-globals: sys.modules ○ access via import statement

Slide 74

Slide 74 text

Interpreter "top-level" namespaces: ● implicit execution-globals: builtins ● explicit execution-globals: sys.modules ○ access via import statement ● id()

Slide 75

Slide 75 text

Summary

Slide 76

Slide 76 text

APIs: mapping-based ● see collections.abc.Mapping property-based ● use properties + "private" attributes attribute-based ● __getattribute__() and __getattr__() ● __setattr__() ● __delattr__()

Slide 77

Slide 77 text

Key object attributes: ● __dict__ ● __dir__() ● __slots__() ● __mro__ (for classes)

Slide 78

Slide 78 text

Helpers: ● globals() ● locals() ● vars() ● dir() ● getattr() ● setattr() ● delattr() ● hasattr() ● inspect.getattr_static() ● id()

Slide 79

Slide 79 text

● namespaces: context + name => value ● containers: ○ mapping-based (volatile) ○ property-based (static) ○ attribute-based (stable) ● pervasive in Python ● rich APIs ● rich tools for objects

Slide 80

Slide 80 text

Questions? Eric Snow Fusion-io ericsnowcurrently@gmail.com Slides: http://goo.gl/gnhXt