Import This, That, and the Other Thing

PyCon 2010

Transcript

1. If you do not know what __path__ is, this talk is NOT for you. Sorry.
2. import this, that, and the other thing Custom importers in Python Brett Cannon www.DrBrett.ca brett@python.org
3. Thanks ... • Python Software Foundation • PyCon Financial Aid committee • Nasuni • Jesse Noller
4. What the heck is an importer?
5. importer = finder + loader
6. A finder finds modules.
7. A loader loads modules.
8. “Why do I want one?”
9. How are custom importers used by import?
10. Meta path sys.meta_path
11. Start for finder in sys.meta_path: loader = finder.find_module(name, path) return loader.load_module(name) ... True False
12. Path sys.path or __path__, sys.path_hooks, & sys.path_importer_cache
13. ... Parent module has __path__ search = parent's __path__ search = sys.path search True False
14. for entry in search: finder = sys.path_importer_cache [entry] loader = finder.find_module(name) return loader.load_module(name) Search path hook False True False finder raise ImportError True
15. for hook in sys.path_hooks: finder = hook(entry) sys.path_importer_cache[entry] = finder sys.path_importer_cache[entry] = dummy path hook finder False True
16. how do I write my own importer? Only masochists need apply.
17. Option 1: Painfully from scratch Read PEP 302 for the gory details.
18. Option 2: Use importlib Available since Python 3.1. I have suffered so you don’t have to.
19. Option 3: importers http://packages.python.org/importers/ File path abstraction on top of importlib. Treating as purgatory for importlib inclusion.
20. Using a zipfile importer as an example
21. we need a hook For sys.path_hooks.
22. Refresher: Hooks look for a finder for a path
23. Hooks can get funky paths E.g. /path/to/file/code.zip/some/pkg
24. Consider caching archive file objects
25. Pass your finder the “location”: 1)the path/object & 2) the package path
26. Raise ImportError if you got nuthin’
27. Have finder, will look for code
28. Don’t treat modules as code but as files
29. You did remember where in the package you are looking, RIGHT?!?
30. fullname.rpartition(‘.’)[-1]
31. Need to care about packages & modules some/pkg/name/__init__.py and some/pkg/name.py
32. Avoid caching within a finder Blame sys.path_importer_cache
33. Tell the loader if package & path to code Don’t Repeat Yourself ... within reason.
34. Nuthin’? Give back None
35. Now it gets tricky Writing a loader.
36. Are you still thinking in terms of file paths?
37. importlib.abc.PyLoader • source_path() • Might be changing... • is_package() • get_data()
38. importlib.abc.PyPycLoader • source_path() • is_package() • get_data() • source_mtime() • bytecode_path() • Might be changing...
39. Reasons to ignore .pyc • Jython, IronPython couldn’t care less. • Safe to support, though. • Another thing to code up. • Bytecode is just an optimization. • If you only ship .pyc for code protection, stop it.
40. What to do when using importlib ABCs
41. Require anchor point for paths somewhere/mod.py is too ambiguous
42. Consider caching stat calls Only for stand-alone loaders! Also consider caching if package or not.
43. Don’t overdo error checking EAFP is your friend.
44. Perk of importers is the abstraction
45. Lazy loader mix-in written in 19 lines
46. class Module(types.ModuleType): pass class Mixin: def load_module(self, name): if name in sys.modules: return super().load_module(name) # Create a lazy module that will type check. module = LazyModule(name) # Set the loader on the module as ModuleType will not. module.__loader__ = self # Insert the module into sys.modules. sys.modules[name] = module return module class LazyModule(types.ModuleType): def __getattribute__(self, attr): # Remove this __getattribute__ method by re-assigning. self.__class__ = Module # Fetch the real loader. self.__loader__ = super(Mixin, self.__loader__) # Actually load the module. self.__loader__.load_module(self.__name__) # Return the requested attribute. return getattr(self, attr)
47. ... or you could use the importers package http://packages.python.org/importers/
48. Fin