Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Import This, That, and the Other Thing

Import This, That, and the Other Thing

PyCon 2010


Brett Cannon

February 12, 2010


  1. If you do not know what __path__ is, this talk

    is NOT for you. Sorry.
  2. import this, that, and the other thing Custom importers in

    Python Brett Cannon www.DrBrett.ca brett@python.org
  3. Thanks ... • Python Software Foundation • PyCon Financial Aid

    committee • Nasuni • Jesse Noller
  4. What the heck is an importer?

  5. importer = finder + loader

  6. A finder finds modules.

  7. A loader loads modules.

  8. “Why do I want one?”

  9. How are custom importers used by import?

  10. Meta path sys.meta_path

  11. Start for finder in sys.meta_path: loader = finder.find_module(name, path) return

    loader.load_module(name) ... True False
  12. Path sys.path or __path__, sys.path_hooks, & sys.path_importer_cache

  13. ... Parent module has __path__ search = parent's __path__ search

    = sys.path search True False
  14. for entry in search: finder = sys.path_importer_cache [entry] loader =

    finder.find_module(name) return loader.load_module(name) Search path hook False True False finder raise ImportError True
  15. for hook in sys.path_hooks: finder = hook(entry) sys.path_importer_cache[entry] = finder

    sys.path_importer_cache[entry] = dummy path hook finder False True
  16. how do I write my own importer? Only masochists need

  17. Option 1: Painfully from scratch Read PEP 302 for the

    gory details.
  18. Option 2: Use importlib Available since Python 3.1. I have

    suffered so you don’t have to.
  19. Option 3: importers http://packages.python.org/importers/ File path abstraction on top of

    importlib. Treating as purgatory for importlib inclusion.
  20. Using a zipfile importer as an example

  21. we need a hook For sys.path_hooks.

  22. Refresher: Hooks look for a finder for a path

  23. Hooks can get funky paths E.g. /path/to/file/code.zip/some/pkg

  24. Consider caching archive file objects

  25. Pass your finder the “location”: 1)the path/object & 2) the

    package path
  26. Raise ImportError if you got nuthin’

  27. Have finder, will look for code

  28. Don’t treat modules as code but as files

  29. You did remember where in the package you are looking,

  30. fullname.rpartition(‘.’)[-1]

  31. Need to care about packages & modules some/pkg/name/__init__.py and some/pkg/name.py

  32. Avoid caching within a finder Blame sys.path_importer_cache

  33. Tell the loader if package & path to code Don’t

    Repeat Yourself ... within reason.
  34. Nuthin’? Give back None

  35. Now it gets tricky Writing a loader.

  36. Are you still thinking in terms of file paths?

  37. importlib.abc.PyLoader • source_path() • Might be changing... • is_package() •

  38. importlib.abc.PyPycLoader • source_path() • is_package() • get_data() • source_mtime() •

    bytecode_path() • Might be changing...
  39. Reasons to ignore .pyc • Jython, IronPython couldn’t care less.

    • Safe to support, though. • Another thing to code up. • Bytecode is just an optimization. • If you only ship .pyc for code protection, stop it.
  40. What to do when using importlib ABCs

  41. Require anchor point for paths somewhere/mod.py is too ambiguous

  42. Consider caching stat calls Only for stand-alone loaders! Also consider

    caching if package or not.
  43. Don’t overdo error checking EAFP is your friend.

  44. Perk of importers is the abstraction

  45. Lazy loader mix-in written in 19 lines

  46. class Module(types.ModuleType): pass class Mixin: def load_module(self, name): if name

    in sys.modules: return super().load_module(name) # Create a lazy module that will type check. module = LazyModule(name) # Set the loader on the module as ModuleType will not. module.__loader__ = self # Insert the module into sys.modules. sys.modules[name] = module return module class LazyModule(types.ModuleType): def __getattribute__(self, attr): # Remove this __getattribute__ method by re-assigning. self.__class__ = Module # Fetch the real loader. self.__loader__ = super(Mixin, self.__loader__) # Actually load the module. self.__loader__.load_module(self.__name__) # Return the requested attribute. return getattr(self, attr)
  47. ... or you could use the importers package http://packages.python.org/importers/

  48. Fin