What happens behind execution of an import statement?

What happens behind execution of an import statement?

0f2e84dbe50c7b470a843853fde090b2?s=128

Shivashis Padhi

August 16, 2020
Tweet

Transcript

  1. What happens behind execution of an import statement? - Shivashis

    Padhi
  2. Who am I? - data/software at Grofers - tinker with

    earth observation data - past GSoC with the PSF - love FOSS. worked on felicette, MSS
  3. None
  4. xkcd/353

  5. Packages in Python

  6. Packages ‣ directories? not necessarily ‣ special types of modules

    ‣ the __path__ attribute ‣ What are subpackages?
  7. None
  8. But how can we import a package from a different

    directory?
  9. $PATH, and $PYTHONPATH

  10. None
  11. Regular and Namespace packages

  12. Regular packages

  13. Namespace packages

  14. Namespace packages - example

  15. Types of packages supported by default ‣ built-in modules ‣

    frozen modules ‣ path entries in $PATH, and $PYTHONPATH
  16. The import system

  17. sys.modules ‣ A dictionary to map module names to modules

    which are ‘already loaded’. ‣ Lives in the global namespace of sys
  18. sys.modules

  19. importlib ‣ provide the implementation of the import statement ‣

    expose parts of the import system for programmatic usage
  20. Common use-cases of importlib ‣ Programmatically loading a module ‣

    Reloading a module in run-time ‣ Base classes for finders and loaders (to be used with import hooks)
  21. Finders and loaders

  22. Finders ‣ find the module ‣ return a spec object

    ‣ Finder object must have a find_spec method defined.
  23. Finders - examples ‣ locate built-in modules - BuiltinImporter ‣

    locate frozen modules - FrozenImporter ‣ import path - imagine $PATH and $PYTHONPATH - PathFinder
  24. Finders - function ‣ If a finder is capable of

    handling import, it returns a module spec. ‣ module spec contains module’s metadata i.e path, name, etc. The import machinery uses this spec to load modules later.
  25. find_spec

  26. importing spam.ham

  27. Import hooks and PEP 302

  28. sys.path_hooks sys.meta_path

  29. an example of sys.meta_path and sys.path_hooks

  30. What are import hooks? ‣ Inserting custom finders and loaders

    into the import mechanism ‣ Dependency injection?
  31. Loaders After spec object is returned from a finder, loaders

    are used to execute the module and store its reference in sys.modules
  32. Tasks of loaders ‣ load from source/byte-code ‣ compile source

    ‣ execute the module and store reference in sys.modules[spec.name]
  33. example - loading

  34. example - loading

  35. example - loading

  36. ImportError ModuleNotFoundError When no finder can find a module When

    loading fails for a package/module
  37. chronological order

  38. Code demonstration of an import hooks’ implementation.

  39. What else can you do with import hooks? ‣ full

    control after what happens after import x ‣ importing from ftp, https, authorization checks. ‣ importing from a variety of storage, not just .py/.pyc
  40. How have people used import hooks? ‣ https://github.com/kragniz/json-sempai ‣ https://github.com/nvbn/import_from_github_com

    ‣ https://github.com/runsascoded/ur
  41. sys.meta_path with virtual environment disabled

  42. sys.meta_path with virtual environment enabled Virtual Environment’s finder

  43. Thank you! https://shivashis.xyz