Slide 1
Slide 1 text
What happens behind execution
of an import statement?
- Shivashis Padhi
Slide 2
Slide 2 text
Who am I?
- data/software at Grofers
- tinker with earth observation data
- past GSoC with the PSF
- love FOSS. worked on felicette, MSS
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
xkcd/353
Slide 5
Slide 5 text
Packages in Python
Slide 6
Slide 6 text
Packages
‣ directories? not necessarily
‣ special types of modules
‣ the __path__ attribute
‣ What are subpackages?
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
But how can we import a package from a different directory?
Slide 9
Slide 9 text
$PATH, and $PYTHONPATH
Slide 10
Slide 10 text
No content
Slide 11
Slide 11 text
Regular and Namespace packages
Slide 12
Slide 12 text
Regular packages
Slide 13
Slide 13 text
Namespace packages
Slide 14
Slide 14 text
Namespace packages - example
Slide 15
Slide 15 text
Types of packages supported by default
‣ built-in modules
‣ frozen modules
‣ path entries in $PATH, and $PYTHONPATH
Slide 16
Slide 16 text
The import system
Slide 17
Slide 17 text
sys.modules
‣ A dictionary to map module names to
modules which are ‘already loaded’.
‣ Lives in the global namespace of sys
Slide 18
Slide 18 text
sys.modules
Slide 19
Slide 19 text
importlib
‣ provide the implementation of the import
statement
‣ expose parts of the import system for
programmatic usage
Slide 20
Slide 20 text
Common use-cases of importlib
‣ Programmatically loading a module
‣ Reloading a module in run-time
‣ Base classes for finders and loaders (to be
used with import hooks)
Slide 21
Slide 21 text
Finders and loaders
Slide 22
Slide 22 text
Finders
‣ find the module
‣ return a spec object
‣ Finder object must have a find_spec method
defined.
Slide 23
Slide 23 text
Finders - examples
‣ locate built-in modules - BuiltinImporter
‣ locate frozen modules - FrozenImporter
‣ import path - imagine $PATH and
$PYTHONPATH - PathFinder
Slide 24
Slide 24 text
Finders - function
‣ If a finder is capable of handling import, it
returns a module spec.
‣ module spec contains module’s metadata i.e
path, name, etc. The import machinery uses
this spec to load modules later.
Slide 25
Slide 25 text
find_spec
Slide 26
Slide 26 text
importing spam.ham
Slide 27
Slide 27 text
Import hooks and PEP 302
Slide 28
Slide 28 text
sys.path_hooks
sys.meta_path
Slide 29
Slide 29 text
an example of sys.meta_path and sys.path_hooks
Slide 30
Slide 30 text
What are import hooks?
‣ Inserting custom finders and loaders into the
import mechanism
‣ Dependency injection?
Slide 31
Slide 31 text
Loaders
After spec object is returned from a finder,
loaders are used to execute the module and
store its reference in sys.modules
Slide 32
Slide 32 text
Tasks of loaders
‣ load from source/byte-code
‣ compile source
‣ execute the module and store reference in
sys.modules[spec.name]
Slide 33
Slide 33 text
example - loading
Slide 34
Slide 34 text
example - loading
Slide 35
Slide 35 text
example - loading
Slide 36
Slide 36 text
ImportError
ModuleNotFoundError
When no finder can find a module
When loading fails for a package/module
Slide 37
Slide 37 text
chronological order
Slide 38
Slide 38 text
Code demonstration of an
import hooks’ implementation.
Slide 39
Slide 39 text
What else can you do with import hooks?
‣ full control after what happens after import x
‣ importing from ftp, https, authorization
checks.
‣ importing from a variety of storage, not
just .py/.pyc
Slide 40
Slide 40 text
How have people used import hooks?
‣ https://github.com/kragniz/json-sempai
‣ https://github.com/nvbn/import_from_github_com
‣ https://github.com/runsascoded/ur
Slide 41
Slide 41 text
sys.meta_path with virtual environment disabled
Slide 42
Slide 42 text
sys.meta_path with virtual environment enabled
Virtual Environment’s finder
Slide 43
Slide 43 text
Thank you!
https://shivashis.xyz