What happens when you import a module? • Repeated code, line after line? Use a loop. • Repeated code, in several different places? Use a function. • Repeated code, in several different programs? Use a library. “DRY up” our code 4
What happens when you import a module? • In Python, our libraries are called “modules.” • They’re usually (but not always) f iles containing Python code. • They also (as an added bonus) provide us with namespaces. Modules 5
What happens when you import a module? • “import” is a statement, not a function • No parentheses! • We give the name of the variable we want to create • Not a string! • Not a f ilename! • So you can’t (directly) tell Python what f ile to load Things to notice 9
What happens when you import a module? >>> import random >>> type(random) module • Module objects are pretty simple • They’re basically namespaces (using attributes) • Modules don’t have any methods Modules are objects 10
What happens when you import a module? • Want to import a module, but give it a different name? • Use “as” import random as r • All this changes is the variable that’s created Changing the variable name 11
What happens when you import a module? • What happens if: • Module a imports b • Module b imports a • We don’t want an in f inite loop! • Fortunately, this doesn’t happen. • Python only imports a module once per session • But it will always de f ine the variable • But wait: How does Python know if a module has been imported? Load once, assign twice 12
What happens when you import a module? • This dict is the cache for Python modules • Keys are strings, the modules’ names • Values are module objects • When we import, Python checks if the module is already there • If not, then it imports the module and adds it to sys.modules • Then it can assign a global variable to the value sys.modules 13
What happens when you import a module? • The Python interactive shell: >>> import sys >>> len(sys.modules) 79 • IPython: In [1]: import sys In [2]: len(sys.modules) Out[2]: 646 • Jupyter: In[1]: import sys Out[2]: len(sys.modules) Out[2]: 1020 How big is sys.modules (by default)? 14
What happens when you import a module? • I’ll now show some simpli f ied versions of what happens • Don’t really run this code • Instead, use “importlib” • It implements the import system • Designed to be read, studied, and used • I’ll refer to it in a few places later on Warning: Pseudocode ahead! 15
What happens when you import a module? def myimport(mod): if mod not in sys.modules: sys.modules[mod] = get_module(mod) globals()[mod] = sys.modules[mod] import pseudocode We’ll discuss this soon 16
What happens when you import a module? def myimport(mod, alias=None): if mod not in sys.modules: sys.modules[mod] = get_module(mod) if alias is None: alias = mod globals()[alias] = sys.modules[mod] Import-as pseudocode 17
What happens when you import a module? • What happens if: • My top-level program says “import a” • Module a says “import b” • Module b says “import a" Consider this example 18
What happens when you import a module? • Our program says “import a” • ‘a’ is not in sys.modules, so a is loaded • Global variable a is assigned to sys.modules[‘a’] • Module a imports b • ‘b’ is not in sys.modules, so b is loaded • Global variable b is assigned to sys.modules[‘b’] • Module b imports a • ‘a’ is in sys.modules, so it doesn’t need to be loaded • Global variable a is assigned to sys.modules[‘a’] What happens? 19
What happens when you import a module? • In many languages, a “global variable” is available throughout the entire language. • In Python, what we call a “global variable” is really global in a single f ile — or, if you prefer, in a single module/namespace. • Module a can have a global x that’s totally different from module b’s global x. • The current namespace is always available via the __name__ variable • The f irst namespace is ‘__main__’. Global variables? 20
What happens when you import a module? • Our program says “import a” • ‘a’ is not in sys.modules, so a is loaded • a in ‘__main__’ is assigned to sys.modules[‘a’] • Module a imports b • ‘b’ is not in sys.modules, so b is loaded • a.b is assigned to sys.modules[‘b’] • Module b imports a • ‘a’ is is sys.modules, so it doesn’t need to be loaded • b.a is assigned to sys.modules[‘a’] Let’s rephrase things, then 21
What happens when you import a module? • You might also have seen the following syntax: from random import randint • Does this save memory? • No! • Does this load the entire module? • Yes! Import speci f ic names 22
What happens when you import a module? from random import randint • Python: • Checks if “random” is in sys.modules • If not, it loads the module (as we’ve already seen) • Then it de f ines “randint” as a global • Note: “random” is not de f ined as a global from .. import 23
What happens when you import a module? def myfrom_import(mod, *names): if mod not in sys.modules: sys.modules[mod] = get_module(mod) for one_name in names: globals()[one_name] = \ getattr(sys.modules[mod],one_name) from-import pseudocode 24
What happens when you import a module? • You can also alias names imported with “from..import” • In this case, you can say from random import randint as ri Aliases with “from..import” 25
What happens when you import a module? def myfrom_import(mod, **kwargs): if mod not in sys.modules: sys.modules[mod] = get_module(mod) for one_name, alias in kwargs.items(): globals()[alias] = \ getattr(sys.modules[mod],one_name) From-import-as pseudocode 26
What happens when you import a module? • You can also say from random import * • This imports all names from “random” as globals • The module can restrict them by de f ining __all__ from .. import *
What happens when you import a module? • get_module is my pseudocode function • It takes a module name as an input • It then needs to do two things • Find the f ile for our module • Load the f ile for our module • It returns a module object, which is stored in sys.modules Get a module name, load a f ile 30
What happens when you import a module? • It’s called importlib.import_module • There’s also __import__ in builtins, but this is easier >>> r = importlib.import_module('random') >>> type(r) module >>> r.randint(0, 100) 92 get_module really exists! 31
What happens when you import a module? • A list of strings, directories where Python will look • De f ined when Python starts up • First match wins! • Namespace collisions can lead to … surprises sys.path 34
What happens when you import a module? >>> print('\n'.join(sys.path)) /usr/local/Cellar/python@3.10/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python310.zip /usr/local/Cellar/python@3.10/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python3.10 /usr/local/Cellar/python@3.10/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python3.10/lib-dynload /usr/local/lib/python3.10/site-packages sys.path (in the Python shell) 35
What happens when you import a module? • sys.path[0] is usually the program’s directory • Not necessarily your working directory! • sys.path[0] can be the empty string • e.g., when you use the interactive Python shell • No matter what: The program’s current directory is always the f irst place we search for module f iles sys.path[0] 36
What happens when you import a module? >>> print('\n'.join(sys.path)) /usr/local/Cellar/python@3.10/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python310.zip /usr/local/Cellar/python@3.10/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python3.10 /usr/local/Cellar/python@3.10/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python3.10/lib-dynload /usr/local/lib/python3.10/site-packages My sys.path 37
What happens when you import a module? • The standard library — modules that come with Python — normally comes next • We’ll talk more about these in a bit • Standard library modules aren’t imported automatically • But they’re always available, via sys.path After sys.path[0] 38
What happens when you import a module? >>> print('\n'.join(sys.path)) /usr/local/Cellar/python@3.10/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python310.zip /usr/local/Cellar/python@3.10/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python3.10 /usr/local/Cellar/python@3.10/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python3.10/lib-dynload /usr/local/lib/python3.10/site-packages My sys.path 39
What happens when you import a module? • pip installs into site-packages • This is where you’ll f ind things from PyPI • It comes after (nearly) everything else Site-packages 40
What happens when you import a module? • sys.path is de f ined when Python starts up • It’s a list of strings… but really, don’t run “list.append” on it • Rather, set PYTHONPATH (environment variable) • Anything here is inserted at sys.path[1] • For example, I’ll set: export PYTHONPATH=/hello:/good/bye Modifying sys.path 41
What happens when you import a module? >>> print('\n'.join(sys.path)) /hello /good/bye /usr/local/Cellar/python@3.10/3.10.5/Frameworks/ Python.framework/Versions/3.10/lib/python310.zip /usr/local/Cellar/python@3.10/3.10.5/Frameworks/ Python.framework/Versions/3.10/lib/python3.10 /usr/local/Cellar/python@3.10/3.10.5/Frameworks/ Python.framework/Versions/3.10/lib/python3.10/lib- dynload /usr/local/lib/python3.10/site-packages The resulting sys.path 42 From PYTHONPATH
What happens when you import a module? def myimport(mod, alias=None): if mod not in sys.modules: for dir in sys.path: file = os.path.join(dir, f'{mod}.py') if os.path.exists(file): sys.modules[mod] = load_module(file) break else: raise ModuleNotFoundError() if alias is None: alias = mod globals()[alias] = sys.modules[mod] We’ll discuss this soon 44
What happens when you import a module? • Some items in sys.path aren’t directories • Modules are stored in a bunch of different places • If you have a f ile called “time.py” in the current directory, the builtin “time” module is still imported • What about caching of .pyc f iles? • Packages — directories containing modules Problems with this story 45
What happens when you import a module? • Finders • Does the module exist? • Which loader should be used? • Return a module spec, containing a loader • Loaders • Create a module object • Execute the module f ile • Populate the module object • Importers combine this functionality Solution: Finders and loaders 46
What happens when you import a module? • sys.meta_path is a list of f inders and importers • Each implements the “ f ind_spec” class method, which: • Returns “None” if it can’t handle this type of module, thus allowing the next f inder to give it a shot • Returns a module spec, telling Python how to load the module • Raises ModuleNotFoundError — couldn’t f ind it! • Python gives each element in sys.meta_path a chance sys.meta_path 47
What happens when you import a module? import sys for one_finder in sys.meta_path: print(one_finder) spec = one_finder.find_spec('random', None) print(spec) if spec: break Finding “random” 48
What happens when you import a module? import sys for one_finder in sys.meta_path: print(one_finder) spec = one_finder.find_spec('time', None) print(spec) if spec: break Finding “time” 50
What happens when you import a module? <_distutils_hack.DistutilsMetaFinder object at 0x10db18370> None <class '_frozen_importlib.BuiltinImporter'> ModuleSpec(name='time', loader=<class '_frozen_importlib.BuiltinImporter'>, origin='built- in') Finding “time” 51
What happens when you import a module? import sys for one_finder in sys.meta_path: print(one_finder) spec = one_finder.find_spec('math', None) print(spec) if spec: break Finding “math” 53
What happens when you import a module? • Its attributes tell Python how to load it, and from where: • name • origin • cached • loader • has_location • parent • submodule_search_locations What’s in a module spec? 55
What happens when you import a module? • Use the spec’s loader and name to create a module mod = spec.create_module(spec) • If it returns a module object, then great! • If not, then you need to create the module object and assign a variety of attributes to it, such as __ f ile__, __loader__, and __spec__ • That’s how these attributes are set on the module object! Creating the module object 56
What happens when you import a module? • A much easier way is to use importlib: mod = importlib.util.module_from_spec(spec) • Even better, it populates the important attributes: >>> dir(mod) ['__cached__', '__doc__', '__file__', ‘__loader__', '__name__', '__package__', '__spec__'] Creating the module 57
What happens when you import a module? • Our module object now exists! • But it doesn’t contain any of our module’s de f initions • For this, we’ll need to load the code from the module Populating the module 58
What happens when you import a module? • The module has a variable? Assignment needs to run • The module has a function? “def” needs to run • The module has a class? “class” needs to run • When we load a module, we run it, top to bottom Loading == executing 59
What happens when you import a module? • Moreover, something weird happens when we import a module • The global variables in the module become attributes • Global “x” in module “mymod” becomes “mymod.x” • Function “hello” becomes “mymod.hello” • How does this happen? Loading == setting attributes 60
What happens when you import a module? • exec is one of the most powerful and dangerous functions • Pass it a string containing Python, and it executes that code >>> s = """ ... print('Hello') ... name = 'Reuven' ... print('Goodbye') """ >>> exec(s) Hello Goodbye >>> name 'Reuven' exec 61
What happens when you import a module? • Pass a dict as a second argument to “exec”, and assignments go there >>> s = """ ... print('Hello') ... name = 'Reuven' ... print('Goodbye') """ >>> d = {} >>> exec(s, d) Hello Goodbye >>> d['name'] 'Reuven' exec into a dict 62
What happens when you import a module? • Module objects have a “__dict__” attribute • Any key in this dict is an attribute in the module • So: • Read the source code from the module • Execute it with “exec” • Pass “exec” a second argument, the module’s __dict__ • Voila! All global assignments in the module are attributes How does this help?
What happens when you import a module? • Loaders have “exec_module”, which does this for us: >>> spec.loader.exec_module(random) >>> random.randint(0, 100) 31 Even better…
What happens when you import a module? • A lot! • Python checks if the module has been cached • If so, it returns the module, and assigns the variable • If not, it goes through f inders in sys.meta_path • If none of them f inds the module, you get an exception • The f inder that works returns a module spec object • Python creates a new, empty module object • It uses the loader to exec the source code into the module’s __dict__, thus making globals into attributes So, what happens when you import?
What happens when you import a module? • We’re still talking about module objects! • Storage is in directories, not f iles • Namespace packages vs. __init__.py • Several more module attributes (__path__) are populated • We need to keep track of levels, for relative imports • This adds complexity, but doesn’t change the overall structure of what I’ve described here A few words about packages
reuven
tak0x00
hr01
inureyes
donkomura
kazaneya
oikon48
maguroalternative
kakehashi
keisuketakiguchi
yoshidashingo
ijin
shirarei24
sonatard
notwaldorf
ufuk
jlugia
jmmastey
samlambert
mongodb
cassininazir
akosma
addyosmani
speakerdeck
wjessup
