What happens when you import a module?

Slide 1

Slide 1 text

What happens when you import a module? Reuven M. Lerner • Euro Python 2022 [email protected] • @reuvenmlerner

Slide 2

Slide 2 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Corporate • Video • Hybrid • Weekly Python Exercise • Python Data Analysis Bootcamp • Books: • Python Workout • Pandas Workout • “Better developers” — free, weekly newsletter about Python • https://BetterDevelopersWeekly.com/ I teach Python 2

Slide 3

Slide 3 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? DRY: Don’t Repeat Yourself 3

Slide 4

Slide 4 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Repeated code, line after line? Use a loop. • Repeated code, in several different places? Use a function. • Repeated code, in several different programs? Use a library. “DRY up” our code 4

Slide 5

Slide 5 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • In Python, our libraries are called “modules.” • They’re usually (but not always) f iles containing Python code. • They also (as an added bonus) provide us with namespaces. Modules 5

Slide 6

Slide 6 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? Let’s import a module! 6

Slide 7

Slide 7 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? 7 import random

Slide 8

Slide 8 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Creates a new module object • Assigns that new module object to a variable “import” does two things 8

Slide 9

Slide 9 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • “import” is a statement, not a function • No parentheses! • We give the name of the variable we want to create • Not a string! • Not a f ilename! • So you can’t (directly) tell Python what f ile to load Things to notice 9

Slide 10

Slide 10 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? >>> import random >>> type(random) module • Module objects are pretty simple • They’re basically namespaces (using attributes) • Modules don’t have any methods Modules are objects 10

Slide 11

Slide 11 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Want to import a module, but give it a different name? • Use “as” import random as r • All this changes is the variable that’s created Changing the variable name 11

Slide 12

Slide 12 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • What happens if: • Module a imports b • Module b imports a • We don’t want an in f inite loop! • Fortunately, this doesn’t happen. • Python only imports a module once per session • But it will always de f ine the variable • But wait: How does Python know if a module has been imported? Load once, assign twice 12

Slide 13

Slide 13 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • This dict is the cache for Python modules • Keys are strings, the modules’ names • Values are module objects • When we import, Python checks if the module is already there • If not, then it imports the module and adds it to sys.modules • Then it can assign a global variable to the value sys.modules 13

Slide 14

Slide 14 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • The Python interactive shell: >>> import sys >>> len(sys.modules) 79 • IPython: In [1]: import sys In [2]: len(sys.modules) Out[2]: 646 • Jupyter: In[1]: import sys Out[2]: len(sys.modules) Out[2]: 1020 How big is sys.modules (by default)? 14

Slide 15

Slide 15 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • I’ll now show some simpli f ied versions of what happens • Don’t really run this code • Instead, use “importlib” • It implements the import system • Designed to be read, studied, and used • I’ll refer to it in a few places later on Warning: Pseudocode ahead! 15

Slide 16

Slide 16 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? def myimport(mod): if mod not in sys.modules: sys.modules[mod] = get_module(mod) globals()[mod] = sys.modules[mod] import pseudocode We’ll discuss this soon 16

Slide 17

Slide 17 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? def myimport(mod, alias=None): if mod not in sys.modules: sys.modules[mod] = get_module(mod) if alias is None: alias = mod globals()[alias] = sys.modules[mod] Import-as pseudocode 17

Slide 18

Slide 18 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • What happens if: • My top-level program says “import a” • Module a says “import b” • Module b says “import a" Consider this example 18

Slide 19

Slide 19 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Our program says “import a” • ‘a’ is not in sys.modules, so a is loaded • Global variable a is assigned to sys.modules[‘a’] • Module a imports b • ‘b’ is not in sys.modules, so b is loaded • Global variable b is assigned to sys.modules[‘b’] • Module b imports a • ‘a’ is in sys.modules, so it doesn’t need to be loaded • Global variable a is assigned to sys.modules[‘a’] What happens? 19

Slide 20

Slide 20 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • In many languages, a “global variable” is available throughout the entire language. • In Python, what we call a “global variable” is really global in a single f ile — or, if you prefer, in a single module/namespace. • Module a can have a global x that’s totally different from module b’s global x. • The current namespace is always available via the __name__ variable • The f irst namespace is ‘__main__’. Global variables? 20

Slide 21

Slide 21 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Our program says “import a” • ‘a’ is not in sys.modules, so a is loaded • a in ‘__main__’ is assigned to sys.modules[‘a’] • Module a imports b • ‘b’ is not in sys.modules, so b is loaded • a.b is assigned to sys.modules[‘b’] • Module b imports a • ‘a’ is is sys.modules, so it doesn’t need to be loaded • b.a is assigned to sys.modules[‘a’] Let’s rephrase things, then 21

Slide 22

Slide 22 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • You might also have seen the following syntax: from random import randint • Does this save memory? • No! • Does this load the entire module? • Yes! Import speci f ic names 22

Slide 23

Slide 23 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? from random import randint • Python: • Checks if “random” is in sys.modules • If not, it loads the module (as we’ve already seen) • Then it de f ines “randint” as a global • Note: “random” is not de f ined as a global from .. import 23

Slide 24

Slide 24 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? def myfrom_import(mod, *names): if mod not in sys.modules: sys.modules[mod] = get_module(mod) for one_name in names: globals()[one_name] = \ getattr(sys.modules[mod],one_name) from-import pseudocode 24

Slide 25

Slide 25 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • You can also alias names imported with “from..import” • In this case, you can say from random import randint as ri Aliases with “from..import” 25

Slide 26

Slide 26 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? def myfrom_import(mod, **kwargs): if mod not in sys.modules: sys.modules[mod] = get_module(mod) for one_name, alias in kwargs.items(): globals()[alias] = \ getattr(sys.modules[mod],one_name) From-import-as pseudocode 26

Slide 27

Slide 27 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • You can also say from random import * • This imports all names from “random” as globals • The module can restrict them by de f ining __all__ from .. import *

Slide 28

Slide 28 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? Never do this. Please. 28

Slide 29

Slide 29 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? Wait: What does get_module do? 29

Slide 30

Slide 30 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • get_module is my pseudocode function • It takes a module name as an input • It then needs to do two things • Find the f ile for our module • Load the f ile for our module • It returns a module object, which is stored in sys.modules Get a module name, load a f ile 30

Slide 31

Slide 31 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • It’s called importlib.import_module • There’s also __import__ in builtins, but this is easier >>> r = importlib.import_module('random') >>> type(r) module >>> r.randint(0, 100) 92 get_module really exists! 31

Slide 32

Slide 32 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? How does Python f ind modules? 32

Slide 33

Slide 33 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? The simple story 33

Slide 34

Slide 34 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • A list of strings, directories where Python will look • De f ined when Python starts up • First match wins! • Namespace collisions can lead to … surprises sys.path 34

Slide 35

Slide 35 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? >>> print('\n'.join(sys.path)) /usr/local/Cellar/[email protected]/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python310.zip /usr/local/Cellar/[email protected]/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python3.10 /usr/local/Cellar/[email protected]/3.10.5/ Frameworks/Python.framework/Versions/3.10/lib/ python3.10/lib-dynload /usr/local/lib/python3.10/site-packages sys.path (in the Python shell) 35

Slide 36

Slide 36 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • sys.path[0] is usually the program’s directory • Not necessarily your working directory! • sys.path[0] can be the empty string • e.g., when you use the interactive Python shell • No matter what: The program’s current directory is always the f irst place we search for module f iles sys.path[0] 36

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • The standard library — modules that come with Python — normally comes next • We’ll talk more about these in a bit • Standard library modules aren’t imported automatically • But they’re always available, via sys.path After sys.path[0] 38

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • pip installs into site-packages • This is where you’ll f ind things from PyPI • It comes after (nearly) everything else Site-packages 40

Slide 41

Slide 41 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • sys.path is de f ined when Python starts up • It’s a list of strings… but really, don’t run “list.append” on it • Rather, set PYTHONPATH (environment variable) • Anything here is inserted at sys.path[1] • For example, I’ll set: export PYTHONPATH=/hello:/good/bye Modifying sys.path 41

Slide 42

Slide 42 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? >>> print('\n'.join(sys.path)) /hello /good/bye /usr/local/Cellar/[email protected]/3.10.5/Frameworks/ Python.framework/Versions/3.10/lib/python310.zip /usr/local/Cellar/[email protected]/3.10.5/Frameworks/ Python.framework/Versions/3.10/lib/python3.10 /usr/local/Cellar/[email protected]/3.10.5/Frameworks/ Python.framework/Versions/3.10/lib/python3.10/lib- dynload /usr/local/lib/python3.10/site-packages The resulting sys.path 42 From PYTHONPATH

Slide 43

Slide 43 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? And now, the pseudocode version 43

Slide 44

Slide 44 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? def myimport(mod, alias=None): if mod not in sys.modules: for dir in sys.path: file = os.path.join(dir, f'{mod}.py') if os.path.exists(file): sys.modules[mod] = load_module(file) break else: raise ModuleNotFoundError() if alias is None: alias = mod globals()[alias] = sys.modules[mod] We’ll discuss this soon 44

Slide 45

Slide 45 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Some items in sys.path aren’t directories • Modules are stored in a bunch of different places • If you have a f ile called “time.py” in the current directory, the builtin “time” module is still imported • What about caching of .pyc f iles? • Packages — directories containing modules Problems with this story 45

Slide 46

Slide 46 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Finders • Does the module exist? • Which loader should be used? • Return a module spec, containing a loader • Loaders • Create a module object • Execute the module f ile • Populate the module object • Importers combine this functionality Solution: Finders and loaders 46

Slide 47

Slide 47 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • sys.meta_path is a list of f inders and importers • Each implements the “ f ind_spec” class method, which: • Returns “None” if it can’t handle this type of module, thus allowing the next f inder to give it a shot • Returns a module spec, telling Python how to load the module • Raises ModuleNotFoundError — couldn’t f ind it! • Python gives each element in sys.meta_path a chance sys.meta_path 47

Slide 48

Slide 48 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? import sys for one_finder in sys.meta_path: print(one_finder) spec = one_finder.find_spec('random', None) print(spec) if spec: break Finding “random” 48

Slide 49

Slide 49 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? <_distutils_hack.DistutilsMetaFinder object at 0x10db18370> None None None ModuleSpec(name='random', loader=<_frozen_importlib_external.SourceFileLoader object at 0x10dab0f40>, origin='/usr/local/Cellar/ [email protected]/3.10.5/Frameworks/Python.framework/ Versions/3.10/lib/python3.10/random.py') Finding “random” 49

Slide 50

Slide 50 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? import sys for one_finder in sys.meta_path: print(one_finder) spec = one_finder.find_spec('time', None) print(spec) if spec: break Finding “time” 50

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? >>> sys.builtin_module_names ('_abc', '_ast', '_codecs', '_collections', '_functools', '_imp', '_io', '_locale', '_operator', '_signal', '_sre', '_stat', '_string', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'atexit', 'builtins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time', 'xxsubtype') What modules are already loaded? 52

Slide 53

Slide 53 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? import sys for one_finder in sys.meta_path: print(one_finder) spec = one_finder.find_spec('math', None) print(spec) if spec: break Finding “math” 53

Slide 54

Slide 54 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? <_distutils_hack.DistutilsMetaFinder object at 0x10367c370> None None None ModuleSpec(name='math', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x103614f40>, origin='/usr/local/Cellar/ [email protected]/3.10.5/Frameworks/Python.framework/Versions/ 3.10/lib/python3.10/lib-dynload/math.cpython-310- darwin.so') Finding “math" 54

Slide 55

Slide 55 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Its attributes tell Python how to load it, and from where: • name • origin • cached • loader • has_location • parent • submodule_search_locations What’s in a module spec? 55

Slide 56

Slide 56 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Use the spec’s loader and name to create a module mod = spec.create_module(spec) • If it returns a module object, then great! • If not, then you need to create the module object and assign a variety of attributes to it, such as __ f ile__, __loader__, and __spec__ • That’s how these attributes are set on the module object! Creating the module object 56

Slide 57

Slide 57 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • A much easier way is to use importlib: mod = importlib.util.module_from_spec(spec) • Even better, it populates the important attributes: >>> dir(mod) ['__cached__', '__doc__', '__file__', ‘__loader__', '__name__', '__package__', '__spec__'] Creating the module 57

Slide 58

Slide 58 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Our module object now exists! • But it doesn’t contain any of our module’s de f initions • For this, we’ll need to load the code from the module Populating the module 58

Slide 59

Slide 59 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • The module has a variable? Assignment needs to run • The module has a function? “def” needs to run • The module has a class? “class” needs to run • When we load a module, we run it, top to bottom Loading == executing 59

Slide 60

Slide 60 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Moreover, something weird happens when we import a module • The global variables in the module become attributes • Global “x” in module “mymod” becomes “mymod.x” • Function “hello” becomes “mymod.hello” • How does this happen? Loading == setting attributes 60

Slide 61

Slide 61 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • exec is one of the most powerful and dangerous functions • Pass it a string containing Python, and it executes that code >>> s = """ ... print('Hello') ... name = 'Reuven' ... print('Goodbye') """ >>> exec(s) Hello Goodbye >>> name 'Reuven' exec 61

Slide 62

Slide 62 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Pass a dict as a second argument to “exec”, and assignments go there >>> s = """ ... print('Hello') ... name = 'Reuven' ... print('Goodbye') """ >>> d = {} >>> exec(s, d) Hello Goodbye >>> d['name'] 'Reuven' exec into a dict 62

Slide 63

Slide 63 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Module objects have a “__dict__” attribute • Any key in this dict is an attribute in the module • So: • Read the source code from the module • Execute it with “exec” • Pass “exec” a second argument, the module’s __dict__ • Voila! All global assignments in the module are attributes How does this help?

Slide 64

Slide 64 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • Loaders have “exec_module”, which does this for us: >>> spec.loader.exec_module(random) >>> random.randint(0, 100) 31 Even better…

Slide 65

Slide 65 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? Whew!

Slide 66

Slide 66 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • All of this is customizable • Custom f inders • Custom path f inders • Custom loaders By the way…

Slide 67

Slide 67 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • A lot! • Python checks if the module has been cached • If so, it returns the module, and assigns the variable • If not, it goes through f inders in sys.meta_path • If none of them f inds the module, you get an exception • The f inder that works returns a module spec object • Python creates a new, empty module object • It uses the loader to exec the source code into the module’s __dict__, thus making globals into attributes So, what happens when you import?

Slide 68

Slide 68 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? Wait! We didn’t talk about packages! 68

Slide 69

Slide 69 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • We’re still talking about module objects! • Storage is in directories, not f iles • Namespace packages vs. __init__.py • Several more module attributes (__path__) are populated • We need to keep track of levels, for relative imports • This adds complexity, but doesn’t change the overall structure of what I’ve described here A few words about packages

Slide 70

Slide 70 text

Reuven M. Lerner • @reuvenmlerner • https://lerner.co.il/ Euro Python 2022: What happens when you import a module? • E-mail: [email protected] • Web: https://lerner.co.il/ • Twitter: @reuvenmlerner • YouTube: https://YouTube.com/reuvenlerner • Free, weekly Python articles: • https://BetterDevelopersWeekly.com/ • PS: I’ve got shirts and stickers to give out! Questions?