Of the utmost import-ance: providing flexible import (and installation) for Python.

A colleague of mine once asked me to write a script for him. Among the many specifications was the requirement was one that created a hurdle to using third-party libraries even the most ubiquitous. It stated that he should be able to run the script from the command-line without any prior setup of virtual environments or installation of packages. Sometimes, the odds are not always in your favor. It led me down a small rabbit hole wanting to solve the problem of using some external library in the script with these restraints.

Credits: https://ioflood.com/blog/python-import/

Whether you just want to use a library without having to check if you installed it. Or during prototyping, you do not wish to create a requirements.txt before sharing the script with someone else. Or if you want to dynamically import a library, the import manager I created is one way to be able to do this. Short of creating my own version of Python to include a new keyword, like Eli did¹, I decided a function would be second best. A function that tries to import a library and attempts to install it when the library is not found.

TLDR

In essence, the function will use two libraries from the Python Standard Library². importlib, which is the same library that also provides an implementation of import³, will provide the function import_module. This wrapper around the function __import__ imports the specified package or module. If it can not find the module, it uses the check_call function from the subprocess library to run a command. However, the entire code has exception handling, descriptive warning messages to inform on the progress, an improved command call, and a method to handle the library where the import name and installation name differ. The full code can be found here.

import importlib
import subprocess

def import_module(name):
    try:
        return importlib.import_module(name)
    except ModuleNotFoundError:
        subprocess.check_call(["pip", "install", name])
        return importlib.import_module(name)

# use
seaborn = import_module("seaborn")
#
plt = import_module("matplotlib.pyplot")

First, I will give some background on the import process in Python. Then I’ll explain how the code works and the steps I took. Next, I’ll reflect on what I learned from this before I conclude this story.

Background

How does Python’s import process work?

To paraphrase the Python documentation, the import process consists of two parts. The first part is to search for the named module, this is handled by a call to import, and the second part is to bind the result of the search to a name in the local scope. The name binding is only done when the import statement is used. The search starts in sys.modules⁵, this is a dictionary that maps module names to modules which have already been loaded⁶. If the named module is not found sys.path is checked⁷.

Imports can be absolute and relative. Relative imports use leading dots to indicate whether it is from the current package or from one of the parents. Absolute imports only use dots between module names.

# absolute import
import module
from module import submodule
# relative import
from .module import submodule

The following exceptions can occur during the import process:

• If the named module cannot be found, a ModuleNotFoundError is raised⁵,
• If an attribute is not found a from module import elements call, an ImportError is raised⁸,
• The use of a wildcard import from seabron import * will raise a SyntaxError⁸.

What is the difference between a module, a package and a library?

A module is the code in a single .py file. A package is one or more modules in a folder. To be considered a package an init.py file is needed within that folder. A library is a broader term for reusable code⁹.

Explanation

The code is split up into two functions; one to handle the import of a module, import_module, and one to handle the installation of a module, _pip_install_package.

Let’s first wrap the import function

The main function import_module attempts to import the requested module using the Python import system. When successful, it returns the module. If it fails to import the module, it calls _pip_install_package to install the package. When this call is successful, it returns a call to itself with the same arguments. Otherwise, it prints a warning and returns None. The only exception that is handled is ModuleNotFoundError because that indicates that the module might not be installed.

def import_module(name: str, package: str = None) -> typing.Optional[typing.types.ModuleType]:
    try:
        return importlib.import_module(name, package=package)
    except ModuleNotFoundError:
        print(f"Module '{name}' not found. Attempting to install...")
        package_name: str = _get_package_name(name, package=package)
        # Check if installation was successful
        if _pip_install_package(package):
            return import_module(name, package=package)
        else:
            warnings.warn(
                f"Installation of module '{name}' failed. Cannot import.",
                UserWarning)
            return None

# use
sns = import_module("seaborn")

How do you deal with different names for importing and installing?

Sometimes, the name for a package differs between the one used for importing and installation. There are two common forms. One is hyphenation that differs. This is dealt with by pip. The other is having distinctly different names. For example, Pillow and beautifulsoup4. To solve the latter problem, a dictionary is used to switch the names. If you know of any other library I could append to the mapping, let me know in the comments.

How to get the package name from the input?

By using a similar approach as used in the importlib library in the import_module function the package name is found¹⁰. First, a check is done if a relative import is made. The package name is then at the beginning of the package variable. Second, if it is an absolute import the name is at the beginning of the module_name variable. Finally, a mapping is used to return the package name for installing for the few known exceptions.

def _get_package_name(module_name: str, package: str = None) -> strp:
    # if it is a relative import check if package is supplied
    if module_name.startswith("."):
        # raise an error if no package is supplied
        if not package:
            raise TypeError(
                "Package must be provided when name starts with a dot.")
        package_name: str = package.split(".")[0]
    else:
        package_name: str = module_name.split(".")[0].strip()
        if not package_name:
            raise TypeError("Module name cannot be resolved.")
    return PACKAGE_MAP.get(package_name, package_name)

How to pip install from within a script?

The main element of the function is the check_call function from the subprocess library which runs a command with arguments¹¹. It waits for the command to complete. A successful completion, signal by exit code 0, returns. Otherwise, it raises a CalledProcessError.

The command is called with sys.executable, which gives the absolute path of the executable binary for the Python interpreter¹². It is the recommended way to launch an interpreter. The argument -m signals the use of a preinstalled module.

def _pip_install_package(name: str) -> bool:
    try:
        subprocess.check_call([sys.executable, "-m", "pip", "install", name])
        return True
    except subprocess.CalledProcessError as e:
        warnings.warn(f"Failed to install package '{name}': {e}", UserWarning)
        return False
    except FileNotFoundError:
        warnings.warn(
            "Pip might not be installed. Please install pip to use this function.",
            UserWarning)
        return False

Two exceptions are handled, i.e., the pip install command is unsuccessful, or pip is not found. The first can be due to various reasons such as network problems or misspelling. The latter can occur if pip is not installed.

Why not catch all exceptions?

There are many more foreseeable exceptions that can occur. I choose to keep it contained to this set because they are needed in the workflow and could be caused by elements of the script. All the libraries I used I trust have well tested exceptions with clear messages. Therefore, there I saw no need to catch and reraise those exceptions.

What I learned

• First of all, I got a slightly better understanding of the import system in Python. Both creating the script as well as writing this story forced me to go through the documentation to try and understand it. I now know about the library, importlib, which is the backbone of that process, and some of its functionality. Going through it to try to understand how it works and how each function calls another was enriching.
• Second, I learned that documenting and explaining a script you made really helps in both documenting and refactoring it. I found variable names that could be more descriptive. There was an exception to be raised in the _get_package_name for a situation that could not logically occur. So after some tests that was removed. Although this is the third time I’m doing this, now it really hit home.
• Third, explaining the script in the story really made me understand every function call and statement in it.
• Fourth, being able to execute command line calls from a script is a handy skill. I have since used to automate a weekly task at work. So the learned skill was directly put to good use.
• Finally, I learned that doing the research for the story to provide adequate background takes up a lot of time and energy. A mental note for future projects is to store links and articles as I write the code. And perhaps, if I want to write a story, do it during refactoring.

To conclude

Investigating and making this helper function to dynamically load modules was a great learning experience. It solved a problem I had, and I learned about the import system and how to run commands. Although there cases in which it won’t work, and it is somewhat artificial, I think it is a nice solution. A future project might be to create an alternative for the import keyword…

This story is my way to share my coding experience and the lessons I learned, and to document my solutions. All claps, comments, and highlights are appreciated, as well as sharing the story. For more code and my other links see: https://github.com/UnicornOnAzur/.

[1] https://eli.thegreenplace.net/2010/06/30/python-internals-adding-a-new-statement-to-python/

[2] https://docs.python.org/3/library/index.html

[3] https://docs.python.org/3/library/importlib.html

[4] https://docs.python.org/3/library/subprocess.html

[5] https://docs.python.org/3/reference/import.html#importsystem

[6] https://docs.python.org/3/library/sys.html#sys.modules

[7] https://bic-berkeley.github.io/psych-214-fall-2016/sys_path.html

[8] https://docs.python.org/3/reference/simple_stmts.html#import

[9] https://learnpython.com/blog/python-modules-packages-libraries-frameworks/

[10] https://github.com/python/cpython/blob/main/Lib/importlib/init.py#L71

[11] https://docs.python.org/3/library/subprocess.html#subprocess.check_call

[12] https://docs.python.org/3/library/sys.html#sys.executable