Installing and importing modules

To extend the capabilities of the included standard Python library of modules, Python was built to be extensible. Third-party modules are downloaded in some format from a provider (often PyPI, the Python Package Index, where most are held) using either the built-in pip program or another method. For us modules such as arcpy and the ArcGIS API for Python are perfect examples: they extend the capabilities of Python to be able to control the tools that are available within ArcGIS Desktop or Pro respectively.

Using pip

To make Python module installation easier, Python is now installed with a program called pip. This name is an recursive acronym which stands for Pip Installs Programs. It simplifies installation by allowing for one line command line calls both locates the requested module on an online repository and runs the installation commands.

Pip connects to the Python Package Index (or PyPI). Stored on this repository are hundreds of thousands of free modules written by other developers. It is worth checking the license of the module to confirm that it will allow for your use of its code.

Pip lives in the Scripts folder, where lots of executable files are stored:

Installing modules

We will cover the

The setup.py file

Often Python 2.x and sometimes in Python 3.x a module is includes a “setup.py” file. This file is not run by pip; instead, it is run by Python itself.

Usually, a module will have a downloadable zip file that should be copied to the /sites/packages folder. This should be unzipped, and then the Python executable should be used to run the setup.py file using the install command: python setup.py install

Installing in virtual environments

Virtual environments are a bit of an odd concept at first, but they are extremely useful when programming in Python. Because you will probably have two different Python versions installed on your computer if you have ArcGIS Desktop and ArcGIS Pro, it is convenient to have these versions located in a virtual environment.

The core idea is to use one of the Python virtual environment modules to create a copy of your preferred Python version, which is then isolated from the rest of the Python versions on your machine. This avoids path issues when calling modules, allowing you to have more than one version of these important modules on the same computer.

Here are a few of the Python virtual environment modules:

Name	Description	Example virtual environment creation
venv	Built into Python 3.3+	python3 -m venv
virtualenv	Most be installed separately. It is very useful and my personal favorite.	virtualenv namenv --python=python3.6
pyenv	Used to isolate Python versions for testing purposes. Must be installed separately.	pyenv install 3.7.7
Conda /Anaconda	Used often in academic and scientific environments. Must be installed separately.	conda create --name snakes python=3.9

Read more about virtual environments here: https://towardsdatascience.com/python-environment-101-1d68bda3094d

Importing modules

To access the wide number of modules in the Python standard library, as well as third-party modules such as arcpy, we need to be able to import these modules in our script (or in the interpreter).

To do this you will use import statements. These declare the module or submodules (smaller components of the module) that you will use in the script, and as long as the modules are in the /sites/packages folder in your Python installation, or in the PATH (as arcpy is after its been installed).

import csv
from datetime import timedelta
from arcpy import da.SearchCursor

Three ways to import

There are three different and related ways to import modules. These modules, from either the standard library or from third-parties, are all imported the same in a script.

Method 1: import the whole module

This is the simplest way to import a module, by importing its top-level object. Its sub-methods are accessed using dot notation (e.g. csv.Reader, a method used to read CSV files):

import csv
reader = csv.Reader

Method 2: import a sub module

Instead of importing a top-level object, you can import only the module or method you need, using the “from X import Y” format:

from datetime import timedelta
from arcpy import da.SearchCursor

Method 3: import all sub modules

Instead of importing one sub-object, you can import all the modules or methods, using the “from X import *” format:

from datetime import *
from arcpy import *

Read more about importing modules here: https://realpython.com/python-import/

Importing custom code

Modules don’t have to just come from “third-parties”: they can come from you as well. With the use of the special __init__.py file, you can convert a normal folder into an importable module

The __init__.py file

This special file, which can contain code but mostly is just an empty file, indicates to Python that a folder is a module that can be imported into a script. The file itself is just a text file with a .py extension and the name __init__.py (that’s two underscores on each side), which is placed inside a folder. As long as the folder with the __init__.py is either next to the script or in the Python Path (e.g. in the site-packages folder), the code inside the folder can be imported.

Example custom module

In this example, we see some code in a script called example_module.py:

import csv
from datetime import timedelta
print('script imported')

Create a folder called mod_test. Copy this script into the folder. Then, create an empty text file called __init__.py:

Import your module

Create a new script next to the mod_test folder. Call it “module_import.py”:

Inside the script you will import the function “test_function” from the example_module script in the mod_test folder using the format below:

Scripts inside the module are accessed using dot notation (e.g. mod_test.example_module). The functions and classes inside the script called example_module.py are able to be imported by name.

Because the module is sitting next to the script that is importing the function, this import statement will work. But if you move your script and don’t copy the module to somewhere that is on the Python Path, it won’t be a successful import

That is because the way import statements work is based on the Python Path. This is a list of folder locations that Python will look for the module that you are requesting. By default, the first location is the local folder, meaning the folder containing your script. The next location is the site-packages folder.

The site-packages folder

Most modules are installed in a folder inside the Python folder. This is called the site-packages folder and it sits at */Lib/sites-packages.

To make your module available to for import without needing it to be next to your script, put your module folder in the site-packages folder. When you run “from mod_test.example_module import test_function” it will locate the module called mod_test in the sites packages folder.