Why Python is different
Because of Van Rossum’s extensive experience with the state of computer languages in the 1980s, he was well positioned to create a language that solved many of their deficiencies. He added features that he admired from many other languages and added a few of his own. Here is an incomplete list of Python features built to improve on other languages:
| Issue | Improvement | Python Feature |
| Memory overrun | Built-in memory management | Garbage collection and memory management |
| Slow compiler times | One line testing, dynamic typing | Python Interpreter |
| Unclear error messages | Messages indicating the offending line and affected code | Error Traceback |
| Spaghetti code | Clean importation and modularization | Importation |
| Unclear code formatting and spacing making code unreadable | Indentation rules and reduced brackets | Forced whitespace |
| Too many ways to do something | There should be only one way: the Pythonic way | The Zen of Python |
Python Versions
The original Python version release in 1991 by Van Rossum, Python 1.0 and its successors, was eventually superseded by the widely popular Python 2.x. Care was taken to ensure that version 2.0 and beyond were backwards-compatible with Python 1.x. However, for the new Python 3.0 and beyond, backwards compatibility with Python 1 and Python 2 was broken.
This break has caused a divergence in the Python ecosystem. Some companies chose to stick with Python 2.x, which has meant that the “sunset” date or retirement date for the older version was extended from 2015 until April 2020. Now that the sunset date has been passed, there is no active work by the Python Software Foundation (PSF) on Python 2.x. Python 3.x development continues and will continue into the future, overseen by the PSF.
Van Rossum served as the Benevolent Dictator for Life of the PSF until he resigned the position in 2018.
Check out more about the history of Python: https://docs.python.org/3/faq/general.html
ArcGIS Python Versions
Since ArcMap version 9.x, Python has been integrated into the ArcGIS software suite. However, ArcGIS Desktop and ArcGIS Pro now both depend on different versions of Python.
ArcGIS Desktop: Python 2.x
ArcGIS Desktop (or ArcMap) version 9.0 and above ships with Python 2.x included. The installer for ArcGIS would automatically install Python 2.x and would add the arcpy module (originally arcgisscripting) to the Python path variable, making it available for scripting.
ArcMap, ArcCatalog, ArcGIS Engine, and ArcGIS Server all depend on arcpy and the Python 2.x version included when the ArcGIS Desktop or Enterprise software is installed.
ArcGIS Pro: Python 3.x
ArcGIS Pro, which was designed after the decision to sunset Python 2.0 was announced, was divorced from the Python 2.x ecosystem and instead shipped with Python 3.x.
Instead of arcpy, ArcGIS Pro uses the ArcGIS API for Python.
Managing both versions
The sunsetting of ArcGIS Desktop has been extended to March 2025, meaning that Python 2.7 will be included by Esri until that time despite it being officially retired by the Python Software Foundation.
Because of this, we will learn use virtual environments to manage the versions, and you will learn about the PATH and PYTHONPATH environmental variables, which control which version of Python is used to execute a script.
IMAGE CREDIT: https://media.geeksforgeeks.org/wp-content/uploads/20190502023317/TIMELINE.jpg
What is Python?
In short, Python is an application: python.exe. This application is also an executable file, meaning it can be run by itself to interpret code, or it can be called from other applications to run custom scripts. This standard interoperability is part of why it is included in applications such as ArcGIS Pro. When ArcGIS is installed, Python is also installed on your computer, along with a series of supporting files and folders.
Python includes a large standard library of tools or “modules”. These include support for internet requests, advanced math, CSV reading and writing, JSON serialization, and many more modules included in the Python core. While these tools are powerful, Python was also built to be extensible, meaning that third-party modules can be easily added to a Python installation. The ArcGIS Python modules are both good examples of extending the capabilities of Python. There are hundreds of thousands of others, covering almost any type of programming need, of varying quality.
Python is written in the programming language C. There are variants of Python written in other languages for a variety of technical reasons, but most implementations of Python are built on top of C. This means that Python is often expanded through modules built on top of C code, usually for speed improvement reasons. A Python code “layer” or “wrapper” is put on top of C code to make it work with normal Python packages, gaining the simplicity of Python and the processing speed boosts of precompiled C code. NumPy and SciPy are examples of this type of module, and are included with the ArcGIS installation of Python.
Python is free and open software, which is another reason it is packaged with so many other software applications for automation purposes. Python can also be installed separately, using a free installer from the Python Software Foundation.
Check out the Python Software Foundation on the internet: https://www.python.org/psf
Download Python versions directly from the PSF: https://www.python.org/downloads/
Where is it installed
On Windows machines, Python is not included by default – it must be installed along with ArcGIS or separately using an installer from the Python Software Foundation. Once the ArcGIS Installer is run, you will see a folder inside the C:\ drive. You can set a custom location or use the default.
Python Interpreter
When you start python.exe by double-clicking on it (see below for multiple other ways to run the executable), it starts what is known as the Python Interpreter.
This is a useful interface, allowing you to enter, one line at a time, bits of code for testing and confirmation. Once the line is entered, push Enter/Return and the code will be executed. This tool helps you both learn coding and test code in the same environment.
Starting the Interpreter
Double-clicking on python.exe from the folder or starting Python (command line) from the Start Menu, will start the interpreter, which allows for one-line commands to be executed.:
Python 3 is very similar:
What is a Python script?
The python.exe executable file, along with being a program where code can be run, will also execute Python scripts. These scripts are simple text files that can be edited by any text editing software. Python scripts are saved with the .py extension.
When a Python script is “run”, it is passed as the first command line argument to the Python executable (python.exe). This program will read and then execute the code from the top to the bottom as long as it is valid Python and it contains no errors. If there is an error encountered, the script will stop and return an error message. If there is no error, nothing will be returned unless you have added “print” statements to return messages from the main loop to the Python window as the script is running.
In this example the script is executed by “passing” the script as an argument to the executable (python.exe), which is explicitly called with the full folder path to the python.exe file to avoid path issues:
C:\Projects>C:\PythonArcGIS\ArcGIS10.5\python.exe chapter1.pyIn this example the script is executed by “passing” the script as an argument to the executable, along with optional parameters that are accepted by the script itself before being run:
C:\Projects>C:\PythonArcGIS\ArcGIS10.5\python.exe chapter1.py arg1 arg2Versions included
Python comes with two versions of the python.exe file. These are the same version of Python, to be clear, but each file has a different role. Python.exe is the main file, and the other version is pythonw.exe. This file will not open an interpreter if double-clicked, as the normal python.exe will. No interpreter is available from pythonw.exe, which is the point: it is used to execute scripts more “silently” than python.exe. Use python.exe for to start the interpreter.
How to call the executable
The Python “executable” (python.exe) is accessed to run the Python Interpreter or to run a custom Python script. There are many different ways to “call” or start the Python executable:
- Double-click on python.exe
- Starts the Python Interpreter
- Open IDLE, the included integrated development environment (IDE)
- Should be accessible in your Start menu on Windows in the ArcGIS folder
- Open a CMD terminal and type “python”
- Only works if the Python executable is in the PATH environment variable
- Using a third-party IDE such as PyCharm
- Each PyCharm project can have its own virtual environment, and therefore its own executable, or it can use the one installed by Esri when ArcGIS is installed.
- There are a lot of IDEs but PyCharm is the one I recommend for a variety of reasons.
- Using a Jupyter Notebook, which we will discuss extensively in this book
- This requires the installation of Jupyter, which is not included in the standard Python installation.
- Inside ArcGIS Desktop or ArcGIS Pro
- There are menu buttons that allow you to start a Python interpreter window inside ArcMap or ArcCatalog or ArcGIS Pro.
- Run code one line at a time or by using the load script command in the right-click menu.
IDLE development environment
The included IDE called IDLE is a useful environment that comes standard with every Python instance. IDLE is useful for the Python Interpreter, but also because you can create and execute scripts in this environment easily by opening a new script from the File menu, and then using the script’s Run menu to execute the script.
The Path environment variable
On Windows there is a system environment variable known as the Windows Path environment variable. This variable is available to all applications installed on the machine. Other programs use it for different purposes, but for Python it is used to find all available Python executables and modules.
This is important to understand because you may end up with multiple versions of Python on your computer one day, or after just one install of ArcGIS Desktop or Pro. When
If a script is run in a CMD window using the “python script.py” (passing the script to Python as an argument), and it contains import statements, then there are three things that have to happen.
First, Windows will look for an executable called python.exe in the Path. If it is there, it will then confirm that the script is valid. If it is, then Python will run the script and the Path environment variable will be checked to look for allowed locations for all modules you are trying to import.
So the Python executable cannot be run by name (instead of file location) until the python.exe is in the Path. Here is how you edit the Path variable:
Open up the Advanced System Settings in the Control Panel:
Locate and double-click on the Path variable (or press edit when selected):
Add a new line to the Path environment variable in the interface. If you have multiple version of Python and you are not using virtual environments, be sure to order the folders in the Path so that the correct version of Python is called when you type “python” into a CMD line window:
If you are not allowed to edit the Path variable, you can still run Python in the command line by referring to it using the whole path to the executable: C:\ArcGIS10.8\Python\python.exe script.py
The operating system and Python system modules
Two modules (code libraries) built into Python need to be mentioned first. The os and sys modules, also called the operating system module (os) and the Python system module (sys) are used to control Windows system operations and Python system operations respectively.
The OS module
The os module is used for many things, including folder path operations such as creating folders, removing folders, checking if a folder or file exists, or executing a file using the operating system-associated application used to run that file extension. Getting the current directory, copying files, and more is possible with this module.
In this example, a string is passed to the os.path.exists method, which is Boolean. If it returns False, the folder does not exist, and is then created using the os.mkdir method:
import os
folderpath = "C:\Test_folder"
if not os.path.exists(folderpath):
os.mkdir(folderpath)Read about the os module here: https://www.geeksforgeeks.org/os-module-python-examples/
The sys module accepts arguments
The sys module allows you to accept arguments to a script at runtime, meaning when it is executed. This is done by using the sys.argv method, which is a list containing all arguments made to Python during the executing of the script.
If a name variable is using the sys module to accept parameters, here is what the script looks like:
import sys
name = sys.argv[1]
print(name)The System path
The sys module contains the Python path or system path (system in this case means Python). This is a list that Python uses to search for importable modules, after accessing the Windows path. If you can’t edit the Windows path as explained above (due to permissions usually), you can alter the Python path at runtime using the system path.
The sys.path list is a part of the sys module built into Python:
Read more about the sys module here: https://www.geeksforgeeks.org/python-sys-module/
