Learning Geospatial Analysis with Python-Second Edition

As a geospatial analyst, you may frequently encounter the following general data types:

Each format contains its own challenges for access and processing. When you perform analysis on data, usually you have to do some form of preprocessing first. You might clip or subset a satellite image of a large area down to just your area of interest, or you might reduce the number of points in a collection to just the ones meeting certain criteria in your data model. A good example of this type of preprocessing is the SimpleGIS example at the end of Chapter 1, Learning Geospatial Analysis with Python. The state dataset included just the state...

Despite dozens of formats, geospatial data have common traits. Understanding these traits can help you approach and understand unfamiliar data formats by identifying the ingredients common to nearly all spatial data. The structure of a given data format is usually driven by its intended use. Some data is optimized for efficient storage or compression, some is optimized for efficient access, some is designed to be lightweight and readable (web formats), while other data formats seek to contain as many different data types as possible.

Interestingly, some of the most popular formats today are also some of the simplest and even lack features found in more capable and sophisticated formats. Ease of use is extremely important to geospatial analysts because so much time is spent integrating data into geographic information systems as well as exchanging data among analysts. Simple data formats facilitate these activities the best.

Geospatial datasets are often very large files easily reaching hundreds of megabytes or even several gigabytes in size. Geospatial software can be quite slow in trying to repeatedly access large files when performing analysis. As discussed briefly in Chapter 1, Learning Geospatial Analysis with Python, spatial indexing creates a guide, which allows software to quickly locate query results without examining every single feature in the dataset. Spatial indexes allow software to eliminate possibilities and perform more detailed searches or comparisons on a much smaller subset of the data.

Overview data is most commonly found in raster formats. Overviews are resampled, lower resolution versions of raster datasets that provide thumbnail views or simply faster loading image views at different map scales. They are also known as pyramids and the process of creating them is known as pyramiding an image. These overviews are usually preprocessed and stored with the full resolution data either embedded with the file or in a separate file. The compromise of this convenience is that the additional images add to the overall file size of the dataset; however, they speed up image viewers. Vector data also has a concept of overviews, usually to give a dataset geographic context in an overview map. However, because vector data is scalable, reduced size overviews are usually created on the fly by software using a generalization operation as mentioned in Chapter 1, Learning Geospatial Analysis with Python.

Occasionally, vector data is rasterized by converting it into a thumbnail image...

As discussed in Chapter 1, Learning Geospatial Analysis with Python, metadata is any data that describes the associated dataset. Common examples of metadata include basic elements such as the footprint of the dataset on the Earth as well as more detailed information such as spatial projection and information describing how the dataset was created. Most data formats contain the footprint or bounding box of the data on the Earth. Detailed metadata is typically stored in a separate location in a standard format such as the U.S. Federal Geographic Data Committee (FGDC) Content Standard for Digital Geospatial Metadata (CSDGM), ISO, or the newer European Union initiative, which includes metadata requirements, called the Infrastructure for Spatial Information in the European Community (INSPIRE).

The preceding elements can be stored in a variety of ways in a single file, multiple files, or database depending on the format. Additionally, this geospatial information can be stored in a variety of formats, including embedded binary headers, XML, database tables, spreadsheets/CSV, separate text, or binary files.

Human readable formats such as XML files, spreadsheets, and structured text files require only a text editor to investigate. These files are also easily parsed and processed using Python's built-in modules, data types, and string manipulation functions. Binary-based formats are more complicated. It is thus typically easier to use a third-party library to deal with binary formats.

However, you don't have to use a third-party library, especially if you just want to investigate the data at a high level. Python's built-in struct module has everything that you need. The struct module lets you read and write binary data as strings. When using the struct module, you need...

Vector data is, by far, the most common geospatial format because it is the most efficient way to store spatial information, and in general, requires less computer resources to store and process than raster data. The OGC has over 16 formats directly related to vector data. Vector data stores only geometric primitives including points, lines, and polygons. However, only the points are stored for each type of shape. For example, in the case of a simple straight vector line shape, only the end points would be necessarily stored and defined as a line. Software displaying this data would read the shape type and then connect the end points with a line dynamically.

Geospatial vector data is similar to the concept of vector computer graphics with some notable exceptions. Geospatial vector data contains positive and negative Earth-based coordinates, while vector graphics typically store computer screen coordinates. Geospatial vector data is also usually linked to other information about...

Raster data consists of rows and columns of cells or pixels, with each cell representing a single value. The easiest way to think of raster data is as images, which is how they are typically represented by software. However, raster datasets are not necessarily stored as images. They can also be ASCII text files or Binary Large Objects (BLOBs) in databases.

Another difference between geospatial raster data and regular digital images is resolution. Digital images express resolution as dots-per-inch if printed in full size. Resolution can also be expressed as the total number of pixels in the image defined as megapixels. However, geospatial raster data uses the ground distance that each cell represents. For example, a raster dataset with a two-foot resolution means that a single cell represents two feet on the ground, which also means that only objects larger than two feet can be identified visually in the dataset.

Raster datasets may contain multiple bands, meaning that different...

Point cloud data is any data collected as the (x, y, z) location of a surface point based on some sort of focused energy return. Point cloud data can be created using lasers, radar waves, acoustic soundings, or other waveform generation devices. The spacing between points is arbitrary and dependent on the type and position of the sensor collecting the data. In this book, we will primarily be concerned with LIDAR data and radar data. Radar point cloud data is typically collected on space missions while LIDAR is typically collected by terrestrial or airborne vehicles. Conceptually, both the types of data are similar.

LIDAR uses powerful laser range-finding systems to model the world with very high precision. The term LIDAR or LiDAR is a combination of the words light and radar. Some people claim it also stands for Light Detection and Ranging. LIDAR sensors can be mounted on aerial platforms including satellites, airplanes, or helicopters. They can also be mounted on vehicles...

Geospatial web services allow users to perform data discovery, data visualization, and data access across the web. Web services are usually accessed by applications based on user input such as zooming an online map or searching a data catalogue. The most common protocols are the Web Map Service (WMS) that returns a rendered map image and Web Feature Service (WFS) that typically returns GML, which was mentioned in this chapter's introduction. Many WFS services can also return KML, JSON, zipped shapefiles, and other formats. These services are called through HTTP GET requests. The following URL is an example of a WMS GET request, which returns a map image of the world that is 640 pixels wide and 400 pixels tall and an EPSG code of 900913:

http://osm.woc.noaa.gov/mapcache?SERVICE=wms&VERSION=1.1.1&REQUEST=GetMap&FORMAT=image/png&WIDTH=600&HEIGHT=400&LAYERS=osm&SRS=EPSG:900913&BBOX=-20037508,-20037508,20037508,20037508

Web services are rapidly evolving...

You now have the background needed to work with common types of geospatial data. You also know the common traits of geospatial datasets that will allow you to evaluate unfamiliar types of data and identify key elements that will drive you towards which tools to use when interacting with this data. In Chapter 3, The Geospatial Technology Landscape, we'll examine the modules and libraries available to work with these datasets. As with all the code in this book, whenever possible, we'll use only pure Python and Python standard libraries.