Java Data Analysis

Data is categorized into types. A data type identifies not only the form of the data but also what kind of operations can be performed upon it. For example, arithmetic operations can be performed on numerical data, but not on text data.

A data type can also determine how much computer storage space an item requires. For example, a decimal value like 3.14 would normally be stored in a 32-bit (four bytes) slot, while a web address such as https://google.com might occupy 160 bits.

Here is a categorization of the main data types that we will be working with in this book. The corresponding Java types are shown in parentheses:

In computer science, we think of a variable as a storage location that holds a data value. In Java, a variable is introduced by declaring it to have a specific type. For example, consider the following statement:

String lastName;

It declares the variable lastName to have type String.

We can also initialize a variable with an explicit value when it is declared, like this:

double temperature = 98.6;

Here, we would think of a storage location named temperature that contains the value 98.6 and has type double.

Structured variables can also be declared and initialized in the same statement:

int[] a = {88, 11, 44, 77, 22};

This declares the variable a to have type int[] (array of ints) and contain the five elements specified.

In data analysis, it is convenient to think of the data as points of information. For example, in a collection of biographical data, each data point would contain information about one person. Consider the following data point:

("Adams", "John", "M", 26, 704601929)

It could represent a 26-year-old male named John Adams with ID number 704601929.

We call the individual data values in a data point fields (or attributes). Each of these values has its own type. The preceding example has five fields: three text and two numeric.

The sequence of data types for the fields of a data point is called its type signature. The type signature for the preceding example is (text, text, text, numeric, numeric). In Java, that type signature would be (String, String, String, int, int).

A dataset is a set of data points, all of which have the same type signature. For example, we could have a dataset that represents a group of people, each point representing a unique member of the group. Since...

In a relational database, we think of each dataset as a table, with each data point being a row in the table. The dataset's signature defines the columns of the table.

Here is an example of a relational database table. It has four rows and five columns, representing a dataset of four data points with five fields:

Because a database table is really a set of rows, the order of the rows is irrelevant, just as the order of the data points in any dataset is irrelevant. For the same reason, a database table may not contain duplicate rows and a dataset may not contain duplicate data points.

A dataset of key-value pairs is usually implemented as a hash table. It is a data structure in which the key acts like an index into the set, much like page numbers in a book or line numbers in a table. This direct access is much faster than sequential access, which is like searching through a book page-by-page for a certain word or phrase.

In Java, we usually use the java.util.HashMap<Key,Value> class to implement a key-value pair dataset. The type parameters Key and Value are specified classes. (There is also an older HashTable class, but it is considered obsolete.)

Here is a data file of seven South American countries:

Figure 2-1 Countries data file

Here is a Java program that loads this data into a HashMap object:

Listing 2-1 HashMap example for Countries data

The Countries.dat file is in the data folder. Line 15 instantiates a java.io.File object named dataFile to represent the file. Line 16 instantiates a java.util.HashMap object named dataset. It is structured to have...

The Countries.dat data file in the preceding example is a flat file—an ordinary text file with no special structure or formatting. It is the simplest kind of data file.

Another simple, common format for data files is the comma separated values (CSV) file. It is also a text file, but uses commas instead of blanks to separate the data values. Here is the same data as before, in CSV format:

Figure 2-4 A CSV data file

For Java to process this correctly, we must tell the Scanner object to use the comma as a delimiter. This is done at line 18, right after the input object is instantiated:

Listing 2-3 A program for reading CSV data

The regular expression ,|\\s means comma or any white space. The Java symbol for white space (blanks, tabs, newline, and so on.) is denoted by '\s'. When used in a string, the backslash character itself must be escaped with another preceding backslash...

Generating numerical test data is easy with Java. It boils down to using a java.util.Random object to generate random numbers.

Listing 2-13 Generating random numeric data

This program generates the following CSV file of eight rows and five columns of random decimal values.

Figure 2-9 Test data file

Metadata

Metadata is data about data. For example, the preceding generated file could be described as eight lines of comma-separated decimal numbers, five per line. That's metadata. It's the kind of information you would need, for example, to write a program to read that file.

That example is quite simple: the data is unstructured and the values are all the same type. Metadata about structured data must also describe that structure.

The metadata of a dataset may be included in the same file as the data itself. The preceding example could be modified with a header line like this:

Figure 2-10 Test data file fragment with metadata in header

Note

When reading a data file in Java, you...

This chapter discussed various organizational processes used to prepare data for analysis. When used in computer programs, each data value is assigned a data type, which characterizes the data and defines the kind of operations that can be performed upon it.

When stored in a relational database, data is organized into tables, in which each row corresponds to one data point, and where all the data in each column corresponds to a single field of a specified type. The key field(s) has unique values, which allows indexed searching.

A similar viewpoint is the organization of data into key-value pairs. As in relational database tables, the key fields must be unique. A hash table implements the key-value paradigm with a hash function that determines where the key's associated data is stored.

Data files are formatted according to their file type's specifications. The comma-separated value type (CSV) is one of the most common. Common structured data file types include XML and JSON.

The information...