In this article by Richard M. Reese and Jennifer L. Reese, authors of the book, Java for Data Science, We will discuss neural networks can be used to perform regression analysis. However, other techniques may offer a more effective solution. With regression analysis, we want to predict a result based on several input variables

(For more resources related to this topic, see here.)

We can perform regression analysis using an output layer that consists of a single neuron that sums the weighted input plus bias of the previous hidden layer. Thus, the result is a single value representing the regression.

Preparing the data

We will use a car evaluation database to demonstrate how to predict the acceptability of a car based on a series of attributes. The file containing the data we will be using can be downloaded from: http://archive.ics.uci.edu/ml/machine-learning-databases/car/car.data. It consists of car data such as price, number of passengers, and safety information, and an assessment of its overall quality. It is this latter element that we will try to predict. The comma-delimited values in each attribute are shown next, along with substitutions. The substitutions are needed because the model expects numeric data:

There are 1,728 instances in the file. The cars are marked with four classes:

Setting up the class

We start with the definition of a CarRegressionExample class, as shown next:

public class CarRegressionExample {

    public CarRegressionExample() {
        try {
            ...
        } catch (IOException | InterruptedException ex) {
            // Handle exceptions
        }
    }

    public static void main(String[] args) {
        new CarRegressionExample();
    }
}

Reading and preparing the data

The first task is to read in the data. We will use the CSVRecordReader class to get the data:

RecordReader recordReader = new CSVRecordReader(0, ",");
recordReader.initialize(new FileSplit(new File("car.txt")));
DataSetIterator iterator = new 
  RecordReaderDataSetIterator(recordReader, 1728, 6, 4);

With this dataset, we will split the data into two sets. Sixty five percent of the data is used for training and the rest for testing:

DataSet dataset = iterator.next();
dataset.shuffle();
SplitTestAndTrain testAndTrain = dataset.splitTestAndTrain(0.65);
DataSet trainingData = testAndTrain.getTrain();
DataSet testData = testAndTrain.getTest();

The data now needs to be normalized:

DataNormalization normalizer = new NormalizerStandardize();
normalizer.fit(trainingData);
normalizer.transform(trainingData);
normalizer.transform(testData);

We are now ready to build the model.

Building the model

A MultiLayerConfiguration instance is created using a series of NeuralNetConfiguration.Builder methods. The following is the dice used. We will discuss the individual methods following the code. Note that this configuration uses two layers. The last layer uses the softmax activation function, which is used for regression analysis:

MultiLayerConfiguration conf = new 
  NeuralNetConfiguration.Builder()
        .iterations(1000)
        .activation("relu")
        .weightInit(WeightInit.XAVIER)
        .learningRate(0.4)
        .list()
        .layer(0, new DenseLayer.Builder()
                .nIn(6).nOut(3)
                .build())
        .layer(1, new OutputLayer
                .Builder(LossFunctions.LossFunction
                        .NEGATIVELOGLIKELIHOOD)
                .activation("softmax")
                .nIn(3).nOut(4).build())
        .backprop(true).pretrain(false)
        .build();

Two layers are created. The first is the input layer. The DenseLayer.Builder class is used to create this layer. The DenseLayer class is a feed-forward and fully connected layer. The created layer uses the six car attributes as input. The output consists of three neurons that are fed into the output layer and is duplicated here for your convenience:

.layer(0, new DenseLayer.Builder()
        .nIn(6).nOut(3)
        .build())

The second layer is the output layer created with the OutputLayer.Builder class. It uses a loss function as the argument of its constructor. The softmax activation function is used since we are performing regression as shown here:

.layer(1, new OutputLayer
        .Builder(LossFunctions.LossFunction
                .NEGATIVELOGLIKELIHOOD)
        .activation("softmax")
        .nIn(3).nOut(4).build())

Next, a MultiLayerNetwork instance is created using the configuration. The model is initialized, its listeners are set, and then the fit method is invoked to perform the actual training. The ScoreIterationListener instance will display information as the model trains which we will see shortly in the output of this example. Its constructor argument specifies the frequency that information is displayed:

MultiLayerNetwork model = new MultiLayerNetwork(conf);
model.init();
model.setListeners(new ScoreIterationListener(100));
model.fit(trainingData);

We are now ready to evaluate the model.

Evaluating the model

In the next sequence of code, we evaluate the model against the training dataset. An Evaluation instance is created using an argument specifying that there are four classes. The test data is fed into the model using the output method. The eval method takes the output of the model and compares it against the test data classes to generate statistics. The getLabels method returns the expected values:

Evaluation evaluation = new Evaluation(4);
INDArray output = model.output(testData.getFeatureMatrix());
evaluation.eval(testData.getLabels(), output);
out.println(evaluation.stats());

The output of the training follows, which is produced by the ScoreIterationListener class. However, the values you get may differ due to how the data is selected and analyzed. Notice that the score improves with the iterations but levels out after about 500 iterations:

12:43:35.685 [main] INFO  o.d.o.l.ScoreIterationListener - Score at iteration 0 is 1.443480901811554
12:43:36.094 [main] INFO  o.d.o.l.ScoreIterationListener - Score at iteration 100 is 0.3259061845624861
12:43:36.390 [main] INFO  o.d.o.l.ScoreIterationListener - Score at iteration 200 is 0.2630572026049783
12:43:36.676 [main] INFO  o.d.o.l.ScoreIterationListener - Score at iteration 300 is 0.24061281470878784
12:43:36.977 [main] INFO  o.d.o.l.ScoreIterationListener - Score at iteration 400 is 0.22955121170274934
12:43:37.292 [main] INFO  o.d.o.l.ScoreIterationListener - Score at iteration 500 is 0.22249920540161677
12:43:37.575 [main] INFO  o.d.o.l.ScoreIterationListener - Score at iteration 600 is 0.2169898450109222
12:43:37.872 [main] INFO  o.d.o.l.ScoreIterationListener - Score at iteration 700 is 0.21271599814600958
12:43:38.161 [main] INFO  o.d.o.l.ScoreIterationListener - Score at iteration 800 is 0.2075677126088741
12:43:38.451 [main] INFO  o.d.o.l.ScoreIterationListener - Score at iteration 900 is 0.20047317735870715

This is followed by the results of the stats method as shown next. The first part reports on how examples are classified and the second part displays various statistics:

Examples labeled as 0 classified by model as 0: 397 times
Examples labeled as 0 classified by model as 1: 10 times
Examples labeled as 0 classified by model as 2: 1 times
Examples labeled as 1 classified by model as 0: 8 times
Examples labeled as 1 classified by model as 1: 113 times
Examples labeled as 1 classified by model as 2: 1 times
Examples labeled as 1 classified by model as 3: 1 times
Examples labeled as 2 classified by model as 1: 7 times
Examples labeled as 2 classified by model as 2: 21 times
Examples labeled as 2 classified by model as 3: 14 times
Examples labeled as 3 classified by model as 1: 2 times
Examples labeled as 3 classified by model as 3: 30 times

==========================Scores========================================
 Accuracy:  0.9273
 Precision: 0.854
 Recall:    0.8323
 F1 Score:  0.843
========================================================================

The regression model does a reasonable job with this dataset.

Summary

In this article, we examined deep learning and regression analysis. We showed how to prepare the data and class, build the model, and evaluate the model. We used sample data and displayed output statistics to demonstrate the relative effectiveness of our model.

Resources for Article:

Further resources on this subject:

• KnockoutJS Templates [article]
• The Heart of It All [article]
• Bringing DevOps to Network Operations [article]