Measuring ROC AUC in a custom callback

Let's use one more callback. This time, we will build a custom callback that computes Receiver Operating Characteristic Area Under the Curve (ROC AUC) at the end of every epoch, on both training and testing sets.  

Creating a custom callback in Keras is actually really simple. All we need to do is create a class, inherent Callback, and override the method we need. Since we want to calculate the ROC AUC score at the end of each epoch, we will override on _epoch_end:

from keras.callbacks import Callback

class RocAUCScore(Callback):
    def __init__(self, training_data, validation_data):
        self.x = training_data[0]
        self.y = training_data[1]
        self.x_val = validation_data[0]
        self.y_val = validation_data[1]
        super(RocAUCScore, self).__init__()

    def on_epoch_end(self, epoch, logs={}):
        y_pred = self.model.predict(self.x)
        roc = roc_auc_score(self.y, y_pred)
        y_pred_val = self.model.predict(self.x_val...