Summary

In this chapter, we have turned our focus to a notebook approach to Apache Spark, and specifically developed R notebooks for estimating and assessing models, with which we developed risk scores to help the company XST to improve their risk management.

We first selected a few machine learning methods with our focus on the logistic regression method, along with random forest and decision trees. We then worked on data cleaning and feature development by using a special tool called OpenRefine. Next, we estimated the model coefficients. We then evaluated these estimated models by using a confusion matrix, ROC, and KS. Then we interpreted our machine learning results. And finally, we deployed our machine learning results with a scoring approach.

With a notebook approach, all the preceding machine learning steps are implemented in R, with all the R codes stored in notebooks so that the process is repeatable and can be partially automated. To get everything organized well and integrated with...