Chapter 2. Data Pipelines

In the first chapter, you were acquainted with some rudimentary concepts regarding data processing, clustering, and classification.

This chapter is dedicated to the creation and maintenance of a flexible end-to-end workflow to train and classify data. The first section of the chapter introduces a data-centric (functional) approach to create number crunching applications, followed by a description of a configurable workflow computation model. The chapter concludes with an overview of different model validation techniques.

You will learn how to do the following:

• Apply the concept of monadic design to create dynamic workflows

• Leverage some of Scala's advanced patterns, such as the cake pattern, to build portable computational workflows

• Take into account the bias-variance trade-off in selecting a model

• Overcome overfitting in modeling

• Break down data into training, test and validation sets

• Implement model validation in Scala using precision, recall, and F score