Feature selection
Our CPU model only came with six features. Often, we encounter real-world datasets that have a very large number of features arising from a diverse array of measurements. Alternatively, we may have to come up with a large number of features when we aren't really sure what features will be important in influencing our output variable. Moreover, we might have categorical variables with many possible levels from which we are forced to create a large number of new indicator variables, as we saw in Chapter 1, Gearing Up for Predictive Modeling. When our scenario involves a large number of features, we often find that our output only depends on a subset of these. Given k input features, there are 2k distinct subsets that we can form, so for even a moderate number of features the space of subsets is too large for us to fully explore by fitting a model on each subset.
Tip
One easy way to understand why there are 2k possible feature subsets is this: we can assign a unique identifying...
