Bayesian-based hyperparameter tuning

There are a couple of approaches to be used when it comes to model-based hyperparameter tuning and these approaches come together under Sequential Model-based Global Optimization (SMBO).

When you think about GridSearchCV or RandomizedSearchCV, you may rightfully feel that the way they cross validate hyperparameters is not very smart. Both pre-define sets of hyperparameters to be validated during training time and are not designed to benefit from the information that they might get during training. If you could find a way to learn from previous iterations of hyperparameter validation based on model performance, then you would have an idea about which hyperparameter set is likely to give a better performance in the next iteration.

SMBO approaches emanated from this reasoning and Bayesian-based hyperparameter optimization is one of these approaches.

Sequential Model-based Algorithm Configuration (SMAC) is a great library that uses Bayesian optimization to configure...