Bias-Variance trade-off

Averaging models helps most, when models make very different predictions.

In regression, error can be decomposed into bias and variance term.

• Bias big if model has too little capacity to fit the data
• Variance big, if model has so much capacity, that it is good at fitting the sampling error in each particular training set.

Averaging ⇒ Averaging the variance, while bias stays low.

⇒ Try to create individual predictors that disagree

Options:

• Different kind of models (Decision trees, SVMs, Gaussian Process Models)
• NN models: Differen hidden layers, differen number of units per layer, types of units, types or strength of weight penalty, learning algorithms
• Train model on different subsets of the data:
  • Bagging: Sampling with replacement, like Random Forests. Very expensive for NN
  • Boosting: Train sequence of low capacity models, weight training cases differently for each model in the seuqence (boost up weights of cases that previous models got wrong). Focus on modeling tricky cases.