Table of Contents

Crossvalidation
- Train/Dev/Test sets
- Mismatched train/test distribution

Crossvalidation

Train/Dev/Test sets

Training set | Hold-out / Dev set | Test set

Goal of dev set: Which algorithm does better
Goal of test set: Performance estimate of final classifier

For NNs with large data sets: Often 98%/1%/1%

Mismatched train/test distribution

Should have same distribution, but sometimes training set needs more data which comes from different distribution
- Then dev/test set should come from same distribution

Sometimes there is no test set, so you don't have a unbiased estimation (might overfit to dev set).