Cross-Validation

Use

Rather than splitting a dataset up into training and test sets, it is possible to “cross validate” by iteratively leaving out a certain fixed number of items as the test set, predicting using the rest of the training set, and replacing the item and moving on to the next. If this is done for a fixed proportion of the dataset at a time (say 1/5), it is called k-fold cross validation (in this case, k=5). If only one item is put in the test set during each iteration (which means there will be n iterations), this method is called leave-one-out cross-validation. Resulting accuracies are then averaged to get the final result.

Example

I have 9 images. Classify on images 2-9, test on image 1, classify on images 1,3-9, test on 2, etc., then average accuracies.

Caveats

• Though useful for minimizing the effects of overfitting, cross-validation was found unable to eliminate it completely.
• You can't make sure that the model will generalize to the properties of other datasets using this technique.