Predict Your Linear Data

Compute simple linear models to predict dependent data and assess performance using observations held out from model fitting.

Learning objectives

At the end of this session you should be able to

  • evaluate the prediction performance of a statistical regression model,
  • decide which cross-validation strategy to use, and
  • calculate one root mean squared error (RMSE) per validation run and interpret its variation across runs.

Basic idea of statistical prediction

We now shift our emphasis from interpreting fitted relationships to evaluating predictions for held-out observations. Recall the distinction between description, inference, and prediction in Unit 05.

Fit on training observations, then predict the response for validation observations left out of fitting. Earlier diagnostic concerns do not prevent us from measuring these errors, but cross-validation does not resolve those concerns.

Cross validation example using linear regressions
Linear regression models computed with left-out data pairs.

The graphic above shows a scatter plot of data pairs between an independent (X axis) and a dependent (Y axis) variable. The red lines represent linear regression models, each computed on a different subset of the data pairs where one data pair has been removed. The difference between the left-out observation and its prediction by the model provides an estimate of prediction error. Such an error estimation strategy is generally called (leave-one-out) cross validation.

Have a look at this video to get started with the basic concepts of cross-validation.

The worked example distinguishes repeated random holdout from four-fold partitioning and computes one RMSE per complete run. Random splits do not by themselves remove spatial dependence.

For a deeper look into cross-validation strategies related to spatially and temporally dependent data, have a look at Meyer et al. 2018.

Extrapolating comic: extending a line from zero husbands yesterday to one today predicts dozens of husbands next month, illustrating an absurd extrapolation.
Randall Munroe / xkcd, CC BY-NC 2.5

Comments?

You can leave comments below if you have questions or remarks about any of the text or code in this unit. Please copy the corresponding line into your comment to make it easier to answer your question.

Updated: