Predict Your Linear Data
Compute simple linear models to predict dependent data and assess performance using observations held out from model fitting.
Learning objectives
At the end of this session you should be able to
- evaluate the prediction performance of a statistical regression model,
- decide which cross-validation strategy to use, and
- calculate one root mean squared error (RMSE) per validation run and interpret its variation across runs.
Basic idea of statistical prediction
We now shift our emphasis from interpreting fitted relationships to evaluating predictions for held-out observations. Recall the distinction between description, inference, and prediction in Unit 05.
Fit on training observations, then predict the response for validation observations left out of fitting. Earlier diagnostic concerns do not prevent us from measuring these errors, but cross-validation does not resolve those concerns.
The graphic above shows a scatter plot of data pairs between an independent (X axis) and a dependent (Y axis) variable. The red lines represent linear regression models, each computed on a different subset of the data pairs where one data pair has been removed. The difference between the left-out observation and its prediction by the model provides an estimate of prediction error. Such an error estimation strategy is generally called (leave-one-out) cross validation.
Have a look at this video to get started with the basic concepts of cross-validation.
The worked example distinguishes repeated random holdout from four-fold partitioning and computes one RMSE per complete run. Random splits do not by themselves remove spatial dependence.
For a deeper look into cross-validation strategies related to spatially and temporally dependent data, have a look at Meyer et al. 2018.
Comments?
You can leave comments below if you have questions or remarks about any of the text or code in this unit. Please copy the corresponding line into your comment to make it easier to answer your question.