Select Your Variables
Evaluate the importance of your independent variables and select a useful subset for your prediction model.
Learning objectives
At the end of this session you should be able to
- compare one predictor addition with the current model,
- explain why AIC and held-out RMSE answer different questions, and
- implement forward selection with a clear stopping rule.
Basic idea of variable selection
Add explanatory variables only when they improve the chosen comparison score. Overfitting means following chance features of the training sample that do not help prediction.
The graphic illustrates a possible pattern: held-out performance improves with useful predictors, then worsens as more predictors are added. This decline is possible, not inevitable. Ordinary training R² cannot decrease when predictors are added on the same rows, so training fit alone is not a sufficient selection rule. The worked example compares models using AIC (a balance between fit and complexity) and held-out RMSE (prediction error).
Scores used to choose predictors belong to model selection. The worked example shows how a reserved test set provides a separate final assessment after that choice.
For a deeper look into variable selection have a look at Meyer et al. 2016
Comments?
You can leave comments below if you have questions or remarks about any of the text or code in this unit. Please copy the corresponding line into your comment to make it easier to answer your question.