Example: Feature Selection in Multiple Regression
Unit 06 assessed predictions from one predictor. Now we ask whether adding another predictor improves the model. Forward selection starts with an intercept-only model, tries each available addition, keeps the best improvement and repeats until no addition improves the score.
Which measure answers which question?
| Measure | What it describes | How to use it here |
|---|---|---|
| R squared (R²) | Fraction of response variation accounted for in the fitting data. | Describes training fit; adding predictors cannot decrease it on the same rows. |
| Adjusted R² | Training fit adjusted for the number of predictors. | Can decrease when a predictor contributes little; still a training summary. |
| Akaike information criterion (AIC) | Balance between fit and the number of estimated parameters. | Prefer smaller values for these models fitted to the same response observations; it is a relative score, not an error in response units. |
| Root mean squared error (RMSE) | Size of prediction errors in response units. | Prefer smaller values on the same held-out observations, as in Unit 06. |
We compare candidates using AIC first and held-out RMSE afterward. Once a score helps choose predictors, it is a selection score. Final evaluation needs observations kept out of that choice. See the R AIC documentation for comparison requirements.
Download selection-example.csv into Data/. This is the existing eleven-row teaching example derived from Anscombe, now supplied directly. It illustrates model selection, not a real prediction problem: x2 was constructed using the response. Real predictors must be available without knowing the response to be predicted.
df <- read.csv("Data/selection-example.csv")
head(df)
## y x1 x2 x3 x4
## 1 8.04 10 14.440800 9.3 7.3
## 2 6.95 8 9.750327 8.4 8.6
## 3 7.58 13 9.480255 13.0 7.7
## 4 8.81 9 8.810000 10.0 7.5
## 5 8.33 11 16.831091 11.6 8.0
## 6 9.96 14 15.960818 14.5 7.9
Multiple linear regression model
A second predictor adds another slope to the model formula:
model <- lm(y ~ x1 + x2, data = df)
summary(model)$coefficients
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.0979017 1.1869105 1.767531 0.11511842
## x1 0.3480164 0.1454046 2.393434 0.04362144
## x2 0.2022383 0.1277803 1.582704 0.15214663
Each slope describes the association with that predictor while holding the other predictor fixed. The residual checks from Unit 05 still matter. We will choose predictors using model scores rather than individual coefficient p values.
Compare models on the same observations
Use the same complete rows for every candidate. Otherwise a score could change because different observations were used, rather than because a predictor was added. The teaching table is complete.
For Assignment 07, first select the 2014 district cohort and keep rows with values in all ten crop-yield columns. Keep region IDs and report exclusions. Only the other nine crop yields are candidate predictors of winter wheat; IDs and years are not crop yields. Excluding incomplete records can limit which districts the results represent.
Forward feature selection
Compare one addition
An intercept-only baseline, y ~ 1, predicts the same fitted mean for every row. Compare it with two possible first additions:
baseline <- lm(y ~ 1, data = df)
with_x1 <- lm(y ~ x1, data = df)
with_x2 <- lm(y ~ x2, data = df)
AIC(baseline, with_x1, with_x2)
## df AIC
## baseline 2 49.76201
## with_x1 3 39.68137
## with_x2 3 42.62529
x1 gives the smaller AIC in this comparison. The next step asks whether adding another variable improves the model that already contains x1.
Score the available additions
To repeat the comparison, we need to build formulas from variable names. reformulate() turns a vector of predictor names into a formula:
reformulate(c("x1", "x2"), response = "y")
## y ~ x1 + x2
## <environment: 0x58e0b3ff0438>
The following short function fits one model per possible addition and returns its AIC. selected contains the predictors already kept; remaining contains those still available.
score_additions <- function(data, selected, remaining) {
unlist(lapply(remaining, function(variable) {
formula <- reformulate(c(selected, variable), response = "y")
model <- lm(formula, data = data)
AIC(model)
}))
}
Start with no predictors. character(0) is an empty vector of names. Compare all four first additions against the baseline:
remaining <- c("x1", "x2", "x3", "x4")
scores <- score_additions(df, selected = character(0), remaining)
data.frame(Added = remaining, AIC = scores)
## Added AIC
## 1 x1 39.68137
## 2 x2 42.62529
## 3 x3 41.01076
## 4 x4 48.16607
AIC(baseline)
## [1] 49.76201
x1 gives the largest improvement. Keep it and compare the three remaining additions:
selected <- "x1"
remaining <- c("x2", "x3", "x4")
scores <- score_additions(df, selected, remaining)
data.frame(Added = remaining, AIC = scores)
## Added AIC
## 1 x2 38.68492
## 2 x3 41.39423
## 3 x4 41.04378
AIC(with_x1)
## [1] 39.68137
Adding x2 improves AIC. Check whether either remaining addition improves this new model:
selected <- c("x1", "x2")
remaining <- c("x3", "x4")
scores <- score_additions(df, selected, remaining)
data.frame(Added = remaining, AIC = scores)
## Added AIC
## 1 x3 40.56990
## 2 x4 39.54815
AIC(lm(y ~ x1 + x2, data = df))
## [1] 38.68492
Neither addition improves AIC, so stop with x1 and x2. Use unrounded scores for decisions. Keep the current model if the best addition only ties it; for equally good improving candidates, document a rule such as taking the first in the supplied order.
Take home: compare additions with the current model, keep only an improvement, then repeat. The marked assignment asks you to implement that repetition yourself.
Which feature selection to choose?
Forward selection takes the best available addition at each step; it does not try every possible predictor combination. Correlated predictors can lead to different selected sets, so the chosen variables are not necessarily uniquely important. Ordinary coefficient p values after selection also ignore the fact that the predictors were chosen using the data.
Compare candidates with four-fold validation
Now use held-out RMSE instead of AIC. We reuse the prediction-vector approach from Unit 06.
For this prediction example, create 100 independent synthetic observations. rnorm() draws random values from a normal distribution. Here the response depends on x1 plus random error; x2 is an unrelated candidate. Reserve 20 rows for final evaluation and use the other 80 for selection:
set.seed(70)
selection_data <- data.frame(x1 = rnorm(100), x2 = rnorm(100))
selection_data$y <- 3 + 2 * selection_data$x1 + rnorm(100)
development_rows <- sample(nrow(selection_data), 80)
development <- selection_data[development_rows, ]
final_test <- selection_data[-development_rows, ]
fold <- sample(rep(1:4, length.out = nrow(development)))
table(fold)
## fold
## 1 2 3 4
## 20 20 20 20
The function below accepts a model formula. Each row receives one held-out prediction; the function returns one RMSE across all four folds:
four_fold_rmse <- function(data, formula, fold) {
predicted <- rep(NA, nrow(data))
for (k in 1:4) {
train <- data[fold != k, ]
validation <- data[fold == k, ]
model <- lm(formula, data = train)
predicted[fold == k] <- predict(model, newdata = validation)
}
sqrt(mean((data$y - predicted)^2))
}
four_fold_rmse(development, y ~ x1, fold)
## [1] 0.8798429
Now compare two formulas on those same rows and folds:
candidate_formulas <- list(y ~ x1, y ~ x1 + x2)
rmse <- unlist(lapply(candidate_formulas, function(formula) {
four_fold_rmse(development, formula, fold)
}))
data.frame(Model = c("x1", "x1 + x2"), RMSE = rmse)
## Model RMSE
## 1 x1 0.8798429
## 2 x1 + x2 0.8911033
Take home: changing predictors should be the only difference between candidate comparisons. Keep the observations and fold assignments fixed throughout a selection path.
Exercise: Compare one addition
Allow 15 minutes. Using the eleven-row df, start from y ~ x2. Fit y ~ x2 + x1 and y ~ x2 + x3, compare their AIC values with the starting model and decide whether to add a variable. Write the fits explicitly, then check with score_additions().
Explain why the best addition still has to improve the starting score. No separate submission is required.
Keep final evaluation independent
The minimum RMSE above helped choose a formula. To assess that choice separately, freeze the formula, fit it to all development rows and predict the untouched final-test rows:
selected_formula <- candidate_formulas[[which.min(rmse)]]
final_model <- lm(selected_formula, data = development)
final_pred <- predict(final_model, newdata = final_test)
sqrt(mean((final_test$y - final_pred)^2))
## [1] 0.8063289
plot(final_test$y, final_pred, xlab = "Observed final-test response",
ylab = "Final-test prediction", asp = 1)
abline(0, 1, col = "red")

Take home: selection chooses the model; final evaluation assesses the frozen choice. If the final score is used to revise the model or try another split, it becomes part of development.
Assignment 07 asks for the selection loop with AIC and four-fold RMSE, its candidate-score plots and selected variables. A separate final-test submission is not required. Label the reported scores as selection criteria; random district folds within 2014 do not establish prediction in a new region or future year.