Unmarked Assignment: Forest vs. Agriculture Revisited

How accurately does agricultural share predict forest share in districts left out of model fitting? Reuse your Unit 05 model and assess its prediction errors with repeated validation runs.

After completing this worksheet you should have gained some experience in applying cross-validation for estimating the performance of a model.

Things you need for this worksheet

  • R — the interpreter can be installed on any operating system.
  • RStudio — we recommend using RStudio for (interactive) programming with R.
  • Git installed and available to RStudio. GitHub Desktop is an optional graphical client.
  • Data on percentage shares of land use at several administrative levels provided by the course instructors.

Input from Unit 04

Reuse Output/lu_clean.rds or download lu_clean.rds and save it in Output/. This is the wide land-use table from the Unit 04 land-use exercise, as in Assignment 05. Load it with readRDS("Output/lu_clean.rds") from your course project. Keep its region/year identifiers when designing validation; the input contains several administrative levels and repeated regions across years.

Prepare comparable observations

Carry forward the same cohort as Assignment 05: 2014, explicitly recorded Admin_unit of District or Urban district, and complete Agriculture–Forest pairs. Omit other levels and unresolved types, retain region IDs, report exclusions, and check that each region occurs once. Split this prepared cohort rather than the full mixed-year, mixed-level input.

As in Assignment 05, both land uses are shares of a district’s total area; predictions describe an association, not a causal effect. Random splits assess prediction for held-out records within this historical district cohort. Nearby districts can still be related; this design does not establish transfer to an unseen geographic area or a future year.

Even if your Unit 05 diagnostics raised concerns, held-out RMSE can measure prediction error for these observations; it does not resolve those concerns.

Prediction error of a linear model

Write an R Markdown document that assesses predictions of forest share (Forest, response) from agricultural share (Agriculture, predictor). Choose and explain a repeated validation procedure: repeated random holdout or repeated complete fold partitions, as introduced in the cross-validation example. Repeated random holdout is the simplest starting point; either procedure is valid. Implement a function that returns one RMSE for one complete run, then repeat it. Choose and document the number of repetitions and the training fraction (or number of folds); no fixed repetition count is prescribed here. Set the seed once before random splitting, keep region IDs and the split indices for each run, and record your seed and R version. The order of the saved splits identifies the runs; a separate prediction-record table is not required.

Calculate one RMSE per run from that run’s held-out predictions. For a fold-based design, pool the predictions from all folds within each complete run before calculating RMSE. Plot the resulting distribution across runs and report its mean and standard deviation, both in forest percentage points. The SD describes variation between splits; it is not a standard error or confidence interval for performance on new data.

Write exactly one sentence summarizing the size of the prediction errors, their variation between splits and the limitation to this historical district cohort.

Save your R Markdown (.Rmd) document in your course repository, knit it, update (i.e. commit) your local repository and publish (i.e. push) it to GitHub. Make sure that the created HTML file is also part of your course repository on GitHub.

Updated: