Introduction
Check, clean and document a dataset so that its observations, values and limitations are clear before analysis.
Learning objectives
At the end of this unit you should be able to
- define an observation and identify a suitable key;
- inspect types, source symbols and missing values before conversion;
- sort complete rows without separating measurements from their identifiers;
- split compound labels without guessing missing administrative information;
- write a small cleaning function, save it in an R script and load it with
source(); - reshape a table and explain what a row represents before and afterward;
- match tables by keys and identify duplicates or unmatched records;
- save a reproducible wide analysis table and document unresolved cases.
How this unit is organized
Build on Unit 03’s shared project layout: keep your Rmd beside the .Rproj, inputs in Data/, function scripts in Scripts/ and generated analysis results in Output/. Reuse Unit 02’s functions and returned values when writing your cleaning helper.
The examples use functions included with R.
Pages labeled Example support the instructor demonstrations. The Exercise is your individual practice in class; the Unmarked Assignment is the follow-up crop task.
Start with Cleaning 101, then work through missing values and sorting. The focused examples on splitting text, reshaping and grouped summaries and matching tables introduce operations used in the land-use exercise. The crop assignment transfers them to another source file.
In class, the instructor introduces the cleaning steps with small general examples. You then apply them individually to the land-use dataset, followed by a shared question collection and class discussion. The worked example is available as an expandable reference.
Keep reporting years such as 2014 as integers. The annual regional tables do not require full dates or timestamps.
Additional explanations are available in sorting and combining data in Base R.
Data for Units 04–07
We retain two historical regional datasets because they provide real cleaning problems and support the later modeling tasks. The English teaching copies translate headings, introductory text and regional descriptors while retaining the source values, IDs, years, indentation, decimal commas and symbols. They are teaching translations, not newer official English exports. Proper place names remain in their source spelling.
Download the English file specified by the exercise or assignment into Data/, keeping its filename. The same files are available on ILIAS. If your browser displays a text file, use Save As and move it into Data/.
| Input | Coverage and role | File layout |
|---|---|---|
| regional-land-use-1996-2014.txt | 5,250 region-year records; four land-use percentages; in-class cleaning exercise. | UTF-8; semicolons; four introductory lines; header line 5. |
| regional-crop-yields-1999-2015.txt | 8,925 region-year records; ten crop yields; independent cleaning assignment. | UTF-8; semicolons; six introductory lines; header line 7; five footer lines. |
The attributed sources are GENESIS/Regional Atlas AI001 (AI001_gebiet_flaeche.txt) and 115-46-4 (115-46-4_feldfruechte.txt), supplied by the course instructors from the Regional Database Germany. The unchanged exports are provenance, rather than extra required learner downloads. The English crosswalk records the original headings and translated labels; source and preparation notes explain the files. -, . and / are retained; preserve them separately before making a numerical conversion decision. A reported 0,0 remains an explicit zero.
The extracts contain several geographic levels and years; they are neither independent district-only samples nor newer versions of Unit 03’s small 2024 state table. In the historical land-use file, Settlement includes transport; Recreation, Agriculture and Forest are also percentages of total area. Do not transfer the different 2024 ALKIS category definitions to this file.
Crop yields use decitonnes per hectare (dt/ha; 1 dt = 100 kg). The English source headings distinguish rye and winter mixed cereals and winter rapeseed; do not describe these as unrestricted rye or rapeseed categories.
Generated outputs and later use
| Output | Where it comes from | Reused in |
|---|---|---|
Output/lu_clean.rds |
Wide land-use table generated in the exercise, with source cells and unresolved metadata saved separately. | Units 05–06. |
Output/cp_clean.rds |
Wide crop table generated in your unmarked assignment, with original source tokens retained separately. | Unit 07. |
Prepared instructor RDS files are alternative checkpoints, not unexplained additional sources. The current checkpoints use the same English metadata rules and output schema as these lessons. Inspect their provenance and structure before using them; unresolved regional types remain explicitly unavailable.
If you need a prepared checkpoint, download lu_clean.rds or cp_clean.rds and save it in Output/. These are alternatives to generating your own output, not additional required inputs.
Comments?
You can leave comments below if you have questions about the text or code. Include the relevant line to make the question easier to answer.