Batch processing

Before you start: one-file import/export; for loops.
Setup: This is a preview. Skip and return after Unit 07 if loops are new; folder names in templates need your own small test inputs.
Your check: Explain the validated one-file operation that the loop repeats. Practise in the unit summary.

Optional preview: if loops are new to you, continue with the Unit 06 summary and return here after for loops in Unit 07. You can practise importing and exporting one file before automating several files.

This lesson uses the working-directory and path skills from Unit 05 and previews the for-loop lesson in Unit 07. If loops are new to you, read that lesson first or return here after Unit 07. In the example, file_path takes each value from file_paths in turn, and the statements inside the braces run once for each file.

Before running the code, replace "~/data/test" with your own working folder. Create its input and output subfolders, put a few small test files in input, and ensure that output is writable. Repeating the run overwrites output files with the same names.

Below is an example for batch processing of multiple files in a for-loop. Each file in the “input” directory is sequentially processed (in this example getting the file size) and the result is written into a corresponding file with the same file name in the “output” directory.

Note that the management of file names and file paths is crucial for batch processing. So know the path to your working directory, the paths to your input and output folders, the paths to your files, the names of your files and their extension.

Setting up your working environment

# set working directory with input and output folders
path_dir <- "~/data/test"
setwd(path_dir)

# get file names and path names
file_names <- list.files(file.path(path_dir, "input")) # only file names
file_paths <- list.files(file.path(path_dir, "input"), full.names = TRUE) # complete paths to files. Necessary to open files.

Batch process your files

# loop over all files. Note that files are identified by their file paths.
for (file_path in file_paths){
  # Some calculation for each input file
  size <- file.info(file_path)$size # Get file size for each file referred to in file_paths
  # Use the results and write out. One output file for one input file
  write(size, file = file.path(path_dir, "output", paste0(basename(file_path), ".out")))
  # Show status of each iteration
  cat("Processing:", file_path, "\t", "result of operation: ", size, "\n")
}

This is all it takes for also performing more complex operations using multiple input files.

Functions you should know

setwd()
file.path()
list.files()
basename()
dirname()
substr(basename(file_path), 1, nchar(basename(file_path))-4) # for removing the file extension, i.e. the last four characters.

library("tools")
file_ext(file_path) # get file extension
file_path_sans_ext(file_path) # get file path without extension
file_path_as_absolute(file_path) # get full file path

# The regex way for getting the pure file name without path and extension
sub(pattern = "(.*)\\..*$", replacement = "\\1", basename(file_path))

Updated: