Fit or evaluate a requested response–dataset grid

Low-level engine for a requested response–dataset–learner grid. Training mode fits and caches caret learners. Evaluation mode first reconstructs projected out-of-fold (OOF) metrics and weighted ensembles, then predicts matching unseen rows. The higher-level trainModel() and predictModel() wrappers call these two modes; direct use is intended for grid-level control.

Usage

trainRegressionModel(.path.data, .path.model, .path.metrics,
  .path.response, OVERRIDE = FALSE, TRAIN = TRUE, PARALLEL = TRUE,
  .min = -Inf, .max = Inf, .numberCV = 10, .responses = NULL,
  .methods = NULL, .datasets = NULL, .tuneLength = 10,
  .preProcess = NULL, .metric = "RMSE")

Arguments

.path.data

Path to an existing directory containing the split CSV files described in Split files.

.path.model

Model directory. Training mode requires it to exist and be writable for .Rds files and their .Rds.hash companions. Evaluation mode requires it to exist and be readable; it reads .Rds files and does not use the hashes. This engine does not create the directory.

.path.metrics

Metrics directory. Training mode does not touch this path, so it need not exist for a training-only call. Evaluation mode requires an existing writable directory, overwrites assigned cell-named CSVs, and reads every .csv already present; use a dedicated clean directory for one run.

.path.response

Prediction directory. Training mode does not touch this path, so it need not exist for a training-only call. Evaluation mode requires an existing writable directory and overwrites one cell-named CSV per cell that reaches prediction. This engine does not create it.

OVERRIDE

A non-missing logical scalar used only in training mode. FALSE reuses a model when both model and hash files exist and the sorted predictor-name hash matches; TRUE fits it again.

TRAIN

A non-missing logical scalar. TRUE selects fitting and returns after model creation. FALSE runs both OOF metric reconstruction and unseen-row prediction; it is not a prediction-only switch.

PARALLEL

A non-missing logical scalar. TRUE registers a PSOCK cluster capped at two logical cores; FALSE registers foreach sequential execution. The backend is left sequential after the call.

.min, .max

Numeric lower and upper response bounds, in the response's units. In evaluation mode each base OOF and unseen prediction is projected onto this interval. The function does not validate finiteness or .min <= .max. The bounds are unused in training mode.

.numberCV

A positive integer passed to the number argument of caret::trainControl() in training mode. The default is 10.

.responses

A character vector of response identifiers. Despite the NULL default, a non-empty vector is required to process cells.

.methods

A character vector of caret method identifiers. Despite the NULL default, at least one successfully fitted method is required for evaluation and ensemble output.

.datasets

A character vector of dataset identifiers. Despite the NULL default, a non-empty vector is required to process cells.

.tuneLength

A positive integer passed to caret::train() in training mode. It requests method-specific grid granularity and is not a count per tuning parameter. The default is 10.

.preProcess

NULL, a character vector, or another preprocessing specification accepted by caret::train() in training mode. It is unused in evaluation mode.

.metric

A single caret performance metric used to select tuning settings in training mode. The default is "RMSE"; it does not change the fixed metrics written by evaluation mode.

Value

In training mode, invisibly NULL. In evaluation mode, invisibly a data.table with the metric schema described in Files written and return value.

Choosing the API layer

This function is not a second supervised workflow. trainModel() discovers split files and calls TRAIN = TRUE with ssel's standard defaults. predictModel() discovers the same grid, calls TRAIN = FALSE, and then performs response-level selection and delivery assembly. semiSupervisedPipeline() also calls TRAIN = FALSE, PARALLEL = FALSE for its final serial re-emission. Use this engine directly only when those wrappers do not expose the required identifier grid, resampling, tuning, projection, or execution control.

Split files

The function expands .responses, .datasets, and .methods as a Cartesian product, in response–dataset–method loop order; it does not accept a sparse list of selected tuples. For each requested response and dataset, the training file is response_dataset_TRAIN.csv. It must contain predictor columns and a numeric column named y. Integer columns are promoted to double. A cell is skipped with a warning when the file is absent, y is absent, y has no non-missing value, or fewer than two distinct non-missing response values remain.

Evaluation mode additionally requires response_dataset_TEST.csv. It must contain the fitted predictors and exactly one column name absent from the training file; that column is the row key. Zero or multiple such names produce an error. Predictor and response values are not scaled or assigned units, so every file, bound, and interpretation must use consistent response units.

Training and reproducibility

Each learner is fitted by caret::train() with ordinary cross-validation, savePredictions = "final", the requested fold count, preprocessing, tuning granularity, and optimization metric. The function does not supply a common index or indexOut; separate learner calls therefore do not promise identical resampling folds. There is no seed argument. Call set.seed() before the function for reproducibility in a fixed R, caret, learner, and parallel environment.

A successful fit writes the caret train object to method_response_dataset.Rds and writes its eight-character predictor-name hash to method_response_dataset.Rds.hash. The hash uses only the sorted predictor names. With OVERRIDE = FALSE, an equal hash reuses the model even when data values or classes, response values, preprocessing, resampling, tuning, or metric settings changed. Training failures warn and skip the learner; successful cells continue.

OOF metrics and ensemble definitions

When a = .min <= .max = b, define interval projection by \(\Pi_{[a,b]}(z)=\min\{b,\max\{a,z\}\}\). For method \(m\), evaluation mode merges the saved final-tuning caret holdout predictions, orders them by caret row index, and applies this projection.

The first requested method that reaches this scoring step for a cell stores its ordered obs vector as the cell's shared reference response. Every later method is scored against that stored vector rather than its own merged obs. The function does not verify cross-method equality of observations, row indices, or vector lengths. Consequently, the supplied method order is material when fitted objects contain inconsistent saved OOF rows. In the equations below, \(y_i\) denotes this first-method reference vector and \(\hat y_{mi}\) the projected OOF prediction for method \(m\) and row \(i=1,\ldots,n\). The reported scores are $$\mathrm{RMSE}_m=\left[n^{-1}\sum_i(y_i-\hat y_{mi})^2\right]^{1/2},$$ $$\mathrm{MAE}_m=n^{-1}\sum_i\left|y_i-\hat y_{mi}\right|,$$ and squared sample correlation $$R_m^2=\mathrm{cor}(y,\hat y_m)^2.$$ RMSE and MAE have the response's units; r2 is dimensionless; n is the training-row count; and p is the predictor count.

For the methods with non-missing scores in a cell, the package defines two OOF weighted means. The R2 ensemble uses $$w_m^{(R2)}=R_m^2/\sum_j R_j^2,$$ and the RMSE ensemble uses $$w_m^{(RMSE)}=(1/\mathrm{RMSE}_m)/\sum_j(1/\mathrm{RMSE}_j).$$ Here \(j\) ranges over the retained methods. Each ensemble prediction is the corresponding weighted mean \(\sum_m w_m\hat y_m\). These are package weighting policies, not claims of optimal weighting. A zero, missing, or otherwise unusable weight sum can skip an ensemble or the cell. The ensemble rows are named ensemble.R2 and ensemble.RMSE.

Unseen-row prediction weight state

The OOF pass stores the exact named R2 and RMSE score lists under each response–dataset cell. Before predicting unseen rows for a cell, the function restores that cell's pair of lists. It does not reconstruct these numerical weights from metric CSVs, and a cell with no stored OOF score state has no unseen-row ensemble.

Each unseen cell uses the intersection of methods predicted successfully for that cell and the restored score-list names. A missing model or a method returning any missing prediction removes that method only from the current cell's local lists. For the R2 ensemble, the unseen pass substitutes equal weights if a retained R2 is missing or their sum is zero; otherwise it normalizes the retained R2 values. For the RMSE ensemble it first takes reciprocal retained RMSE values, then substitutes equal weights if a reciprocal is missing or their sum is zero; otherwise it normalizes those reciprocals. The formulas and fallbacks match the preceding policy, while score state and pruning remain response–dataset-cell local.

Base unseen predictions are projected before weighting.

Evaluation conditions

During OOF reconstruction, a missing training file, missing model, failed merge of saved holdout predictions, unsupported response, no valid method, invalid R2 or RMSE weight vector, or unusable ensemble prediction emits a warning and skips the affected method, ensemble, or cell according to where it occurs. An invalid R2 vector skips the remainder of that cell before its metrics file is written. A missing test file warns and skips unseen prediction; a missing companion training file is skipped silently. During unseen prediction, a missing model or missing predicted value warns and prunes that method from the current cell, while no common method names warns and skips the cell. A prediction error, NULL prediction, row-count mismatch, or invalid key schema stops the call.

Files written and return value

A cell-named metrics file is written only when the OOF cell reaches the post-ensemble write. The file .path.metrics/response_dataset.csv contains every current-call metric row accumulated to that point, not only rows for the cell in its name. Base rows accumulated for a cell that skipped before writing can therefore first appear in a later cell's file. Columns are method, set, dataset, response, rmse, mae, r2, n, and p. Existing assigned files are overwritten.

After the OOF pass, every CSV in .path.metrics is row-bound to determine which base methods are eligible for unseen-row prediction. Stale or unrelated compatible rows can therefore affect method discovery even though ensemble weights come only from the current call's cell-local score lists. A clean, dedicated metrics directory is required for an isolated evaluation.

Each predicted cell writes .path.response/response_dataset.csv with one row per unseen row and available base or ensemble method. Columns are the original key name, value, method, dataset, and response; value has the response's units. Training mode invisibly returns NULL. Evaluation mode invisibly returns the cumulative metrics data.table assembled during the current call. Progress is emitted as messages; skips and degraded cells emit warnings.

References

Kuhn, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28(5), 1–26. doi:10.18637/jss.v028.i05 . See also the caret documentation for caret::train() and caret::trainControl().

See also

trainModel() for split discovery and standard fitting defaults, predictModel() for summary and residual-offset products, and auditQuantiles() for the separate OOF residual-offset estimator.

Examples

root <- tempfile("ssel-trainRegression-")
data_dir <- file.path(root, "data")
model_dir <- file.path(root, "model")
metrics_dir <- file.path(root, "metrics")
response_dir <- file.path(root, "response")
for (path in c(data_dir, model_dir, metrics_dir, response_dir)) {
  dir.create(path, recursive = TRUE)
}

train <- data.frame(
  x = seq_len(18),
  z = rep(c(0, 1, 2), 6),
  y = seq_len(18) / 4 + rep(c(-1, 0.5, 1), 6)
)
test <- data.frame(
  x = c(19, 20),
  z = c(0, 1),
  SampleID = c("P1", "P2")
)
utils::write.csv(
  train,
  file.path(data_dir, "target_demo_TRAIN.csv"),
  row.names = FALSE
)
utils::write.csv(
  test,
  file.path(data_dir, "target_demo_TEST.csv"),
  row.names = FALSE
)

set.seed(2026)
suppressMessages(trainRegressionModel(
  .path.data = data_dir,
  .path.model = model_dir,
  .path.metrics = metrics_dir,
  .path.response = response_dir,
  TRAIN = TRUE,
  PARALLEL = FALSE,
  .numberCV = 3,
  .responses = "target",
  .methods = "lm",
  .datasets = "demo",
  .tuneLength = 1
))

metrics <- suppressMessages(trainRegressionModel(
  .path.data = data_dir,
  .path.model = model_dir,
  .path.metrics = metrics_dir,
  .path.response = response_dir,
  TRAIN = FALSE,
  PARALLEL = FALSE,
  .responses = "target",
  .methods = "lm",
  .datasets = "demo"
))
metrics[, .(method, response, dataset, rmse, r2)]
#>           method response dataset      rmse        r2
#>           <char>   <char>  <char>     <num>     <num>
#> 1:            lm   target    demo 0.2779726 0.9720389
#> 2:   ensemble.R2   target    demo 0.2779726 0.9720389
#> 3: ensemble.RMSE   target    demo 0.2779726 0.9720389
utils::read.csv(file.path(response_dir, "target_demo.csv"))
#>   SampleID    value        method dataset response
#> 1       P1 3.916667            lm    demo   target
#> 2       P2 5.166667            lm    demo   target
#> 3       P1 3.916667   ensemble.R2    demo   target
#> 4       P2 5.166667   ensemble.R2    demo   target
#> 5       P1 3.916667 ensemble.RMSE    demo   target
#> 6       P2 5.166667 ensemble.RMSE    demo   target
# Base and two package-defined ensemble rows are returned per SampleID.

unlink(root, recursive = TRUE)