Audit package-defined in-sample optimism

Compare projected in-sample model error with the resampling summaries stored in fitted caret models, then classify a package-defined relative-optimism ratio. Use this table as a diagnostic for unusually large training-versus- resampling gaps. The ratio and default thresholds are ssel policy; they are not a hypothesis test, a universal overfitting measure, or published cutoffs.

Usage

auditOverfit(.path.model, .path.train, .path.summary,
  .min = -Inf, .max = Inf, REL_OPTIMISM_CATASTROPHIC = 0.95,
  REL_OPTIMISM_HIGH = 0.60,
  .response_pattern = NULL, verbose = TRUE)

Arguments

.path.model: Path to an existing directory containing model files named method_response_dataset.Rds. The method identifier must be a non-empty string without an underscore. Matching objects must support predict(model, newdata = TRAIN) and contain a non-empty resample table with RMSE and Rsquared columns.
.path.train: Path to an existing directory containing response_dataset_TRAIN.csv. Each eligible file must contain the fitted predictors and a column named y coercible to numeric.
.path.summary: Path to the output directory. It is created recursively when absent; overfit.csv is overwritten.
.min, .max: Numeric lower and upper response bounds, in response units. Only in-sample predictions are projected with pmax(.min, pmin(.max, z)). Bound order and finiteness are not validated.
REL_OPTIMISM_CATASTROPHIC: Numeric package-policy threshold for the "catastrophic" label. It is tested first and defaults to 0.95.
REL_OPTIMISM_HIGH: Numeric package-policy threshold for the "high_optimism" label. It is tested second and defaults to 0.60. Threshold length, finiteness, range, and ordering are not validated.
.response_pattern: A non-empty character regular expression inserted into the model-filename parser. Use noncapturing groups when grouping is needed because additional capturing groups change parser field positions. The default NULL deliberately fails.
verbose: A non-missing logical value. If true, invoke debug reporting for the output path and flag counts. Messages are emitted, and their arguments evaluated, only when options(aR.verbose = TRUE) and options(aR.quiet = FALSE) are both in effect.

Value

Invisibly, a one-element list named overfit. Its value is the table with the schema above. When no model survives, this is an empty zero-column table unless fully enabled debug evaluation errors first, as described in Files, skips, and side effects.

Diagnostic definition

For an eligible model, let $y_i$ be numeric TRAIN$y and let $\hat y_i^{in}$ be the generic in-sample prediction projected onto $[a,b]$. The helper reports $$\mathrm{RMSE}_{in} =\left[n^{-1}\sum_i(y_i-\hat y_i^{in})^2\right]^{1/2}$$ and, when the sample variance of $y$ is nonzero, $$R^2_{in}=\mathrm{cor}(y,\hat y^{in})^2.$$ It separately reads each row of model$resample, reports the mean, sample standard deviation, minimum, and maximum of its RMSE column, and the mean and sample standard deviation of Rsquared, using na.rm = TRUE. The output field k_folds is exactly the number of rows in model$resample; its name does not establish that those rows are unique folds rather than another caret resampling design.

With $\overline{\mathrm{RMSE}}_{CV}$ equal to that resample mean, ssel defines $$\mathrm{rel\_optimism} =\frac{\overline{\mathrm{RMSE}}_{CV}-\mathrm{RMSE}_{in}} {\max(\overline{\mathrm{RMSE}}_{CV},10^{-9})}.$$ The unscaled numerator is optimism_rmse. With finite response bounds, this compares projected in-sample error with caret's stored, unprojected resample RMSE summary; it is not a like-for-like reprojection of both terms.

Non-finite relative optimism is labeled "unclassified". Otherwise, the catastrophic threshold is tested first, followed by the high threshold; remaining rows are "reliable". Negative values are possible. These labels express only the configured package policy.

Files, skips, and side effects

For every retained model, overfit.csv contains method, response, dataset, n_train, k_folds, rmse_in, rmse_cv_mean, rmse_cv_sd, rmse_cv_min, rmse_cv_max, optimism_rmse, rel_optimism, r2_in, r2_cv_mean, r2_cv_sd, and flag. RMSE and optimism fields have response units; relative optimism and R-squared fields are dimensionless.

Nonmatching filenames, absent TRAIN files, unreadable models, failed predictions, prediction-length mismatches, and missing or empty resample tables are skipped silently. Numeric coercion and summary edge cases retain base R behavior, including missing or infinite fields and warnings. If no model survives, data.table::fwrite() warns and creates a zero-byte overfit.csv. With call-level reporting disabled, or with either global debug gate closed, the function then returns a zero-row, zero-column table. When verbose = TRUE, aR.verbose = TRUE, and aR.quiet = FALSE, flag-count evaluation references the absent flag column and errors after the file is written, so no list is returned. Malformed-column, data.table, and file errors not caught by the helper propagate.

The function reads models and training data, writes only overfit.csv, does not fit models, and neither sets a seed nor draws random values directly. Model-specific prediction methods govern any of their own random-number effects.

References

Kuhn, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28(5), 1–26. doi:10.18637/jss.v028.i05 .

Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning, second edition, Chapter 7. Springer. doi:10.1007/978-0-387-84858-7 .

Examples

root <- tempfile("ssel-overfit-")
model_dir <- file.path(root, "model")
train_dir <- file.path(root, "train")
summary_dir <- file.path(root, "summary")
dir.create(model_dir, recursive = TRUE)
dir.create(train_dir)

train <- data.frame(
  x = seq_len(8),
  y = c(1, 2, 2, 4, 5, 5, 7, 8)
)
fit <- stats::lm(y ~ x, data = train)
fit$resample <- data.frame(
  RMSE = c(0.2, 0.3),
  Rsquared = c(0.8, 0.9)
)
saveRDS(fit, file.path(model_dir, "lm_target_demo.Rds"))
data.table::fwrite(train, file.path(train_dir, "target_demo_TRAIN.csv"))

diagnostic <- auditOverfit(
  .path.model = model_dir,
  .path.train = train_dir,
  .path.summary = summary_dir,
  .response_pattern = "target",
  verbose = FALSE
)
diagnostic$overfit[, .(
  method, rmse_in, rmse_cv_mean, rel_optimism, flag
)]
#>    method   rmse_in rmse_cv_mean rel_optimism     flag
#>    <char>     <num>        <num>        <num>   <char>
#> 1:     lm 0.4330127         0.25   -0.7320508 reliable
# The label follows ssel's configured ratio thresholds.

unlink(root, recursive = TRUE)