Ensemble estimators and residual offsets
ensemble-theory.RmdThis article explains the estimators used by the supervised helper
family. It is important to keep them separate:
trainRegressionModel() constructs cell-level OOF and
unseen-row ensembles, oofEnsemble() independently
reconstructs training-row residuals, auditQuantiles()
summarizes those residuals as signed offsets, and the two delivery
helpers apply different selection rules. modelPipeline()
orchestrates these helpers; it is not an additional statistical
estimator.
Choose the right API layer
The two training helpers are active at different levels; they are not duplicate implementations.
| Task | Public helper |
|---|---|
| run the complete supervised workflow from assembled dataset CSVs | modelPipeline() |
| discover all TRAIN splits and fit them with ssel’s standard defaults | trainModel() |
| fit an explicit cell grid with custom resampling or tuning controls | trainRegressionModel(TRAIN = TRUE) |
| evaluate fitted cells, select deliveries, and assemble user-facing products | predictModel() |
| obtain raw cell metrics and long predictions without delivery selection | trainRegressionModel(TRAIN = FALSE) |
trainModel() calls the engine in training mode and
returns after model files are written. The metrics and response paths in
its signature are prepared for the later evaluation phase; the wrapper
does not populate them. Evaluation mode is likewise more than
predict(): it first rebuilds OOF scores and ensembles, then
predicts unseen rows. Most applications should therefore enter through
modelPipeline() or the phase wrappers and use the cell
engine directly only for controls those wrappers do not expose.
Notation and cells
Let r identify a response, d a dataset, m a learner, and i a training row. A cell is one response–dataset pair (r,d). For a cell, y_i is the observed response and \hat y_{mi}^{OOF} is learner m’s out-of-fold (OOF) prediction. RMSE and response predictions have the response’s units; squared sample correlation R^2 and normalized weights are dimensionless.
The principal API mapping is:
| Mathematical role | Public helper |
|---|---|
| discover the standard fitting grid | trainModel() |
| fit explicit cells and retain selected-tuning OOF rows | trainRegressionModel(TRAIN = TRUE) |
| construct cell-level OOF and unseen-row ensembles | trainRegressionModel() |
| independently reconstruct training-row OOF residuals | oofEnsemble() |
| summarize residuals as signed offsets | auditQuantiles() |
| select a response-level cell | predictModel() |
| select final row-level RMSE candidates | aggregateResponses() |
| compute the package optimism diagnostic | auditOverfit() |
Fitting and OOF scores
Learners are fitted separately with caret::train() (Kuhn 2008); current
resampling, tuning, and reproducibility controls are documented in the
caret
training and tuning guide. Through trainModel(), ssel
requests five-fold cross-validation, RMSE tuning, selected-tuning
holdout predictions, and tuneLength = 10. A direct
trainRegressionModel() call instead uses its
.numberCV, .tuneLength, and
.metric arguments. The tuning length requests
method-specific grid granularity; it is not a number of points per
tuning parameter.
Separate learner calls do not receive a common explicit resampling index. Their saved OOF rows therefore support learner-specific score estimation, but ssel does not claim identical folds across learners.
Within one cell, the current evaluation engine uses the first requested learner that reaches scoring to establish the shared ordered observation vector. Later learners are scored against that vector; their own saved observations and row indices are not checked for equality. The equations below therefore describe the intended consistent-fit case. If fitted objects carry inconsistent OOF rows, learner order becomes material and the cell should be treated as invalid input rather than as a comparable ensemble.
For each retained learner in cell (r,d), evaluation reports
\operatorname{RMSE}_{mrd} =\left\{\frac{1}{n}\sum_i(y_i-\hat y_{mi}^{OOF})^2\right\}^{1/2}.
R^2_{mrd}=\operatorname{cor}(y,\hat y_m^{OOF})^2.
The package defines two cell-local weighting policies:
w^{R2}_{mrd}=\frac{R^2_{mrd}}{\sum_jR^2_{jrd}}, \qquad w^{RMSE}_{mrd} =\frac{\operatorname{RMSE}_{mrd}^{-1}} {\sum_j\operatorname{RMSE}_{jrd}^{-1}}.
The corresponding OOF ensemble is
\hat y_{ird}^{g,OOF}=\sum_m w^g_{mrd}\hat y_{mi}^{OOF}, \qquad g\in\{R2,RMSE\}.
These normalized scores are package policies, not estimates of theoretically optimal ensemble weights and not guarantees that an ensemble improves every learner.
Unseen-row ensembles are cell-local
During OOF evaluation, trainRegressionModel() stores the
exact named R^2 and RMSE lists for
every cell. Before predicting unseen rows in (r,d), it restores that cell’s lists and
intersects their names with the learners that produced valid predictions
for that same cell. A missing model or a missing prediction removes a
learner only from the current cell.
The unseen-row pass uses the same normalization formulas. Its documented fallback substitutes equal weights when the retained R^2 values, or the reciprocal RMSE values, contain a missing value or sum to zero. A cell without stored OOF score state has no unseen-row ensemble. Thus one cell neither borrows scores from nor prunes methods for another cell.
Independent OOF residual reconstruction
oofEnsemble() does not reuse the preceding score state.
It recomputes a method-level RMSE from each fitted object’s saved,
projected OOF predictions and defines independent inverse-RMSE weights
\tilde w_{mrd}. Its cast key is jointly
h=(i,o), where i is the caret row index and o the averaged saved observation. With A_h denoting learners available at that
key,
\hat y_h^{OOF}=\Pi_{[a,b]}\!\left( \frac{\sum_{m\in A_h}\tilde w_{mrd}\tilde y_{mh}} {\sum_{m\in A_h}\tilde w_{mrd}} \right).
e_h=o-\hat y_h^{OOF}.
Available learners are renormalized row by row. If saved observations differ between learners for one row index, they form different joint keys; repeated row indices and sample identifiers can therefore be legitimate output rows.
For valid bounds a\le b, the interval projection used above is
\Pi_{[a,b]}(z)=\min\{b,\max\{a,z\}\}.
This is the ordinary projection onto a closed interval (Parikh and Boyd 2014). Bounds are not proactively reordered, so this name applies only when a\le b.
Signed residual offsets
auditQuantiles() pools reconstructed residual rows
separately within each cell. For p\in\{0.50,0.90,0.95\}, sort the n residuals and define
H=(n-1)p+1.
j=\lfloor H\rfloor,\qquad \gamma=H-j.
R’s type-7 sample quantile is then
q_{p,rd}=(1-\gamma)e_{(j)}+\gamma e_{(j+1)},
with the endpoint convention described by Hyndman and Fan (1996). The values are signed, cell-pooled, and expressed in response units. Delivery helpers form
Q_{p,ird}=\Pi_{[a,b]}\{\hat y_{ird}+q_{p,rd}\}.
These are descriptive point-plus-residual-offset estimates. They are not conditional predictive quantiles or intervals with a calibrated coverage guarantee. Coverage-guaranteed procedures such as jackknife+ require fold- or leave-one-out-specific predictions at each test point, which this estimator does not construct (Barber et al. 2021).
Two distinct selection rules
predictModel() first chooses one eligible cell per
response. For ensemble.RMSE it takes the first minimum-RMSE
row; for ensemble.R2 it takes the first maximum R^2 row:
d_r^{RMSE}=\operatorname*{first\,argmin}_{d}\operatorname{RMSE}_{rd}.
d_r^{R2}=\operatorname*{first\,argmax}_{d}R^2_{rd}.
aggregateResponses() applies a later and different rule.
Its candidates are only unseen ensemble.RMSE points and
reconstructed training OOF points. For each sample i and response r, it selects
d^*_{ir}=\operatorname*{first\,argmax}_{d}R^2_{rd}.
The selected candidate supplies all three signed offsets. Because the
second rule is grouped by sample and excludes ensemble.R2
candidates, the two selectors must not be interpreted as one global
best-dataset operation.
Optimism diagnostic
auditOverfit() compares projected in-sample RMSE with
the arithmetic mean of the fitted caret object’s stored resample RMSE
values:
\operatorname{rel\_optimism} =\frac{\overline{\operatorname{RMSE}}_{CV}-\operatorname{RMSE}_{in}} {\max\{\overline{\operatorname{RMSE}}_{CV},10^{-9}\}}.
Its labels and thresholds are ssel policy. The value is a diagnostic, not a hypothesis test, a universal measure of overfitting, or an input to final aggregation.
Executable residual-offset example
The example below creates one fitted-object fixture with two equal signed OOF residuals. All three type-7 offsets must therefore equal one.
root <- tempfile("ssel-offsets-")
model_dir <- file.path(root, "model")
train_dir <- file.path(root, "train")
summary_dir <- file.path(root, "summary")
dir.create(model_dir, recursive = TRUE)
dir.create(train_dir)
data.table::fwrite(
data.table::data.table(y = c(1, 3)),
file.path(train_dir, "target_demo_TRAIN.csv")
)
data.table::fwrite(
data.table::data.table(SampleID = c("S1", "S2")),
file.path(train_dir, "target_demo_TRAIN_ids.csv")
)
fit <- list(
pred = data.frame(
tune = 1L, pred = c(0, 2), obs = c(1, 3), rowIndex = 1:2
),
bestTune = data.frame(tune = 1L)
)
saveRDS(fit, file.path(model_dir, "lm_target_demo.Rds"))
offsets <- auditQuantiles(
.path.model = model_dir,
.path.train = train_dir,
.path.summary = summary_dir,
.response_pattern = "target",
verbose = FALSE
)
offsets$quantiles
#> response dataset n q_50 q_90 q_95
#> <char> <char> <int> <num> <num> <num>
#> 1: target demo 2 1 1 1
stopifnot(all(offsets$quantiles[, c("q_50", "q_90", "q_95")] == 1))
unlink(root, recursive = TRUE)The complete argument, file, projection, skip, and side-effect contracts live in the linked function reference pages. The iterative multi-response and pseudo-label policies are described in their own articles.