Fit a range-ratio pseudo-label promotion loop

Refit the supervised ssel workflow while promoting selected prediction rows as ordinary ensemble.RMSE pseudo-labels. Selection uses a package-defined cross-method/dataset prediction range; acceptance uses a separate out-of-fold squared-correlation gauge. This is a pseudo-label policy, not tri-training, co-training regression, held-out validation, or calibrated predictive uncertainty.

Usage

semiSupervisedPipeline(
  .path.datasets,
  .path.work,
  key,
  responses,
  datasets,
  methods,
  seed,
  tau,
  maxRounds = 10L,
  epsRevert = 0.02,
  promotionCap = 0.8,
  boundsMin = -Inf,
  boundsMax = Inf,
  .path.iter = NULL,
  allResponses = responses
)

Arguments

.path.datasets: Directory scanned non-recursively for assembled CSV files whose basenames occur in datasets. It is ignored after a non-NULL .path.iter selects an iteration-specific dataset directory.
.path.work: Working directory. Every fit recursively removes and recreates its round child without rollback.
key: Character key column used for joins and pool membership. The final OOF gauge normally requires "SampleID" because that is the key column in the accepted residual schema. Missing key values are not rejected; their downstream behavior is described under input discovery.
responses: Character vector of target response columns. Exactly one is required when .path.iter is supplied. A standalone call can receive multiple responses, but its final serial evaluation has the failure mode described under Final serial re-emission.
datasets: Character dataset identifiers matched to CSV basenames. Matching preserves list.files() order, not this vector's order.
methods: Character vector of caret method identifiers. The initial check requires at least two entries; promotion additionally requires at least two distinct method names among available predictions.
seed: Value passed unchanged to every full modelPipeline() fit. It is not passed to the final serial re-emission.
tau: Numeric strict upper threshold for the dimensionless range ratio $D$. Equality does not promote. Length, missingness, and finiteness are not proactively validated.
maxRounds: Value passed to seq_len() as the maximum number of candidate rounds after the baseline fit. No terminal history row is added merely because this sequence is exhausted.
epsRevert: Numeric absolute tolerance in squared-correlation units. A candidate reverts only below bestR2 - epsRevert; the value is not validated as finite or nonnegative.
promotionCap: Numeric multiplier used in the key budget floor(promotionCap * length(U)). It is not restricted to $[0,1]$.
boundsMin, boundsMax: Numeric response bounds forwarded to supervised fitting and final evaluation. Let $a$ be boundsMin and $b$ be boundsMax. When $a\le b$, both ordinary and reconstructed OOF predictions use interval projection $\Pi_{[a,b]}(z)=\min\{b,\max\{a,z\}\}$. Order and finiteness are not checked. If $a>b$, the two executed nestings differ: ordinary evaluation computes $\min\{b,\max\{a,z\}\}=b$, whereas OOF reconstruction computes $\max\{a,\min\{b,z\}\}=a$. Neither is then an interval projection.
.path.iter: NULL, or a chainPipeline() iteration directory containing metrics_<integer>.csv and the corresponding datasets_<integer> directories. Selection is described in Input discovery and optional chain warm start.
allResponses: Character vector intended to name every response column. Before each fit, named non-target responses are removed. Missing names are ignored; an omitted non-target response remains and is numerically coerced as a predictor. The default is responses.

Value

Visibly, a list with:

history: A data.table with integer round, promoted, and cumulative; numeric oofR2; and character action, under the counting and action rules above.
promoted: NULL, or a data.table with the configured key, response, and numeric value, one row per accepted matching ensemble label row.
baselineR2: The numeric baseline gauge $g_0$.
finalR2: The numeric historical maximum gauge, not necessarily the final fitted-state gauge.
iterStar: NA_integer_ without .path.iter; otherwise the selected iteration integer.

Input discovery and optional chain warm start

With .path.iter = NULL, non-recursive .csv files in .path.datasets are filtered by basename membership in datasets; list.files() order is retained. With .path.iter supplied, the function reads every lexically ordered metrics_<integer>.csv, retains rows for method == "ensemble.RMSE" and the single target response, and selects the first row attaining the global maximum r2. Its integer becomes iterStar, and the input directory becomes datasets_<iterStar>.

This selects one augmented dataset snapshot. It does not consume a stitched model, stitch response-specific features, or rerun a chain sweep. The selected directory must exist; in particular, no fallback maps iteration zero to the original dataset directory. With no retained metric row or only missing r2, which.max() returns no position, no CSV is discovered, and the later non-empty-file assertion fails.

The first retained CSV alone defines the labelled-key vector $L=\{u:\mathrm{set}(u)=\text{"train"}\}$ and original prediction-key vector $U=\{u:\mathrm{set}(u)=\text{"predict"}\}$. Row order and duplicate keys are retained. Missing key values are also retained without a missingness check. They enter membership filters, update joins, length(U), unique-key accounting, and the final completeness count under ordinary R/data.table semantics. Round CSV writing and subsequent type inference can materialize a missing character key as an empty string while preserving a missing numeric key, so no cross-file missing-key normalization is guaranteed. Other files are not checked for the same key pools, and missing set or key columns fail through data.table evaluation. The base budget size $|U|$ is length(U), including duplicates and missing values.

Before every fit, each retained CSV is copied after removing only names in setdiff(allResponses, responses). Every remaining column except key and set, including targets and predictors, is replaced by as.numeric(). Text and factor-like input can therefore warn and become missing. Missing features, targets, or role columns are left to the composed supervised helpers and data.table operations to reject or omit under their contracts.

Destructive round fit

For each baseline, candidate, or restoration fit, the function recursively deletes .path.work/round, recreates its datasets directory, and writes all selected CSV copies. Accepted pseudo-label rows update targets by (key, response), and every matching key is changed to set = "train". Duplicate label keys and labels from multiple datasets are not collapsed; update-join order can make the last matching value material.

The rewritten tables are passed to modelPipeline() in its default "full" mode with ensemble = "ensemble.RMSE" and the same seed on every fit. Round products therefore follow that helper's split, fit, OOF, prediction, summary, aggregation, bounded-PSOCK, and deletion contracts. Ordinary prediction ensemble.RMSE rows used below are distinct from the reconstructed OOF inverse-RMSE estimator underlying the acceptance gauge.

Range-ratio promotion score

For original prediction key $u$, response $r$, method $m$, dataset $d$, and round $s$, let $f_{mdr}^{(s)}(u)$ be every available projected ordinary base-method prediction. Pool all method and dataset rows into $$\mathcal A_{ur}^{(s)}=\{f_{mdr}^{(s)}(u)\},\qquad S_{ur}^{(s)}=\max\mathcal A_{ur}^{(s)}-\min\mathcal A_{ur}^{(s)}.$$ A pair is retained only when at least two distinct method names occur; two datasets alone do not suffice.

Let $\bar f_{ur,\mathrm{last}}^{(s)}(u)$ be the last matching ordinary ensemble.RMSE value in response-file concatenation and update-join order. ssel defines $$D_{ur}^{(s)} =\frac{S_{ur}^{(s)}} {\max\{1,|\bar f_{ur,\mathrm{last}}^{(s)}(u)|\}}.$$ This dimensionless one-dimensional range ratio is package policy, not a variance, probability, interval, or calibrated uncertainty. The fixed denominator floor makes it non-invariant to response rescaling and translation. Pooling across datasets and choosing the last ensemble match also make row composition and order material.

Neither maximum nor minimum removes missing values. A missing base value makes the score missing; it cannot satisfy D < tau. Identical finite base predictions give $D=0$, which promotes only when tau > 0. A score equal to tau does not promote. Equal scores retain their preceding group order. Previously accepted (key, response) pairs are removed before this strict gate.

Key budget and pseudo-label rows

Let $c$ be promotionCap, let $P$ be the accepted pseudo-label table, and let $K(P)$ be its unique keys. The remaining budget is $$B_s=\lfloor c|U|\rfloor-|K(P)|.$$ New keys are encountered in ascending-score, first-row order. When they exceed $B_s$, only the first $B_s$ new keys are retained, together with all their eligible response pairs and new response pairs on already promoted keys. A nonpositive budget or absence of a new key produces saturation-stop. Because $|U|$ counts rows while spending counts unique keys, duplicate prediction keys can allow promotionCap < 1 to admit every unique key. Invalid cap values retain base arithmetic and condition behavior.

Pseudo-labels are all response-file rows matching method == "ensemble.RMSE" and the candidate pairs; no median or cross-dataset reduction is applied. Multiple datasets can therefore contribute multiple label rows for one pair. history$promoted counts candidate pairs, whereas accepted history$cumulative and the returned promoted table count label rows.

OOF gauge and accept/revert rule

Let $H_r^{(s)}$ contain all rows of residuals_oof.csv for response $r$ whose configured key occurs in the first-file labelled vector $L$. Rows from all datasets remain pooled. For each response, compute squared sample correlation and then define $$g_s=\operatorname{mean}_r \{\operatorname{cor}(y,\hat y_{\mathrm{OOF}})^2\},$$ removing response-level missing values and weighting each surviving response equally rather than by row count. The gauge is dimensionless. It is not held-out risk, a fully nested OOF estimate, or a no-degradation guarantee. Duplicate pooled rows retain their multiplicity. Missing values or constant vectors can make a response correlation missing; if every response is missing, $g_s$ is NaN and a later logical comparison can error. A key other than "SampleID" generally fails when the accepted residual product has no such column.

Baseline round zero sets historical bestR2 to $g_0$. Candidate round $s$ reverts only when $$g_s < \mathrm{bestR2}-\varepsilon,$$ where $\varepsilon$ is epsRevert. Equality is accepted. An accepted round may be worse than the historical maximum by up to this absolute tolerance; bestR2 becomes max(bestR2, g_s). Thus the returned finalR2 is a historical maximum and need not equal the final fitted state's gauge. A rejected candidate writes a REVERT-stop history row, then the accepted pool is materially refitted, deleting the rejected round state. That restoration gauge is not recomputed.

Stops and history

history begins with action "baseline", zero counts, and $g_0$. Accepted rounds use "accept", the candidate-pair count, accepted label-row count, and candidate gauge. If no pair passes, "empty-stop" records zero promoted rows, the accepted label-row count, and historical bestR2. A nonpositive/exhausted budget or no new key similarly records "saturation-stop". A rejection uses uppercase "REVERT-stop", records the attempted pair count, the preceding accepted label-row count, and the rejected gauge. Exhausting maxRounds adds no separate stop row.

No-promotion runs return promoted = NULL. Depending on cap and duplicate keys, every unique prediction key can be promoted. Subsequent scoring uses only rows still emitted as predictions by the latest accepted full fit.

Final serial re-emission

After the loop and any restoration, the function rebuilds one TEST file for every declared dataset/response cell. It uses all rows from each original selected CSV whose key belongs to the first-file $U$, but takes feature names from the final round dataset. Missing declared datasets or required feature columns fail through file/data.table operations.

In a standalone multi-response call, full promotion rounds can finish before this final step fails. For target $r$, final TEST construction removes only set and $r$; other target-response columns remain. The supervised TRAIN split excludes those response columns, so trainRegressionModel() can find more than its required single TEST-only column: the key plus the other responses. Its exactly-one-key invariant then errors before successful final re-emission. The function does not proactively restrict standalone calls to one response.

trainRegressionModel() then runs with TRAIN = FALSE and PARALLEL = FALSE, overwriting cell metric and response CSVs. For each cell, it asserts that the number of unique ensemble.RMSE keys equals length(U). It does not compare key sets. Duplicate keys in $U$ make the assertion fail even when all unique keys were emitted, while a different key set of the same cardinality could pass.

This serial pass does not rerun auditQuantiles(), predictModel(), auditOverfit(), or aggregateResponses(). Consequently, the cell response files cover the original prediction pool, but summary and export products remain from the preceding full round and can still represent only its then-unpromoted remainder.

Conditions, RNG, and side effects

Apart from the method-count and optional single-response/file assertions, most validation is delegated to base R, data.table, caret, and the composed ssel helpers. Paths, scalar lengths, response/dataset agreement, bounds, thresholds, caps, and tolerances are not proactively normalized. Failures can leave partial round files; there is no transaction or rollback.

Every full round receives the same seed, so modelPipeline() resets RNG before its sampled method/dataset order and downstream fitting. Reproducible results remain conditional on the R, RNG, caret, learner, and parallel environment. Rounds are sequential, each full round uses the supervised bounded-PSOCK contracts, and final re-emission is explicitly serial. The helper emits the conditions of its composed functions and returns no fitted model or prediction object.

Examples

# \donttest{
root <- tempfile("ssel-semi-")
datasets_dir <- file.path(root, "datasets")
dir.create(datasets_dir, recursive = TRUE)

n_train <- 30L
assembled <- data.frame(
  SampleID = c(paste0("S", seq_len(n_train)), "P1", "P2"),
  set = c(rep("train", n_train), "predict", "predict"),
  x = seq_len(n_train + 2L),
  z = rep(c(0, 1), length.out = n_train + 2L),
  target = c(
    1.5 * seq_len(n_train) + rep(c(-0.4, 0.2, 0.6), 10),
    NA, NA
  )
)
utils::write.csv(
  assembled, file.path(datasets_dir, "demo.csv"),
  row.names = FALSE, na = ""
)

old_quiet <- getOption("aR.quiet")
options(aR.quiet = TRUE)
result <- semiSupervisedPipeline(
  .path.datasets = datasets_dir,
  .path.work = file.path(root, "work"),
  key = "SampleID",
  responses = "target",
  allResponses = "target",
  datasets = "demo",
  methods = c("lm", "glm"),
  seed = 2026,
  tau = 0.1,
  maxRounds = 1,
  epsRevert = Inf,
  promotionCap = 0.5
)
options(aR.quiet = old_quiet)

result$history
#>    round promoted cumulative     oofR2   action
#>    <int>    <int>      <int>     <num>   <char>
#> 1:     0        0          0 0.9987861 baseline
#> 2:     1        1          1 0.9986357   accept
result$promoted
#> Index: <response>
#>    SampleID response    value
#>      <char>   <char>    <num>
#> 1:       P1   target 46.70476
c(baseline = result$baselineR2, historical_best = result$finalR2)
#>        baseline historical_best 
#>       0.9987861       0.9987861 
# The one-round policy promotes at most one of the two original keys.
# iterStar is NA because no chain iteration directory was supplied.

unlink(root, recursive = TRUE)
# }