Fit an iterative multi-response companion chain

Refit the supervised ssel workflow while using other responses' preceding delivery values as predictors. The helper supports simultaneous Jacobi updates and an asymmetric Gauss–Seidel update in which training companions can refresh within a sweep but prediction-row companions remain frozen. It is filesystem orchestration around modelPipeline(), not a standard regressor-chain estimator or a general fixed-point solver.

Usage

chainPipeline(
  .path.datasets,
  .path.train,
  .path.model,
  .path.metrics,
  .path.response,
  .path.summary,
  .path.export,
  .path.iter,
  key,
  responses,
  datasets,
  colsId = character(0),
  features = NULL,
  setCol = "set",
  methods = c("ranger", "earth", "gbm"),
  seed = NULL,
  ensemble = "ensemble.RMSE",
  boundsMin = -Inf,
  boundsMax = Inf,
  maxIter,
  tol,
  patience,
  sweep = "jacobi",
  suffix = ".o",
  chainGate = "fixed",
  chainThresh = 0,
  chainStable = 1L,
  resume = FALSE
)

Arguments

.path.datasets: Existing directory containing the original assembled CSV files. Every non-recursive CSV is copied into augmented iteration directories; modelPipeline() applies its all-CSV split contract.
.path.train, .path.model, .path.metrics, .path.response, .path.summary, .path.export: Mutable supervised-workflow directories. They are repeatedly overwritten or removed under the rules in Files and final stitch.
.path.iter: Mutable chain-state directory. A fresh run recursively removes it. It then contains augmented datasets, iteration response and metric copies, five-file summary snapshots, and convergence.csv.
key: Non-empty character key passed to joins and modelPipeline(). Final response products still inherit aggregateResponses()'s independent SampleID requirement.
responses: Non-empty character vector of response column names. The names also determine sorted Gauss–Seidel order, metric columns, and suffixed companion names. Supervised split-file discovery interprets the first underscore field as the response, so compatible response identifiers cannot contain underscores.
datasets: Non-empty character vector passed to modelPipeline() for final feature-file precedence. It does not restrict all-CSV split/model discovery. Upstream split parsers likewise require dataset identifiers without underscores.
colsId: Character vector of identifier columns excluded from inferred predictors.
features: NULL, or exact base predictor names. With exact names, available response companions are appended automatically by buildDataset(). The shadow predictor is not appended automatically and must be named explicitly when chainGate = "shadow"; NULL inference includes it.
setCol: Non-empty character name of the assembled row-role column.
methods: Character vector of caret method identifiers passed to every supervised fit. The default is c("ranger", "earth", "gbm").
seed: NULL, or a value passed to iteration-zero modelPipeline(). Iteration $t$ receives seed + t. The function does not validate arithmetic compatibility.
ensemble: Forwarded to modelPipeline() for its first delivery selector. Chain metric comparison and final iteration selection instead use ensemble.RMSE rows under their separate contracts. Active-set extraction does not use ensemble rows: it reads the individual fitted model files in .path.model. Final aggregation consumes the selected snapshots rather than reapplying this argument.
boundsMin, boundsMax: Response bounds forwarded to every supervised call. They apply to response and companion deliveries as documented by the called helpers; order and finiteness are not checked here.
maxIter: Numeric number of additional chain iterations to attempt. On resume it is additional to the last completed iteration, not an absolute total cap. Positivity, integrality, finiteness, and length are not proactively checked.
tol: Numeric strict change-score threshold. The chain stops when the current $\delta_t$ is strictly less than this value. See Change and stopping policies.
patience: Numeric strike threshold used independently for consecutive non-improvement and increasing change. It is not validated as a positive integer.
sweep: One of "jacobi" or "gauss-seidel".
suffix: Character suffix appended to companion columns and passed to importance extraction. It is not validated as non-empty or collision-free.
chainGate: One of "fixed" or "shadow". The fixed policy uses model-specific importance and an inclusive mean threshold. The shadow policy uses ssel's strict-majority single-shadow rule; it is not Boruta.
chainThresh: Numeric normalized-importance percentage threshold used only by the fixed gate. It is forwarded without validation.
chainStable: Value coerced with as.integer() to the raw active-set intersection window. Missing, nonpositive, or otherwise unsuitable values are not proactively rejected.
resume: Logical flag. Resume occurs only when both convergence.csv and Y.csv exist; otherwise a fresh run starts. The value is tested by ordinary logical conditions rather than validated as a scalar.

Value

Invisibly, NULL. Results are the mutable supervised products, iteration state, convergence history, and stitched response/summary files described above.

Companion values and row classes

Let $Y_k^{(t)}(i)$ denote response $k$'s p = 0.5 value in iteration $t$'s wide Y.csv for training key $i$. Let $Z_k^{(t)}(u)$ denote its p = 0.5 value in Yo<t>.csv for prediction key $u$. These are median signed-residual-offset delivery values after interval projection, in response $k$'s units; they are not unadjusted point predictions.

Before a fit, the helper casts the selected Yo responses wide and joins them to Y by key. For a common key and response, Y takes precedence and Yo fills only a missing Y value. Companion columns are renamed with suffix and joined to every original assembled CSV. buildDataset() removes target $j$'s own suffixed column from that target's predictors. Its complete-case rules can remove rows with missing companions.

Jacobi and Gauss–Seidel recurrences

Let $A^{(t-1)}$ be the active companion-response set entering sweep $t$. In a Jacobi sweep, one augmented dataset snapshot is built before any target is refitted. Every target $j$ therefore uses $$\{Y_k^{(t-1)}(i):k\in A^{(t-1)},k\ne j\}$$ for a training row and $$\{Z_k^{(t-1)}(u):k\in A^{(t-1)},k\ne j\}$$ for a prediction row.

Gauss–Seidel targets are processed in sorted response-name order. The augmented CSVs are rebuilt before every target. Earlier targets' new training-row values can therefore enter later targets through the evolving Y.csv; later targets retain their preceding-sweep values. In contrast, the PrevYo path is fixed for the entire sweep, so every target uses only $Z_k^{(t-1)}(u)$ on prediction rows. Thus the immediate-update analogy applies only to training companions. No within-sweep prediction-row companion is consumed.

These recurrences are ssel package definitions related to response-as-input multi-target methods. They do not inherit published regressor-chain ordering, out-of-fold, convergence, or optimality claims.

Active companion policy

This policy is independent of the ensemble.RMSE metric rows used for change history and final stitching. After each completed fit, "fixed" calls extractChainImportance() with useModel = TRUE, then applies activeByImportance() at chainThresh. The "shadow" policy calls the extractor with useModel = FALSE and applies activeByShadow(). Both routes scan individual .Rds model files in .path.model; neither extracts importance from an ensemble row. The shadow route is a strict-majority comparison to one injected Gaussian predictor and does not implement Boruta's iterative random probes or statistical tests.

An empty raw gate result is first replaced by all declared responses. If enough history exists, the current raw set is then intersected with the preceding chainStable - 1 returned active sets. These stored sets can already reflect earlier intersections. A new intersection can still be empty and is recorded with active-set size zero. If another iteration is attempted from that empty set, augmentation reaches data.table column renaming with no companion columns and errors before the supervised fit. The pipeline invokes computeActiveByImportance() with its stability disabled and applies this common history rule itself for both gate types.

Change and stopping policies

For metrics, only ensemble.RMSE rows are considered and the first minimum-RMSE dataset row is retained for each response. Let $R_{r,t}^2$ be that row's squared sample correlation. An iteration is improving only when its unweighted mean across responses is strictly greater than the mean at the historical accepted metric baseline. That baseline is replaced only on strict improvement.

With response columns of Y.csv compared by current row and column position, ssel defines $$\delta_t=\max_{i,r} \frac{|Y_r^{(t)}(i)-Y_r^{(t-1)}(i)|} {\max\{1,|Y_r^{(t-1)}(i)|\}},$$ removing missing cells in the final maximum. Compatible table order is therefore a precondition; no key join aligns the two matrices. Although numerically dimensionless, $\delta_t$ is not invariant to response-unit rescaling or translation because of the fixed floor 1. It is a package stopping heuristic, not proof that an equation solver converged.

Non-improvement strikes reset after an improving iteration. Divergence strikes increment only when $\delta_t$ strictly exceeds the preceding value and reset otherwise. After all current artifacts are written, stop tests run in this order: current change strictly below tol, divergence strikes at patience, then non-improvement strikes at patience. If every cellwise change is missing, the final max(..., na.rm = TRUE) produces -Inf; that value is written and satisfies Delta < tol for a finite tol, so the tolerance stop can fire. If the current or accepted mean R2 is missing, the strict comparison yields NA and the subsequent if (!OK) condition errors before the current convergence row and snapshots are written. Other invalid or non-finite inputs retain their underlying base R and data.table comparison behavior. Stored change and metric values are rounded as described in Files and final stitch.

Resume behavior

A fresh run recursively removes .path.iter, runs iteration zero on the original assembled CSVs, computes the first active set, and writes its snapshots. A recognized resume additionally requires current summary metrics.csv; absence stops with a diagnostic. Older convergence files lacking nActive are rewritten with that missing column.

Resume takes the largest recorded iteration, last recorded change, current summary metrics, and the matching Yo snapshot when present. It resets both strike counters, does not reconstruct active-set history, and sets the first resumed active set to NULL; that first augmentation therefore retains every available companion. It attempts maxIter further iterations. Resume state and subsequent writes are not transactional.

Files and final stitch

Iteration zero copies Y0.csv, Yo0.csv, metrics_0.csv, and a summary_0 directory. Every later iteration writes datasets_t, Yt.csv, Yot.csv, metrics_t.csv, summary_t, and a convergence row. Summary snapshots cover metrics.csv, response_long.csv, residuals_oof.csv, prediction_quantiles.csv, and overfit.csv. Missing source summary files are simply absent from a snapshot. Convergence rows contain iteration, change rounded to eight decimals, mean RMSE to six, mean R2 and per-response R2 to four, improvement, strike count, and active-set size.

At completion, response $r$ selects the first maximum R2 among baseline iteration zero and recorded iterations. Baseline R2 is read without rounding; later candidates use the four-decimal convergence values, so rounding can affect selection. Yo.csv and all five current summary files are then overwritten response by response from their selected snapshots. Missing snapshots or a missing response column stop the stitch.

Models, TRAIN/TEST splits, cell metric/response files, Y.csv, X.csv, and Xo.csv remain from the last fitted iteration; they are not stitched. Filesystem deletion, fitting, and writes have no rollback. Progress uses ssel's condition channels. RNG and bounded PSOCK effects are those of the composed supervised helpers; each fitted iteration receives the seed described above.

References

Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W., and Vlahavas, I. (2016). Multi-target regression via input space expansion: treating targets as inputs. Machine Learning, 104, 55–98. doi:10.1007/s10994-016-5546-z .

Examples

# \donttest{
root <- tempfile("ssel-chain-")
datasets_dir <- file.path(root, "datasets")
dir.create(datasets_dir, recursive = TRUE)

assembled <- data.frame(
  SampleID = c(paste0("S", 1:18), "P1", "P2"),
  set = c(rep("train", 18), "predict", "predict"),
  x = 1:20,
  z = rep(c(0, 1), 10),
  targetA = c(1.0, 2.4, 1.8, 3.7, 2.9, 5.1, 4.0, 6.2, 5.4,
              7.0, 6.1, 8.3, 7.2, 9.1, 8.0, 10.5, 9.4, 11.2,
              NA, NA),
  targetB = c(2.2, 1.3, 3.6, 2.7, 4.1, 3.8, 5.7, 4.6, 6.5,
              5.9, 7.8, 6.4, 8.6, 7.5, 9.9, 8.7, 10.4, 9.6,
              NA, NA)
)
utils::write.csv(assembled, file.path(datasets_dir, "demo.csv"),
                 row.names = FALSE, na = "")

old_quiet <- getOption("aR.quiet")
options(aR.quiet = TRUE)
chainPipeline(
  .path.datasets = datasets_dir,
  .path.train = file.path(root, "train"),
  .path.model = file.path(root, "model"),
  .path.metrics = file.path(root, "metrics"),
  .path.response = file.path(root, "response"),
  .path.summary = file.path(root, "summary"),
  .path.export = file.path(root, "export"),
  .path.iter = file.path(root, "iter"),
  key = "SampleID",
  responses = c("targetA", "targetB"),
  datasets = "demo",
  methods = "lm",
  seed = 2026,
  maxIter = 1,
  tol = 0,
  patience = 2,
  sweep = "jacobi"
)
options(aR.quiet = old_quiet)

data.table::fread(file.path(root, "iter", "convergence.csv"))
#>     iter     delta meanRMSE meanR2 improving strikes nActive r2_targetA
#>    <int>     <num>    <num>  <num>    <lgcl>   <int>   <int>      <num>
#> 1:     1 0.1585136 0.268126  0.991     FALSE       1       2     0.9926
#>    r2_targetB
#>         <num>
#> 1:     0.9894
data.table::fread(file.path(root, "export", "Yo.csv"))
#>     SampleID response     p    value        r2 dataset
#>       <char>   <char> <num>    <num>     <num>  <char>
#>  1:       P1  targetA  0.50 10.40260 0.9940376    demo
#>  2:       P2  targetA  0.50 12.36927 0.9940376    demo
#>  3:       P1  targetA  0.90 10.73040 0.9940376    demo
#>  4:       P2  targetA  0.90 12.69706 0.9940376    demo
#>  5:       P1  targetA  0.95 10.73522 0.9940376    demo
#>  6:       P2  targetA  0.95 12.70189 0.9940376    demo
#>  7:       P1  targetB  0.50 11.68119 0.9915743    demo
#>  8:       P2  targetB  0.50 10.75897 0.9915743    demo
#>  9:       P1  targetB  0.90 11.97954 0.9915743    demo
#> 10:       P2  targetB  0.90 11.05732 0.9915743    demo
#> 11:       P1  targetB  0.95 12.00615 0.9915743    demo
#> 12:       P2  targetB  0.95 11.08393 0.9915743    demo
# Final Yo rows can come from different best iterations by response.

unlink(root, recursive = TRUE)
# }