Evaluate Pareto Solutions on Held-Out Data

Fits and evaluates every Pareto-optimal solution from optimize_panel() on held-out validation data. This provides the metrics data frame used by plot_pareto_front() without requiring a plot.

evaluate_pareto_solutions(
  optimization_result,
  x = NULL,
  y = NULL,
  cohort = NULL,
  objectives = NULL,
  on_error = c("warn", "stop"),
  regularized = TRUE,
  verbose = interactive()
)

Arguments

optimization_result

An OptimizationResult from optimize_panel().

x

Held-out validation data: a matrix, data.frame, SummarizedExperiment, or list of such objects. When NULL (default), automatically uses held-out data stored on optimization_result by optimize_panel() (present when train_ratio + val_ratio < 1).

y

Held-out validation labels: a factor (or list of factors when x is a list). When NULL (default), automatically uses held-out labels stored the same way.

cohort

Optional factor identifying cohort membership for each sample when x is a single matrix. Ignored when x is a list of cohorts. When x/y are taken from the stored held-out partition, the stored cohort labels are used unless overridden here.

objectives

Optional named list of objective descriptors (as returned by define_objectives()) overriding the objectives stored on optimization_result. Set to NA to skip objective re-scoring entirely and return only the descriptive columns.

on_error

One of "warn" or "stop". Controls behaviour when a solution fails evaluation. "warn" (default) skips the solution with a warning; "stop" raises an error immediately.

regularized

Logical; passed to fit_panel(). Default TRUE.

verbose

Logical; print progress messages. Default interactive().

Value

A data frame with one row per successfully evaluated solution and columns solution_id, n_features, n_base_features, sensitivity, specificity, auc, plus one obj_<name> column per optimisation objective. Attributes objective_labels and objective_directions map the obj_ columns back to their metric labels and optimisation directions.

Details

In addition to the always-present descriptive columns (sensitivity, specificity, auc at a 0.5 cutoff), each solution is re-scored on the objectives the optimisation actually targeted. Those are recovered from optimization_result@control$objectives (stored by optimize_panel()) and reported in columns prefixed obj_, so obj_auc or obj_sensitivity_at_specificity are directly comparable to the training-side values in solutions(optimization_result).

Objectives whose metric is cohort-aware (e.g. cohort_auc_gap, min_cohort_auc) cannot be computed when the evaluation data contains a single cohort; those columns are returned as NA with a warning. Supply x as a list of cohorts, or pass cohort, to make them computable.

See also

optimize_panel(), fit_panel(), evaluate_panel(), plot_pareto_front()

Examples

if (FALSE) { # \dontrun{
opt <- optimize_panel(x_train, y_train, objectives = define_objectives())
evaluate_pareto_solutions(opt, x_test, y_test)
} # }