1 Introduction to the Trio Class
15 May 2025
Source:vignettes/v01_intro_trio.Rmd
v01_intro_trio.Rmd
# devtools::load_all()
library(BenchHub)
library(clusterProfiler)
library(DOSE)
library(DO.db)
library(EnsDb.Hsapiens.v86)Motivation
BenchHub is an R ecosystem built to make benchmarking easier.
BenchHub contains two key components: Trio and
BenchmarkInsight:
- Trio object constucts benchmarking data by organising
data, evaluation metrics, gold standards (auxiliary data).
- BenchmarkInsight is a visualization tool that helps
interpret the results of benchmarking studies.
With BenchHub, researchers can quickly compare new methods, gain insights, and produce trustworthy their benchmarking studies.
Creating Trio object
The Trio class is designed to facilitate the storing and sharing of benchmarking datasets. Each Trio is structured around a single dataset but can include multiple metrics and multiple auxiliary data (such as references or gold standards).
Trio objects can be created using the Trio$new
constructor. There are 3 ways to create a Trio object:
- Curated Trio Datasets
- Source and ID
- Load an object directly
If the dataset can’t be loaded using Trio’s inbuilt loader, a custom loader can be provided.
Method 1: Curated Trio Datasets
You can directly use the name from the Curated Trio Datasets sheet to initialise a Trio object populated with some metrics and auxiliary data. This method is useful when you want to quickly start with a predefined dataset.
##
## ── Trio Object ─────────────────────────────────────────────────────────────────
##
## ── Dataset
## Dataset Details:
## spc_tbl_ [137 × 9] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
## - attr(*, "spec")=
## .. cols(
## .. ...1 = col_double(),
## .. trt = col_double(),
## .. celltype = col_double(),
## .. time = col_double(),
## .. status = col_double(),
## .. karno = col_double(),
## .. diagtime = col_double(),
## ... (truncated)
## Data Source: "figshare"
## Dataset ID: "26142922/47361073"
## Cache Path: "/tmp/Rtmpij1VMS"
## Split Indices: "None"
##
## ── Auxilliary Data
## Number of Auxiliary Data: 1
## Names of Auxiliary Data: "survival_data"
##
## ── Metrics
## Number of Metrics: 6
## Names of Metrics: "harrell_cindex", "begg_cindex", "uno_cindex", "gh_cindex",
## "brier_score", and "time_dep_auc"The above output shows that we have a Trio with a dataset, metrics, and auxiliary data. The dataset contains 137 rows and 9 columns, and the metrics and auxiliary data are already populated and printed.
This Trio is ready for use in survival prediction evaluation.
Method 2: Source and ID
Trio objects can be created by specifying an ID from a source with a valid trio downloader. This method is useful when you have a specific dataset ID from a supported source like Figshare, GEO, or ExperimentHub.
For example, if you have a dataset ID from Figshare, GEO, or ExperimentHub, you can create a Trio object as follows:
- figshare:
Trio$new("figshare:figshareID[/fileID]")-
fileIDcan optionally be provided to specify a specific file in the collection.
-
- GEO:
Trio$new("geo:GSEID[/Supplementary_filename]")-
Supplementary_filenamecan optionally be provided to specify a specific supplementary file in the series.
-
- experiementhub:
Trio$new("experiementhub:experimenthubID")
The example below shows how to create a Trio object using a Figshare
dataset with a fileID.
trioA <- Trio$new("figshare:26142922/47361079", cachePath = tempCache)
trioA##
## ── Trio Object ─────────────────────────────────────────────────────────────────
##
## ── Dataset
## Dataset Details:
## spc_tbl_ [58 × 19,820] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
## - attr(*, "spec")=
## .. cols(
## .. ...1 = col_character(),
## .. A1BG = col_double(),
## .. `A1BG-AS1` = col_double(),
## .. A1CF = col_double(),
## .. A2M = col_double(),
## .. `A2M-AS1` = col_double(),
## .. A2ML1 = col_double(),
## ... (truncated)
## Data Source: "figshare"
## Dataset ID: "26142922/47361079"
## Cache Path: "/tmp/Rtmpij1VMS"
## Split Indices: "None"
##
## ── Auxilliary Data
## Number of Auxiliary Data: 0
## Names of Auxiliary Data:
##
## ── Metrics
## Number of Metrics: 0
## Names of Metrics:Method 3: Load an object directly
Trio can also be created by passing an object directly into the constructor. This method is useful when you already have a dataset loaded in your R environment and want to use it with Trio.
If you have your own dataset, you can easily create a trio object as well. Below is an example using a microbiome dataset.
data("lubomski_microbiome_data", package = "BenchHub")
trioB <- Trio$new(data = lubomPD, datasetID = "lubomski_microbiome")
trioB##
## ── Trio Object ─────────────────────────────────────────────────────────────────
##
## ── Dataset
## Dataset Details:
## Factor w/ 2 levels "0","1": 2 1 2 1 2 2 1 2 2 1 ...
## Data Source:
## Dataset ID: "lubomski_microbiome"
## Cache Path:
## Split Indices: "None"
##
## ── Auxilliary Data
## Number of Auxiliary Data: 0
## Names of Auxiliary Data:
##
## ── Metrics
## Number of Metrics: 0
## Names of Metrics:Bonus: Using a custom loader
Trio supports custom loaders for data formats not directly supported by Trio. A loader is any function that takes in a path, provided by a downloader, and returns an object to be loaded into Trio.
Below, we use an anonymous function to wrap
GEOquery::getGEO and provide it with the path of the
downloaded file.
# This code will not run as it requires a GEO dataset to be downloaded
# GEO is unreliable
trioGEO <- Trio$new(
"GEO:GSE46474",
dataLoader = \(path) suppressMessages(GEOquery::getGEO(filename = path)),
cachePath = tempdir()
)
trioGEOAdding components to Trio
Adding metrics
In benchmarking studies, a metric refers to the measurement used to evaluate a specific task. In this example, we define a task called survival model prediction.
In Trio, a metric is any pairwise function of the form
f(expected, predicted) which returns a single value.
We can also add metrics that have additional arguments by passing a list of arguments to the args parameter.
eq <- \(expected, predicted, inequality = FALSE) {
if (inequality) {
return(!expected == predicted)
}
expected == predicted
}
trio$addMetric("equality", eq)
# Trio also supports passing through arguments to a metric
# Note: parameter names added for clarity
trio$addMetric(
name = "inequality", metric = eq, args = list(inequality = TRUE)
)In the above example, we added two metrics based on the same function: “equality” and “inequality”. The “equality” metric checks if the expected and predicted values are equal, while the “inequality” metric checks if they are not equal.
Underneeth the hood, Trio creates a wrapper function that calls the metric function with the specified arguments.
trio$metrics$inequality## function (auxData, to_eval)
## {
## do.call(metric, append(list(auxData, to_eval), args))
## }
## <environment: 0x558eaef5e590>Adding and getting auxData
auxData (Auxiliary Data) refers to gold standard in
evaluation tasks and falls into two categories:
- Internal: Extracted directly from the Trio dataset. This may include metadata such as cell types (e.g., B cells, CD4, CD8), patient conditions (e.g., healthy, disease), and spatial domain information. The specific content depends on the dataset itself.
- External: Retrieved from external databases, such as disease pathway databases or gene marker databases.
With the equality and inequality metrics
added above, we can now run an evaluation.
First we add some auxiliary data to the Trio object to evaluate against.
Auxiliary data is any data that allows you to make an evaluation. Auxiliary data is added linked to metrics that allow you to evaluate it.
# It can be a fixed quantity
trio$addAuxData(
name = "Number One", auxData = 1, metrics = c("equality", "inequality")
)
# Or it can be a function that acts on the data
trio$addAuxData(
name = "length", auxData = length, metrics = c("equality", "inequality")
)To view the auxiliary data one can use getAuxData.
trio$getAuxData("length") # This is equivalent to `length(trio$data)`## [1] 9Evaluation Example
Once Trio is set up with relevant metrics and auxData, we are ready to evaluate.
For simple evaluation:
| datasetID | auxData | metric | result |
|---|---|---|---|
| 26142922/47361073 | Number One | equality | TRUE |
| 26142922/47361073 | Number One | inequality | FALSE |
| 26142922/47361073 | length | equality | FALSE |
| 26142922/47361073 | length | inequality | TRUE |
For multi-method evaluation:
result <- trio$evaluate(list(
great_method = list("Number One" = 1, length = 575),
poor_method = list("Number One" = 2, length = 182)
))
knitr::kable(result)| datasetID | method | auxData | metric | result |
|---|---|---|---|---|
| 26142922/47361073 | great_method | Number One | equality | TRUE |
| 26142922/47361073 | great_method | Number One | inequality | FALSE |
| 26142922/47361073 | great_method | length | equality | FALSE |
| 26142922/47361073 | great_method | length | inequality | TRUE |
| 26142922/47361073 | poor_method | Number One | equality | FALSE |
| 26142922/47361073 | poor_method | Number One | inequality | TRUE |
| 26142922/47361073 | poor_method | length | equality | FALSE |
| 26142922/47361073 | poor_method | length | inequality | TRUE |
For a more detailed look at evaluation using Trio, please refer to vignette 2.
Other Features
Caching
Trio uses caching to avoid lengthy downloads after the first time a
data set is accessed. The cachePath parameter specifies the
path to the cache directory. If not specified, the cache directory
defaults to ~/.cache/R/TrioR/.
Data Splitting
Trio supports data splitting for cross-validation. The
split method splits the data into training and test sets
for cross-validation. The splitIndices attribute stores the
indices for each sample.
Indices are generated using the splitTools package. The
split method takes in the outcome variable and the number
of folds and repeats. The stratify parameter can be used to
stratify the outcome variable.
trio$split(y = 1:137, n_fold = 2, n_repeat = 5, seed = 1234, stratify = FALSE)
trio$splitIndices## $Fold1.Rep1
## [1] 2 3 4 5 6 7 8 9 10 11 12 13 18 21 22 24 25 26 30
## [20] 31 34 36 37 42 45 48 49 55 58 59 60 61 64 66 68 70 71 73
## [39] 76 77 81 82 87 89 90 91 92 93 96 97 98 102 103 104 105 108 109
## [58] 110 111 115 122 125 126 130 132 133 134 135
##
## $Fold2.Rep1
## [1] 1 14 15 16 17 19 20 23 27 28 29 32 33 35 38 39 40 41 43
## [20] 44 46 47 50 51 52 53 54 56 57 62 63 65 67 69 72 74 75 78
## [39] 79 80 83 84 85 86 88 94 95 99 100 101 106 107 112 113 114 116 117
## [58] 118 119 120 121 123 124 127 128 129 131 136 137
##
## $Fold1.Rep2
## [1] 3 4 5 6 7 8 13 15 21 22 23 24 28 29 30 31 32 34 35
## [20] 42 47 48 51 52 54 57 59 60 61 63 64 66 67 73 74 77 78 79
## [39] 82 84 85 89 90 92 93 94 96 98 99 102 103 108 109 110 112 114 116
## [58] 122 123 124 125 126 128 131 132 133 135 136
##
## $Fold2.Rep2
## [1] 1 2 9 10 11 12 14 16 17 18 19 20 25 26 27 33 36 37 38
## [20] 39 40 41 43 44 45 46 49 50 53 55 56 58 62 65 68 69 70 71
## [39] 72 75 76 80 81 83 86 87 88 91 95 97 100 101 104 105 106 107 111
## [58] 113 115 117 118 119 120 121 127 129 130 134 137
##
## $Fold1.Rep3
## [1] 2 5 6 8 9 11 13 14 17 19 20 21 24 26 27 30 31 32 34
## [20] 36 37 38 41 42 44 46 47 48 52 53 54 58 59 60 62 65 66 69
## [39] 70 71 72 75 76 77 78 80 84 85 86 88 89 90 91 92 93 94 99
## [58] 100 103 104 109 111 112 113 121 126 127 128 129
##
## $Fold2.Rep3
## [1] 1 3 4 7 10 12 15 16 18 22 23 25 28 29 33 35 39 40 43
## [20] 45 49 50 51 55 56 57 61 63 64 67 68 73 74 79 81 82 83 87
## [39] 95 96 97 98 101 102 105 106 107 108 110 114 115 116 117 118 119 120 122
## [58] 123 124 125 130 131 132 133 134 135 136 137
##
## $Fold1.Rep4
## [1] 1 2 5 8 10 14 16 17 19 22 23 24 25 27 28 31 33 34 37
## [20] 39 40 42 43 44 48 53 54 56 58 61 65 67 68 70 71 72 74 77
## [39] 78 80 81 82 83 84 85 87 89 91 93 95 97 102 104 111 113 119 120
## [58] 122 123 125 126 127 129 131 132 133 134 135
##
## $Fold2.Rep4
## [1] 3 4 6 7 9 11 12 13 15 18 20 21 26 29 30 32 35 36 38
## [20] 41 45 46 47 49 50 51 52 55 57 59 60 62 63 64 66 69 73 75
## [39] 76 79 86 88 90 92 94 96 98 99 100 101 103 105 106 107 108 109 110
## [58] 112 114 115 116 117 118 121 124 128 130 136 137
##
## $Fold1.Rep5
## [1] 2 3 5 6 7 8 9 10 11 13 14 15 18 19 21 23 29 31 33
## [20] 35 37 40 42 43 45 46 47 51 52 54 56 57 60 61 62 66 67 68
## [39] 69 72 76 79 81 86 87 93 94 97 100 101 102 103 106 107 108 111 115
## [58] 116 117 118 119 123 126 127 128 129 132 134
##
## $Fold2.Rep5
## [1] 1 4 12 16 17 20 22 24 25 26 27 28 30 32 34 36 38 39 41
## [20] 44 48 49 50 53 55 58 59 63 64 65 70 71 73 74 75 77 78 80
## [39] 82 83 84 85 88 89 90 91 92 95 96 98 99 104 105 109 110 112 113
## [58] 114 120 121 122 124 125 130 131 133 135 136 137Conclusion
In this vignette, we introduced the Trio class and demonstrated how to create a Trio object using curated datasets, source and ID, or loading an object directly. We also showed how to add metrics and auxiliary data to a Trio object and evaluate the performance of different methods using these metrics and auxiliary data. We hope this vignette helps you get started with Trio and conduct benchmarking studies more effectively.
Session Info
## R version 4.5.0 (2025-04-11)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.2 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0
##
## locale:
## [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
## [4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
## [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
## [10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
##
## time zone: UTC
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats4 stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] EnsDb.Hsapiens.v86_2.99.0 ensembldb_2.32.0
## [3] AnnotationFilter_1.32.0 GenomicFeatures_1.60.0
## [5] GenomicRanges_1.60.0 GenomeInfoDb_1.44.0
## [7] DO.db_2.9 AnnotationDbi_1.70.0
## [9] IRanges_2.42.0 S4Vectors_0.46.0
## [11] Biobase_2.68.0 BiocGenerics_0.54.0
## [13] generics_0.1.4 DOSE_4.2.0
## [15] clusterProfiler_4.16.0 BenchHub_0.99.5
## [17] BiocStyle_2.36.0
##
## loaded via a namespace (and not attached):
## [1] splines_4.5.0 BiocIO_1.18.0
## [3] polspline_1.1.25 bitops_1.0-9
## [5] ggplotify_0.1.2 tibble_3.2.1
## [7] R.oo_1.27.1 cellranger_1.1.0
## [9] datawizard_1.1.0 XML_3.99-0.18
## [11] rpart_4.1.24 httr2_1.1.2
## [13] lifecycle_1.0.4 vroom_1.6.5
## [15] lattice_0.22-6 MASS_7.3-65
## [17] insight_1.2.0 backports_1.5.0
## [19] magrittr_2.0.3 Hmisc_5.2-3
## [21] sass_0.4.10 rmarkdown_2.29
## [23] jquerylib_0.1.4 yaml_2.3.10
## [25] ggtangle_0.0.6 cowplot_1.1.3
## [27] DBI_1.2.3 RColorBrewer_1.1-3
## [29] abind_1.4-8 multcomp_1.4-28
## [31] purrr_1.0.4 R.utils_2.13.0
## [33] RCurl_1.98-1.17 yulab.utils_0.2.0
## [35] nnet_7.3-20 TH.data_1.1-3
## [37] rappdirs_0.3.3 sandwich_3.1-1
## [39] GenomeInfoDbData_1.2.14 enrichplot_1.28.2
## [41] ggrepel_0.9.6 tidytree_0.4.6
## [43] MatrixModels_0.5-4 performance_0.13.0
## [45] pkgdown_2.1.2 DelayedArray_0.34.1
## [47] codetools_0.2-20 tidyselect_1.2.1
## [49] aplot_0.2.5 UCSC.utils_1.4.0
## [51] farver_2.1.2 matrixStats_1.5.0
## [53] base64enc_0.1-3 googledrive_2.1.1
## [55] GenomicAlignments_1.44.0 jsonlite_2.0.0
## [57] Formula_1.2-5 survival_3.8-3
## [59] systemfonts_1.2.3 tools_4.5.0
## [61] treeio_1.32.0 ragg_1.4.0
## [63] Rcpp_1.0.14 glue_1.8.0
## [65] SparseArray_1.8.0 gridExtra_2.3
## [67] xfun_0.52 MatrixGenerics_1.20.0
## [69] qvalue_2.40.0 dplyr_1.1.4
## [71] withr_3.0.2 BiocManager_1.30.25
## [73] fastmap_1.2.0 SparseM_1.84-2
## [75] digest_0.6.37 R6_2.6.1
## [77] gridGraphics_0.5-1 textshaping_1.0.1
## [79] colorspace_2.1-1 GO.db_3.21.0
## [81] RSQLite_2.3.11 R.methodsS3_1.8.2
## [83] googlesheets4_1.1.1 tidyr_1.3.1
## [85] ggsci_3.2.0 data.table_1.17.2
## [87] rtracklayer_1.68.0 S4Arrays_1.8.0
## [89] httr_1.4.7 htmlwidgets_1.6.4
## [91] parameters_0.25.0 pkgconfig_2.0.3
## [93] gtable_0.3.6 blob_1.2.4
## [95] XVector_0.48.0 htmltools_0.5.8.1
## [97] bookdown_0.43 fgsea_1.34.0
## [99] ProtGenerics_1.40.0 scales_1.4.0
## [101] png_0.1-8 ggfun_0.1.8
## [103] splitTools_1.0.1 knitr_1.50
## [105] rstudioapi_0.17.1 tzdb_0.5.0
## [107] rjson_0.2.23 reshape2_1.4.4
## [109] checkmate_2.3.2 nlme_3.1-168
## [111] curl_6.2.2 survAUC_1.3-0
## [113] ggcorrplot_0.1.4.1 cachem_1.1.0
## [115] zoo_1.8-14 stringr_1.5.1
## [117] parallel_4.5.0 foreign_0.8-90
## [119] restfulr_0.0.15 desc_1.4.3
## [121] pillar_1.10.2 grid_4.5.0
## [123] vctrs_0.6.5 cluster_2.1.8.1
## [125] htmlTable_2.4.3 evaluate_1.0.3
## [127] readr_2.1.5 Rsamtools_2.24.0
## [129] mvtnorm_1.3-3 cli_3.6.5
## [131] compiler_4.5.0 rlang_1.1.6
## [133] crayon_1.5.3 rms_8.0-0
## [135] plyr_1.8.9 fs_1.6.6
## [137] stringi_1.8.7 BiocParallel_1.42.0
## [139] Biostrings_2.76.0 lazyeval_0.2.2
## [141] marginaleffects_0.25.1 bayestestR_0.15.3
## [143] GOSemSim_2.34.0 quantreg_6.1
## [145] Matrix_1.7-3 hms_1.1.3
## [147] patchwork_1.3.0 bit64_4.6.0-1
## [149] ggplot2_3.5.2 KEGGREST_1.48.0
## [151] SummarizedExperiment_1.38.1 dotwhisker_0.8.4
## [153] gargle_1.5.2 igraph_2.1.4
## [155] broom_1.0.8 memoise_2.0.1
## [157] bslib_0.9.0 ggtree_3.16.0
## [159] fastmatch_1.1-6 bit_4.6.0
## [161] ggstance_0.3.7 ape_5.8-1
## [163] gson_0.1.0