sc bench benchmark dashboard – sc_bench_benchmark

Category
Survey Analysis

Pure benchmark papers

130

New method development papers

152

Total number of readers

33

Total number of reading

433

Pure benchmark papers

New method development papers

d3 = require("d3@7")
jQuery = require("jquery")

category_list = [...new Set(paper_data.Paper_category)];

viewof category = Inputs.checkbox(
  category_list, 
  {
    value:  category_list, 
    label: "Data category: "
  })

datatype_list = [...new Set(
  paper_data.Paper_category
    .map((value, index) => category.includes(value) ? paper_data.data_type[index] : null)
    .filter(value => value !== null)
)];

//console.log("datatype_list", datatype_list)

viewof data_type = Inputs.select(
  datatype_list, 
  {
    value: "Single-cell RNA-seq",
    label: "Data type: "
  })

broaderTopic_list = 
  paper_data.Paper_category
    .map((value, index) => category.includes(value) && data_type.includes(paper_data.data_type[index]) ? paper_data.broader_topic[index] : null)
    .filter(value => value !== null);

viewof broader_topic = Inputs.select(
  broaderTopic_list, 
  {
    value: "Intermediate analysis",
    label: "Broader topic: ",
    unique: true
  })

finerTopic_list = [...new Set(
  paper_data.Paper_category
    .map((value, index) => category.includes(value) && 
    data_type.includes(paper_data.data_type[index]) && 
    broader_topic.includes(paper_data.broader_topic[index]) ? paper_data.finer_topic[index] : null).filter(value => value !== null)
)];

viewof finer_topic = Inputs.select(
 finerTopic_list, 
  {
    value: "Cell type/state identification",
    label: "Finer topic: "
  })

paperTitle_list = [...new Set(
  paper_data.Paper_category
    .map((value, index) => category.includes(value) && 
    data_type.includes(paper_data.data_type[index]) && 
    broader_topic.includes(paper_data.broader_topic[index]) &&
    finer_topic.includes(paper_data.finer_topic[index]) ? 
    paper_data.Paper_title[index] : null).filter(value => value !== null)
)];

viewof paper_title = Inputs.select(
 paperTitle_list, 
  {
    label: "Paper title: ",
    width: "100%"
  })

selected_paper_link = paper_links.Paper_title
    .map((value, index) => paper_title.includes(value) ? 
    paper_links.Link[index] : null).filter(value => value !== null)[0];
    
html`
  <div style="border: 1px solid #ddd; padding: 15px; border-radius: 5px; margin: 10px 0;">
    ${
      selected_paper_link
        ? html`
            <div>
              <p style="margin-bottom: 10px;">
                <strong>Research Paper Link:</strong>
              </p>
              <a href="${selected_paper_link}" target="_blank" rel="noopener noreferrer" style="word-wrap: break-word;">${selected_paper_link}</a>
            </div>
          `
        : html`
            <p>No paper link available.</p>
          `
    }
  </div>
`

selected_columns = [
  "Paper_category",
  "data_type",
  "broader_topic",
  "finer_topic",
  "publication_date",
  "Max_number_of_cells",
  "Data_availability",
  "Recommendation",
  "Applicability",
  "Sensitivity_analysis",
  "Variability_of_score"
]

filtered_paper_data = Object.keys(paper_data).reduce((obj, key) => {
  const index = paper_data.Paper_title.indexOf(paper_title);
  
  if (index !== -1) {
    obj[key] = paper_data[key][index];
  }
  
  return obj;
}, {});


selected_data = selected_columns.reduce((obj, key) => {
  if (filtered_paper_data[key] !== undefined) {
    obj[key] = filtered_paper_data[key];
  }
  return obj;
}, {});


// Create HTML for cards
html`
  <div style="display: flex; flex-wrap: wrap; gap: 10px;">
    ${Object.entries(selected_data).map(([key, value]) => {
      return html`
        <div style="border: 1px solid #ccc; border-radius: 8px; padding: 5px; width: 220px; box-shadow: 2px 2px 5px rgba(0,0,0,0.1);">
          <h4 style="margin: 0 0 0 0; font-size: 18px; color: #fff; background-color: #2C5488; padding: 5px; border-radius: 4px;">
            ${key.charAt(0).toUpperCase() + key.slice(1).replaceAll("_", " ")}
          </h4>
          <p style="margin: 5; font-size: 14px; color: #555;">
            ${value}
          </p>
        </div>
      `;
    })}
  </div>
`

Result table

Please refresh the page if the table is not shown.

Pure Benchmark

New method development paper

Pure Benchmark

New method development paper

Pure Benchmark

New method development paper

Statistics of attributes

The current landscape and emerging challenges of benchmarking single-cell methods

Introduction

A comprehensive evaluation strategy is critical for single-cell methodological development to assess the applicability of the method and to examine under what circumstances it works or fails.Informative evaluation results can drive the direction of method refinement. As a result, method development and evaluation should be considered as an iterative process.

Such evaluation should be distinguished from benchmarking purely based on evaluation metrics, as it aims to not only provide the method developers with a better understanding of their own methods but also demonstrate to the biologists how they can use the developed methods to gain new biological insights, which is what all the computational methods should be developed for. Due to the complexity of single-cell omics data, several key aspects should be taken into account while evaluating single-cell computational methods, including:

Accuracy

Whether the method does what it is designed to do. For example, data integration methods should remove the dataset effect and other unwanted variations while retaining the biological signals; and cell type classification methods should accurately classify the cell types and reveal the novel cell types that are not in the reference data. This aspect can usually be quantified by evaluation metrics, such as silhouette coefficients, adjusted rand index, Local Inverse Simpson’s Index (LISI) harmony and kBETbuttner2017assessment for data integration methods; and classification accuracy and F1-score for cell type classification methods.

Scalability

Whether the method can handle large-scale single-cell data in a computationally efficient manner. This requires benchmarking both the computational time and memory usage of the method.

Stability

Generalisability and Applicability (Stability of input): whether the method can be applied to data with different properties. This involves evaluating the developed method on data from diverse technologies (low- and high-throughput methods), tissues, organisms, stages (developmental and well-differentiated stages), disease, and sample sizes (tissue-specific and atlas-scale).
Robustness (Stability of model): whether the method is robust to noise in the data and the choice of hyper parameter. It can be evaluated by whether the method’s performance is significantly impacted when (i) only a subset of data is used; (ii) simulated noise is introduced into the data and (iii) models are run using different hyper parameter settings.
Reproducibility (Stability of output): technical vs broader. Whether the method produce same result when repeated with the same setting

Interpretability

Impact on downstream analysis: whether applying the method improves or hinders the performance of downstream analysis that is described in Section downstream. For example, for data integration of developmental data, it is important that the integrated data preserve the continuous signals so that the trajectory analysis can be applied to infer the pseudotime. For example, batch correction methods that return adjusted matrix would be more useful than methods that return embedding.
Biological impact: whether the developed method can help biologists to gain new biological insights. For example, data integration of multi-omics that is able to reveal rare and novel cell types, which could not be identified using single omics, would allow biologists to examine the cell characteristics of such rare cell types.