scATAC-benchmarking

Recent innovations in single-cell Assay for Transposase Accessible Chromatin using sequencing (scATAC-seq) enable profiling of the epigenetic landscape of thousands of individual cells. scATAC-seq data analysis presents unique methodological challenges. scATAC-seq experiments sample DNA, which, due to low copy numbers (diploid in humans) lead to inherent data sparsity (1-10% of peaks detected per cell) compared to transcriptomic (scRNA-seq) data (10-45% of expressed genes detected per cell). Such challenges in data generation emphasize the need for informative features to assess cell heterogeneity at the chromatin level.

We present a benchmarking framework that was applied to 10 computational methods for scATAC-seq on 13 synthetic and real datasets from different assays, profiling cell types from diverse tissues and organisms. Methods for processing and featurizing scATAC-seq data were evaluated by their ability to discriminate cell types when combined with common unsupervised clustering approaches. We rank evaluated methods and discuss computational challenges associated with scATAC-seq analysis including inherently sparse data, determination of features, peak calling, the effects of sequencing coverage and noise, and clustering performance. Running times and memory requirements are also discussed.

Single Cell ATAC-seq Benchmarking Framework

Our benchmarking results highlight SnapATAC, cisTopic, and Cusanovich2018 as the top performing scATAC-seq data analysis methods to perform clustering across all datasets and different metrics. Methods that preserve information at the peak-level (cisTopic, Cusanovich2018, Scasat) or bin-level (SnapATAC) generally outperform those that summarize accessible chromatin regions at the motif/k-mer level (chromVAR, BROCKMAN, SCRAT) or over the gene-body (Cicero, Gene Scoring). In addition, methods that implement a dimensionality reduction step (BROCKMAN, cisTopic, Cusanovich2018, Scasat, SnapATAC) generally show advantages over the other methods without this important step. SnapATAC is the most scalable method; it was the only method capable of processing more than 80,000 cells. Cusanovich2018 is the method that best balances analysis performance and running time.

All the analyses performed are illustrated in Jupyter Notebooks.

Within each dataset folder, the folder 'output' stores all the output files and it consists of five sub-folders including 'feature_matrices', 'umap_rds', 'clusters', 'metrics', and 'figures'.

Real Data

• The Buenrostro2018 dataset (folder 'Buenrostro_2018')

  • input
  • run_methods
    • BROCKMAN
    • chromVAR-kmers
    • chromVAR-motifs
    • Cicero
    • cisTopic
    • Control
    • Cusanovich2018
    • GeneScoring
    • scABC
    • Scasat
    • SCRAT
    • SnapATAC
  • clustering evaluation
  • UMAP visualization
  • end-to-end clustering (folder 'extra_clustering')
    • Cicero
    • cisTopic
    • Cusanovich2018
    • scABC
    • Scasat
    • SnapATAC
  • end-to-end clustering evaluation
  • output

• The Buenrostro2018 using bulk peaks dataset (folder 'Buenrostro_2018_bulkpeaks')

  • input
  • run_methods
    • chromVAR-kmers
    • chromVAR-motifs
    • Cicero
    • cisTopic
    • Control
    • Cusanovich2018
    • GeneScoring
    • scABC
    • Scasat
  • clustering evaluation
  • UMAP visualization
  • output

• The 10X PBMCs dataset (folder '10x_PBMC_5k')

  • input
  • run_methods
    • BROCKMAN
    • chromVAR-kmers
    • chromVAR-motifs
    • Cicero
    • cisTopic
    • Control
    • Cusanovich2018
    • GeneScoring
    • scABC
    • Scasat
    • SCRAT
    • SnapATAC
  • clustering evaluation
  • UMAP visualization
  • end-to-end clustering (folder 'extra_clustering')
    • Cicero
    • cisTopic
    • Cusanovich2018
    • scABC
    • Scasat
    • SnapATAC
  • end-to-end clustering evaluation
  • output

• The downsampled sci-ATAC-seq-mouse dataset (folder 'Cusanovich_2018_subset')

  • input
  • run_methods
    • BROCKMAN
    • chromVAR-kmers
    • chromVAR-motifs
    • Cicero
    • cisTopic
    • Control
    • Cusanovich2018
    • GeneScoring
    • scABC
    • Scasat
    • SCRAT
    • SnapATAC
  • clustering evaluation
  • UMAP visualization
  • end-to-end clustering (folder 'extra_clustering')
    • Cicero
    • cisTopic
    • Cusanovich2018
    • scABC
    • Scasat
    • SnapATAC
  • end-to-end clustering evaluation
  • output

• The full sci-ATAC-seq-mouse dataset (folder 'Cusanovich_2018')

  • input
  • run_methods
    • SnapATAC
  • clustering evaluation
  • UMAP visualization
  • output

Synthetic Data

• Data Simulation (folder 'Simulate_scATAC')

  • Bone Marrow

    • varying noise levels
    • varying coverages

  • Erythropoiesis

    • varying noise levels

• The simulated bone marrow (noise level 0.0) dataset (folder 'BoneMarrow_clean')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

• The simulated bone marrow (noise level 0.2) dataset (folder 'BoneMarrow_noisy_p2')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

• The simulated bone marrow (noise level 0.4) dataset (folder 'BoneMarrow_noisy_p4')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

• The simulated bone marrow (coverage 5k reads) dataset (folder 'BoneMarrow_cov5000')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

• The simulated bone marrow (coverage 2.5k reads) dataset (folder 'BoneMarrow_cov2500')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

• The simulated bone marrow (coverage 1k reads) dataset (folder 'BoneMarrow_cov1000')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

• The simulated bone marrow (coverage 500 reads) dataset (folder 'BoneMarrow_cov500')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

• The simulated bone marrow (coverage 250 reads) dataset (folder 'BoneMarrow_cov250')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

• The simulated erythropoiesis (noise level 0.0) dataset (folder 'Erythropoiesis_clean')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

• The simulated erythropoiesis (noise level 0.2) dataset (folder 'Erythropoiesis_noisy_p2')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

• The simulated erythropoiesis (noise level 0.4) dataset (folder 'Erythropoiesis_noisy_p4')

  • input
  • run_methods
  • clustering evaluation
  • UMAP visualization
  • output

Extra

• Test first PC

  • on the Buenrostro2018 dataset
  • on the 10X PBMCs dataset
  • on the downsampled sci-ATAC-seq-mouse dataset

• Test peaks

  • frequency-based peak selection
    • Control on the Buenrostro2018 dataset
    • Control on the simulated bone marrow (noise level 0.2) dataset
    • Cusanovich2018 on the Buenrostro2018 dataset
    • Cusanovich2018 on the simulated bone marrow (noise level 0.2) dataset
  • intensity-based peak selection
    • Control on the Buenrostro2018 dataset
    • Control on the simulated bone marrow (noise level 0.2) dataset
    • Cusanovich2018 on the Buenrostro2018 dataset
    • Cusanovich2018 on the simulated bone marrow (noise level 0.2) dataset

• Test pseudo-bulk

  • Cusanovich2018 on the Buenrostro2018 dataset
  • Cusanovich2018 on the 10X PBMCs dataset
  • Cusanovich2018 on the downsampled sci-ATAC-seq-mouse dataset

• Test blacklist

  • SCRAT on the Buenrostro2018 dataset
  • SnapATAC on the Buenrostro2018 dataset
  • SCRAT on the 10X PBMCs dataset
  • SnapATAC on the 10X PBMCs dataset
  • SCRAT on the downsampled sci-ATAC-seq-mouse dataset
  • SnapATAC on the downsampled sci-ATAC-seq-mouse dataset

• Test parameter settings

  • SCRAT: log2transform and adjustlen

Citation: Please cite our paper if you find this benchmarking work is helpful to your research. Huidong Chen, Caleb Lareau, Tommaso Andreani, Michael E. Vinyard, Sara P. Garcia, Kendell Clement, Miguel A. Andrade-Navarro, Jason D. Buenrostro & Luca Pinello. Assessment of computational methods for the analysis of single-cell ATAC-seq data. Genome Biology 20, 241 (2019).

Credits: H Chen, C Lareau, T Andreani, ME Vinyard, SP Garcia, K Clement, MA Andrade-Navarro, JD Buenrostro, L Pinello