Introduction Usage Output Results

Introduction

This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline.

The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory.

Pipeline overview

The pipeline is built using Nextflow and processes data using the following steps:

  1. Per-sample preprocessing
    1. Convert all RDS files to h5ad format
    2. Create filtered matrix (if not provided)
    3. Present QC for raw counts (MultiQC)
    4. Remove ambient RNA
    5. Apply user-defined QC filters (can be defined per sample in the samplesheet)
    6. Doublet detection (Majority vote possible)
  2. Sample aggregation
    1. Merge into a single h5ad file
    2. Present QC for merged counts (MultiQC)
    3. Integration
  3. Cell type annotation
  4. Clustering and dimensionality reduction
    1. Leiden clustering
    2. UMAP
  5. Create report (MultiQC)

Per-sample preprocessing

Output files
  • preprocess/${sample_id}/
    • converted/: Contains the result of converting input file formats to h5ad.
    • unified/: Versions of the input files that have been optimized for usage in the pipeline.
    • empty_droplet_removal/: Results of empty droplet removal. Only if no filtered matrix is provided in the samplesheet.
    • qc_raw/: QC plots for the raw input data.
    • ambient_rna_removal/: Results of ambient RNA removal.
    • custom_thresholds/: Results of applying user-defined QC thresholds.
    • doublet_detection/: Directories related to doublet detection.
      • input_rds/: RDS version of the h5ad file that is used as input to the doublet detection tools.
      • (doubletdetection|scds|scrublet|solo)/: Results of doublet detection. Each directory contains a filtered h5ad/rds and a csv/pkl file with the doublet annotations.
      • ${sample_id}.h5ad: The h5ad without doublets.
    • qc_preprocessed/: QC plots for the preprocessed data.

nf-core/scdownstream covers a range of preprocessing methods. The output of each step is stored in the preprocess directory. The preprocess directory contains a subdirectory for each sample, which contains the results of the preprocessing steps.

Sample aggregation

Output files
  • combine/
    • merge/
      • merged_inner.h5ad: The merged h5ad file with only the intersection of the genes. Will be used for integration.
      • merged_outer.h5ad: The merged h5ad file with all genes. Will be used as base for the final h5ad file.
      • merged_sample_genes.png: UpSet plot showing the overlap of genes between samples.
    • integrate/
      • input_hvg
        • *.h5ad: The h5ad file that is used as input to the integration tools.
        • *.rds: RDS version of the h5ad file.
      • ${tool}
        • *.h5ad/*.rds: The integrated h5ad or rds file.
        • X_${tool}.pkl: Low-dimensional representation of the integrated data.

The combine directory contains the results of the sample aggregation step. The merge directory contains the merged h5ad files, which are used as input to the integration tools. The integrate directory contains the results of the integration step. The integrated h5ad files are stored in subdirectories named after the integration tool used.

Cell type annotation

Output files
  • celltypes/
    • celltypist/
      • *.h5ad: The h5ad file with cell type annotations.
      • *.pkl: The cell type annotations in a pickle file.

The celltypes directory contains the results of the cell type annotation step. So far, only celltypist is supported.

Clustering and dimensionality reduction

Output files
  • cluster_dimred/
    • ${integration}/
      • neighbors/
        • *.h5ad: The h5ad file with the neighborhood graph.
      • leiden/
        • ${resolution}/`
          • *.h5ad: The h5ad file with the leiden clustering.
          • *.pkl: The leiden clustering in a pickle file.
      • umap/
        • *.h5ad: The h5ad file with the UMAP coordinates.
        • *.pkl: The UMAP coordinates in a pickle file.

The cluster_dimred directory contains the results of the clustering and dimensionality reduction step. The results are stored in subdirectories named after the integration tool used.

Finalize

Output files
  • finalize/
    • merged.h5ad: The final h5ad file with all results.
    • merged.rds: RDS version of the final h5ad file.
    • merged_metadata.csv: Metadata of the final h5ad file.

The finalize directory contains the final results of the pipeline. The final h5ad file contains all results from the pipeline and is stored in the merged.h5ad file. The metadata of the final h5ad file is stored in the merged_metadata.csv file.

MultiQC

Warning

The MultiQC report is not yet fully implemented and will be improved in future releases. Until then, feel free to utilize CELLxGENE for interactive exploration of the results.

Output files
  • multiqc/
    • multiqc_report.html: a standalone HTML file that can be viewed in your web browser.
    • multiqc_data/: directory containing parsed statistics from the different tools used in the pipeline.
    • multiqc_plots/: directory containing static images from the report in various formats.

MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory.

Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info.

Pipeline information

Output files
  • pipeline_info/
    • Reports generated by Nextflow: execution_report.html, execution_timeline.html, execution_trace.txt and pipeline_dag.dot/pipeline_dag.svg.
    • Reports generated by the pipeline: pipeline_report.html, pipeline_report.txt and software_versions.yml. The pipeline_report* files will only be present if the --email / --email_on_fail parameter’s are used when running the pipeline.
    • Reformatted samplesheet files used as input to the pipeline: samplesheet.valid.csv.
    • Parameters used by the pipeline run: params.json.

Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.