nf-core/mag

Edit

Introduction

This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline.

The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory.

Pipeline overview

The pipeline is built using Nextflow and processes data using the following steps:

• Quality control of input reads - trimming and contaminant removal
• Taxonomic classification of trimmed reads
• Digital sequencing normalisation
• Assembly of trimmed reads
• Protein-coding gene prediction of assemblies
• Virus identification of assemblies
• Binning and binning refinement of assembled contigs
• Taxonomic classification of binned genomes
• Genome annotation of binned genomes
• Additional summary for binned genomes
• Ancient DNA
• MultiQC - aggregate report, describing results of the whole pipeline
• Pipeline information - Report metrics generated during the workflow execution

Note that when specifying the parameter --coassemble_group, for the corresponding output filenames/directories of the assembly or downsteam processes the group ID, or more precisely the term group-[group_id], will be used instead of the sample ID.

Quality control

These steps trim away the adapter sequences present in input reads, trims away bad quality bases and sicard reads that are too short. It also removes host contaminants and sequencing controls, such as PhiX or the Lambda phage. FastQC is run for visualising the general quality metrics of the sequencing runs before and after trimming.

FastQC

FastQC gives general quality metrics about your sequenced reads. It provides information about the quality score distribution across your reads, per base sequence content (%A/T/G/C), adapter contamination and overrepresented sequences. For further reading and documentation see the FastQC help pages.

fastp

fastp is a all-in-one fastq preprocessor for read/adapter trimming and quality control. It is used in this pipeline for trimming adapter sequences and discard low-quality reads. Its output is in the results folder and part of the MultiQC report.

AdapterRemoval2

AdapterRemoval searches for and removes remnant adapter sequences from High-Throughput Sequencing (HTS) data and (optionally) trims low quality bases from the 3’ end of reads following adapter removal. It is popular in the field of palaeogenomics. The output logs are stored in the results folder, and as a part of the MultiQC report.

Remove PhiX sequences from short reads

The pipeline uses bowtie2 to map the reads against PhiX and removes mapped reads.

Host read removal

The pipeline uses bowtie2 to map short reads against the host reference genome specified with --host_genome or --host_fasta and removes mapped reads. The information about discarded and retained reads is also included in the MultiQC report.

Remove Phage Lambda sequences from long reads

The pipeline uses Nanolyse to map the reads against the Lambda phage and removes mapped reads.

Long read adapter removal

The pipeline uses porechop_abi or porechop to perform adapter trimming of the long reads that are eventually provided with the TSV input file.

Long read filtering

The pipeline uses filtlong, chopper, or nanoq for quality filtering of long reads, specified with --longread_filtering_tool <filtlong|chopper|nanoq>. Only filtlong is capable of filtering long reads against short reads, and is therefore currently recommended in the hybrid mode. If chopper is selected as long read filtering tool, Lambda Phage removal will be performed with chopper as well, instead of nanolyse.

Trimmed and filtered FASTQ output directories and files will only exist if --save_porechop_reads and/or --save_filtered_longreads (respectively) are provided to the run command.

No direct host read removal is performed for long reads. However, since within this pipeline filtlong uses a read quality based on k-mer matches to the already filtered short reads, reads not overlapping those short reads might be discarded. Note that this only applies when using filtlong as long read filtering tool. The lower the parameter --longreads_length_weight, the higher the impact of the read qualities for filtering. For further documentation see the filtlong online documentation.

Quality visualisation for long reads

NanoPlot is used to calculate various metrics and plots about the quality and length distribution of long reads. For more information about NanoPlot see the online documentation.

Digital normalization with BBnorm

If the pipeline is called with the --bbnorm option, it will normalize sequencing depth of libraries prior assembly by removing reads to 1) reduce coverage of very abundant kmers and 2) delete very rare kmers (see --bbnorm_target and --bbnorm_min parameters). When called in conjunction with --coassemble_group, BBnorm will operate on interleaved (merged) FastQ files, producing only a single output file. If the --save_bbnorm_reads parameter is set, the resulting FastQ files are saved together with log output.

Taxonomic classification of trimmed reads

Kraken

Kraken2 classifies reads using a k-mer based approach as well as assigns taxonomy using a Lowest Common Ancestor (LCA) algorithm.

Centrifuge

Centrifuge is commonly used for the classification of DNA sequences from microbial samples. It uses an indexing scheme based on the Burrows-Wheeler transform (BWT) and the Ferragina-Manzini (FM) index.

More information on the Centrifuge website

Assembly

Trimmed (short) reads are assembled with both megahit and SPAdes. Hybrid assembly is only supported by SPAdes.

MEGAHIT

MEGAHIT is a single node assembler for large and complex metagenomics short reads.

SPAdes

SPAdes was originally a single genome assembler that later added support for assembling metagenomes.

SPAdesHybrid

SPAdesHybrid is a part of the SPAdes software and is used when the user provides both long and short reads.

Metagenome QC with QUAST

QUAST is a tool that evaluates metagenome assemblies by computing various metrics. The QUAST output is also included in the MultiQC report, as well as in the assembly directories themselves.

Gene prediction

Protein-coding genes are predicted for each assembly.

Virus identification in assemblies

geNomad

geNomad identifies viruses and plasmids in sequencing data (isolates, metagenomes, and metatranscriptomes)

Binning and binning refinement

Contig sequencing depth

Sequencing depth per contig and sample is generated by MetaBAT2’s jgi_summarize_bam_contig_depths --outputDepth. The values correspond to (sum of exactly aligned bases) / ((contig length)-2*75). For example, for two reads aligned exactly with 10 and 9 bases on a 1000 bp long contig the depth is calculated by (10+9)/(1000-2*75) (1000bp length of contig minus 75bp from each end, which is excluded).

These depth files are used for downstream binning steps.

MetaBAT2

MetaBAT2 recovers genome bins (that is, contigs/scaffolds that all belongs to a same organism) from metagenome assemblies.

All the files and contigs in these folders will be assessed by QUAST and BUSCO.

All other files that were discarded by the tool, or from the low-quality unbinned contigs, can be found here.

All the files in this folder contain small and/or unbinned contigs that are not further processed.

Files in these two folders contain all contigs of an assembly.

MaxBin2

MaxBin2 recovers genome bins (that is, contigs/scaffolds that all belongs to a same organism) from metagenome assemblies.

All the files and contigs in these folders will be assessed by QUAST and BUSCO.

All the files in this folder contain small and/or unbinned contigs that are not further processed.

Files in these two folders contain all contigs of an assembly.

CONCOCT

CONCOCT performs unsupervised binning of metagenomic contigs by using nucleotide composition, coverage data in multiple samples and linkage data from paired end reads.

All the files and contigs in these folders will be assessed by QUAST and BUSCO, if the parameter --postbinning_input is not set to refined_bins_only.

Note that CONCOCT does not output what it considers ‘unbinned’ contigs, therefore no ‘discarded’ contigs are produced here. You may still need to do your own manual curation of the resulting bins.

DAS Tool

DAS Tool is an automated binning refinement method that integrates the results of a flexible number of binning algorithms to calculate an optimized, non-redundant set of bins from a single assembly. nf-core/mag uses this tool to attempt to further improve bins based on combining the MetaBAT2 and MaxBin2 binning output, assuming sufficient quality is met for those bins.

DAS Tool will remove contigs from bins that do not pass additional filtering criteria, and will discard redundant lower-quality output from binners that represent the same estimated ‘organism’, until the single highest quality bin is represented.

⚠️ If DAS Tool does not find any bins passing your selected threshold it will exit with an error. Such an error is ‘ignored’ by nf-core/mag, therefore you will not find files in the GenomeBinning/DASTool/ results directory for that particular sample.

By default, only the raw bins (and unbinned contigs) from the actual binning methods, but not from the binning refinement with DAS Tool, will be used for downstream bin quality control, annotation and taxonomic classification. The parameter --postbinning_input can be used to change this behaviour.

⚠️ Due to ability to perform downstream QC of both raw and refined bins in parallel (via --postbinning_input), bin names in DAS Tools’s *_allBins.eval file will include Refined. However for this particular file, they actually refer to the ‘raw’ input bins. The pipeline renames the input files prior to running DASTool to ensure they can be disambiguated from the original bin files in the downstream QC steps.

Tiara

Tiara is a contig classifier that identifies the domain (prokarya, eukarya) of contigs within an assembly. This is used in this pipeline to rapidly and with few resources identify the most likely domain classification of each bin or unbin based on its contig identities.

Typically, you would use tiara_summary.tsv as the primary file to see which bins or unbins have been classified to which domains at a glance, whereas the files in Taxonomy/Tiara provide classifications for each contig.

Bin sequencing depth

For each bin or refined bin the median sequencing depth is computed based on the corresponding contig depths.

QC for metagenome assembled genomes with QUAST

QUAST is a tool that evaluates genome assemblies by computing various metrics. The QUAST output is in the bin directories shown below. This QUAST output is not shown in the MultiQC report.

QC for metagenome assembled genomes

BUSCO

BUSCO is a tool used to assess the completeness of a genome assembly. It is run on all the genome bins and high quality contigs obtained by the applied binning and/or binning refinement methods (depending on the --postbinning_input parameter). By default, BUSCO is run in automated lineage selection mode in which it first tries to select the domain and then a more specific lineage based on phylogenetic placement. If available, result files for both the selected domain lineage and the selected more specific lineage are placed in the output directory. If a lineage dataset is specified already with --busco_db, only results for this specific lineage will be generated.

If the parameter --save_busco_db is set, additionally the used BUSCO lineage datasets are stored in the output directory.

Besides the reference files or output files created by BUSCO, the following summary files will be generated:

CheckM

CheckM provides a set of tools for assessing the quality of genomes recovered from isolates, single cells, or metagenomes. It provides robust estimates of genome completeness and contamination by using collocated sets of genes that are ubiquitous and single-copy within a phylogenetic lineage

By default, nf-core/mag runs CheckM with the check_lineage workflow that places genome bins on a reference tree to define lineage-marker sets, to check for completeness and contamination based on lineage-specific marker genes. and then subsequently runs qa to generate the summary files.

If the parameter --save_checkm_reference is set, additionally the used the CheckM reference datasets are stored in the output directory.

CheckM2

CheckM2 is a tool for assessing the quality of metagenome-derived genomes. It uses a machine learning approach to predict the completeness and contamination of a genome regardless of its taxonomic lineage.

If the parameter --save_checkm2_data is set, the CheckM2 reference datasets will be stored in the output directory.

GUNC

Genome UNClutterer (GUNC) is a tool for detection of chimerism and contamination in prokaryotic genomes resulting from mis-binning of genomic contigs from unrelated lineages. It does so by applying an entropy based score on taxonomic assignment and contig location of all genes in a genome. It is generally considered as a additional complement to CheckM results.

GUNC will be run if specified with --run_gunc as a standalone, unless CheckM is also activated via --qc_tool 'checkm', in which case GUNC output will be merged with the CheckM output using gunc merge_checkm.

If --gunc_save_db is specified, the output directory will also contain the requested database (progenomes, or GTDB) in DIAMOND format.

Taxonomic classification of binned genomes

CAT

CAT is a toolkit for annotating contigs and bins from metagenome-assembled-genomes. The nf-core/mag pipeline uses CAT to assign taxonomy to genome bins based on the taxnomy of the contigs.

If the parameters --cat_db_generate and --save_cat_db are set, additionally the generated CAT database is stored:

GTDB-Tk

GTDB-Tk is a toolkit for assigning taxonomic classifications to bacterial and archaeal genomes based on the Genome Database Taxonomy GTDB. nf-core/mag uses GTDB-Tk to classify binned genomes which satisfy certain quality criteria (i.e. completeness and contamination assessed with the BUSCO analysis).

Genome annotation of binned genomes

Prokka

Whole genome annotation is the process of identifying features of interest in a set of genomic DNA sequences, and labelling them with useful information. Prokka is a software tool to annotate bacterial, archaeal and viral genomes quickly and produce standards-compliant output files.

MetaEuk

In cases where eukaryotic genomes are recovered in binning, MetaEuk is also available to annotate eukaryotic genomes quickly with standards-compliant output files.

Additional summary for binned genomes

Ancient DNA

Optional, only running when parameter -profile ancient_dna is specified.

PyDamage

Pydamage, is a tool to automate the process of ancient DNA damage identification and estimation from contigs. After modelling the ancient DNA damage using the C to T transitions, Pydamage uses a likelihood ratio test to discriminate between truly ancient, and modern contigs originating from sample contamination.

variant_calling

Because of aDNA damage, de novo assemblers sometimes struggle to call a correct consensus on the contig sequence. To avoid this situation, the consensus is optionally re-called with a variant calling software using the reads aligned back to the contigs when --run_ancient_damagecorrection is supplied.

MultiQC

MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory.

Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info.

The general stats table at the top of the table will by default only display the most relevant pre- and post-processing statistics prior to assembly, i.e., FastQC, fastp/Adapter removal, and Bowtie2 PhiX and host removal mapping results.

Note that the FastQC raw and processed columns are right next to each other for improved visual comparability, however the processed columns represent the input reads after fastp/Adapter Removal processing (the dedicated columns of which come directly after the two FastQC set of columns). Hover your cursor over each column name to see the which tool the column is derived from.

Summary tool-specific plots and tables of following tools are currently displayed (if activated):

• FastQC (pre- and post-trimming)
• fastp
• Adapter Removal
• bowtie2
• BUSCO
• QUAST
• Kraken2 / Centrifuge
• PROKKA

Pipeline information

Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.