Troubleshooting basics
These are the recommended steps for troubleshooting a pipeline.
Start small
Before using the pipeline with your data, make sure to run a test using:
nextflow run nf-core/<pipeline_name> -profile test,docker
nextflow run nf-core/<pipeline_name> -profile test,docker
If Docker is not installed, you can replace docker
with singularity
or conda
, see
the Getting Started tutorial for further information. If a test fails, it might indicate that
there is an issue with the installation or configuration of Nextflow or software management tool, rather than a pipeline error.
You might also want to check the following:
- Nextflow is up to date. Use
nextflow self-update
to update a typical installation orconda update nextflow
for a Bioconda installation. - There is enough disk space, this will avoid running out of space while you are running the pipeline.
- Docker daemon is running (if you are using Docker to manage dependencies).
Categorize the type of error
For this step you try to identify when the error occurs:
-
Before the first process: errors that occur before the first process might be related to an outdated version of Nextflow, updating to the newest version could help solving the issue. An example error is:
N E X T F L O W ~ version 0.27.3 Launching `./main.nf` [prickly_snyder] - revision: bb0fa33a13 ERROR ~ Unknown config attribute: projectDir -- check config file: nextflow.config null -- Check '.nextflow.log' file for details
N E X T F L O W ~ version 0.27.3 Launching `./main.nf` [prickly_snyder] - revision: bb0fa33a13 ERROR ~ Unknown config attribute: projectDir -- check config file: nextflow.config null -- Check '.nextflow.log' file for details
-
During the first process: when an error appears during the first process it might indicate an issue with software dependencies, to specify how Nextflow should handle dependencies you need to select a configuration profile. This type of error might also be related to a missing command required to run the pipeline. Example error:
Command exit status: 127 Command output: (empty) Command error: .command.sh: line 3: rsem-prepare-reference: command not found Work dir: /home/lfaller/nextflow/rnaseq/work/f7/b6ef5a3f12f5efbf641f19046aca74 Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh` Unexpected error [AbortedException] -- Check script '/home/lfaller/.nextflow/assets/nf-core/rnaseq/./workflows/rnaseq.nf' at line: 603 or see '.nextflow.log' file for more details
Command exit status: 127 Command output: (empty) Command error: .command.sh: line 3: rsem-prepare-reference: command not found Work dir: /home/lfaller/nextflow/rnaseq/work/f7/b6ef5a3f12f5efbf641f19046aca74 Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh` Unexpected error [AbortedException] -- Check script '/home/lfaller/.nextflow/assets/nf-core/rnaseq/./workflows/rnaseq.nf' at line: 603 or see '.nextflow.log' file for more details
-
During run: for errors that occur while a pipeline is running or generating outputs it might be helpful to check log files as explained below.
-
While generating outputs: if an expected process output is missing, Nextflow will fail with the message
Missing output file(s)
. Then the error message of that tool will be displayed. Example error:[6:16 PM] Error executing process > 'FASTQC (hct116_h3k4me1_IP_R1_T1)' Caused by: Missing output file(s) `*.{zip,html}` expected by process `FASTQC (hct116_h3k4me1_IP_R1_T1)` Command executed: [ ! -f hct116_h3k4me1_IP_R1_T1.fastq.gz ] && ln -s hct116_h3k4me1_clean.fastq.gz hct116_h3k4me1_IP_R1_T1.fastq.gz fastqc -q -t 6 hct116_h3k4me1_IP_R1_T1.fastq.gz Command exit status: 0 Command output: (empty) Command error: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. Failed to process file hct116_h3k4me1_IP_R1_T1.fastq.gz uk.ac.babraham.FastQC.Sequence.SequenceFormatException: Ran out of data in the middle of a fastq entry. Your file is probably truncated at uk.ac.babraham.FastQC.Sequence.FastQFile.readNext(FastQFile.java:179) at uk.ac.babraham.FastQC.Sequence.FastQFile.next(FastQFile.java:125) at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:77) at java.base/java.lang.Thread.run(Thread.java:834)
[6:16 PM] Error executing process > 'FASTQC (hct116_h3k4me1_IP_R1_T1)' Caused by: Missing output file(s) `*.{zip,html}` expected by process `FASTQC (hct116_h3k4me1_IP_R1_T1)` Command executed: [ ! -f hct116_h3k4me1_IP_R1_T1.fastq.gz ] && ln -s hct116_h3k4me1_clean.fastq.gz hct116_h3k4me1_IP_R1_T1.fastq.gz fastqc -q -t 6 hct116_h3k4me1_IP_R1_T1.fastq.gz Command exit status: 0 Command output: (empty) Command error: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. Failed to process file hct116_h3k4me1_IP_R1_T1.fastq.gz uk.ac.babraham.FastQC.Sequence.SequenceFormatException: Ran out of data in the middle of a fastq entry. Your file is probably truncated at uk.ac.babraham.FastQC.Sequence.FastQFile.readNext(FastQFile.java:179) at uk.ac.babraham.FastQC.Sequence.FastQFile.next(FastQFile.java:125) at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:77) at java.base/java.lang.Thread.run(Thread.java:834)
Read the log and check the work directory
Checking the log files can help you to identify the type of error and where the error occurred. In order to search the output related to the error we need to understand the anatomy of the work directory:
command.out
STDOUT from tool.command.err
STDERR from tool.command.log
contains both STDOUT and STDERR from tool.command.begin
created as soon as the job launches.exitcode
created when the job ends, with exit code.command.trace
logs of compute resource usage.command.run
wrapper script used to run the job.command.sh
process command used for this task.
If you checked the files and identified the type of error and where it occurred but were unable to solve it you can always ask for help.
Asking for help
If you still have an issue with running the pipeline then feel free to contact us via the Slack channel. Please, consider the following guidelines:
- Pick the correct Slack channel to post in.
- Provide as much information as you can.
- As a minimum the command and configs you used.
- Use a Slack thread under your message if in doubt.
- Use markdown code blocks.
- Narrow the issue down as much as possible before asking.
- Explain the steps to reproduce if possible.
You can also open an issue in the respective pipeline repository on GitHub asking for help. In order to open the issue:
- Narrow the issue down as much as possible before opening the issue.
- Fill in the bug issue template.
- Explain the steps to reproduce.
- If you think you know the solution, please say so.
- If you think you can fix the problem, please make a pull request.
If you have problems that are directly related to Nextflow and not our pipelines or the nf-core framework tools then check out the Nextflow Slack Channel.
Troubleshooting talk
For more detailed information about troubleshooting a pipeline run, you can check out the Bytesize talk by Phil Ewels and the accompanying slides.
Configuration
Manual debugging on clusters using schedulers
In some cases, when testing configuration files on clusters using a scheduler, you may get failed jobs with ‘uninformative’ errors and vague information being given for the cause.
In such cases, a good way of debugging such a failed job is to change to the working directory of the failed process (which should be reported by Nextflow), and try to manually submit the job.
You can do this by submitting to your cluster the .command.run
file found in the working directory using the relevant submission command.
For example, let’s say you get an error like this on a SLURM cluster.
Caused by:
Failed to submit process to grid scheduler for execution
Command executed:
sbatch .command.run
Command exit status:
-
Command output:
(empty)
Work dir:
/<path>/<to>/work/e5/6cc8991c2b16c11a6356028228377e
Caused by:
Failed to submit process to grid scheduler for execution
Command executed:
sbatch .command.run
Command exit status:
-
Command output:
(empty)
Work dir:
/<path>/<to>/work/e5/6cc8991c2b16c11a6356028228377e
This does not tell you why the job failed to submit, but is often is due to a ‘invalid’ resource submission request, and the scheduler blocks it. But unfortunately, Nextflow does not pick the message reported by the cluster.
Therefore, in this case I would switch to the working directory, and submit the .command.run
file using SLURM’s sbatch
command (for submitting batch scripts).
$ cd /<path>/<to>/work/e5/6cc8991c2b16c11a6356028228377e
$ sbatch .command.run
sbatch: error: job memory limit for shared nodes exceeded. Must be <= 120000 MB
sbatch: error: Batch job submission failed: Invalid feature specification
$ cd /<path>/<to>/work/e5/6cc8991c2b16c11a6356028228377e
$ sbatch .command.run
sbatch: error: job memory limit for shared nodes exceeded. Must be <= 120000 MB
sbatch: error: Batch job submission failed: Invalid feature specification
In this case, SLURM has printed to my console the reason why, the job failed to be submitted.
With this information, I can go back to my configuration file, and tweak the settings accordingly, and run the pipeline again.
Input files not found
If the pipeline can’t find your files then you will get the following error
ERROR ~ Cannot find any reads matching: *{1,2}.fastq.gz
ERROR ~ Cannot find any reads matching: *{1,2}.fastq.gz
Direct input
This section mostly refers to DSL1 pipelines! Most DSL2 pipelines now use samplesheet inputs rather than direct read inputs.
Or when you’re using a input method like --input '/<path>/<to>/*_fq.gz'
, but only pick up one file, or only one file per pair being processed during the run, please note the following:
- The path must be enclosed in quotes (
'
or"
) - The path must have at least one
*
wildcard character i.e. following a ‘glob’ pattern. This is even if you are only running one paired end sample. - When using the pipeline with paired end data, the path must use
{1,2}
or{R1,R2}
notation to specify read pairs.- This notation is interpreted by Nextflow to mean anything with the same string other than R1 and R2 in the file name, will be be assumed to be a pair of files.
- If you are running single-end data make sure to specify
--single_end
- Your data should be organised in a ‘tidy’ manner
A few examples are as follows:
-
Running with a single, single-end FASTQ file as input (this will produce output files for this sample only)
nextflow run nf-core/<pipeline> --input 'my_data.fastq.gz` --single_end
nextflow run nf-core/<pipeline> --input 'my_data.fastq.gz` --single_end
-
Running multiple single-end FASTQ files as input using a wildcard glob pattern. This will find all files in the directory beginning with
my_
, and ending in.fastq.gz
, with each file with any other characters between those two strings being considered distinct samples (and will produce output files for each of the multiple input files).nextflow run nf-core/<pipeline> --input 'my_*.fastq.gz` --single_end
nextflow run nf-core/<pipeline> --input 'my_*.fastq.gz` --single_end
-
Running multiple paired-end FASTQ files as input using wildcard and grouping glob patterns. This will find all files in the directory beginning with
my_
, and ending in.fastq.gz
, with each file with any other characters between those two strings being considered distinct samples. However, any pair of file names that are exactly the same other thanR1
andR2
will be grouped together, and processed as related files. You will in most cases get output files for each distinct file, but with the*{R1,R2}
syntax, R1 and R2 pairs are collapsed into one.nextflow run nf-core/<pipeline> --input 'my_*{R1,R2}.fastq.gz`
nextflow run nf-core/<pipeline> --input 'my_*{R1,R2}.fastq.gz`
Note that if your sample name is “messy” then you have to be very particular with your glob specification (see point 2 above). A file name like L1-1-D-2h_S1_L002_R1_001.fastq.gz
can be difficult enough for a human to read. Specifying *{1,2}*.gz
will not give you what you want, whilst *{R1,R2}*.gz
will.
Output for only a single sample although I specified multiple with wildcards
You must specify paths to files in quotes, otherwise your shell (e.g. bash) will evaluate any wildcards (*) rather than Nextflow.
For example:
nextflow run nf-core/<pipeline> --input /path/to/sample_*/*.fq.gz
nextflow run nf-core/<pipeline> --input /path/to/sample_*/*.fq.gz
Maybe evaluated by your shell as:
nextflow run nf-core/<pipeline> --input /path/to/sample_1/sample_1.fq.gz /path/to/sample_1/sample_1.fq.gz /path/to/sample_1/sample_1.fq.gz
nextflow run nf-core/<pipeline> --input /path/to/sample_1/sample_1.fq.gz /path/to/sample_1/sample_1.fq.gz /path/to/sample_1/sample_1.fq.gz
And Nextflow will only take the first path after --input
, ignoring the others.
On the other hand, encapsulating the path in quotes will allow Nextflow to evaluate the paths.
nextflow run nf-core/<pipeline> --input "/path/to/sample_*/*.fq.gz"
nextflow run nf-core/<pipeline> --input "/path/to/sample_*/*.fq.gz"
Sample sheet input
If you are using a sample sheet or TSV input method, check there is not a mistake or typo in the path in a given column. Common mistakes are a trailing space at the end of the path, which can cause problems.
Data organization
The pipeline can’t take a list of multiple input files - it takes a glob expression. If your input files are scattered in different paths then we recommend that you generate a directory with symlinked files. If running in paired end mode please make sure that your files are sensibly named so that they can be properly paired. See the previous point.
The pipeline crashes almost immediately with an early pipeline step
Sometimes a newly downloaded and set up nf-core pipeline will encounter an issue where a run almost immediately crashes (e.g. at fastqc
, output_documentation
etc.) saying the tool could not be found or similar.
The first thing to do is always check the .nextflow.log
to see if it reports contains specific error. Common cases are described below.
Note that just because Nextflow reports a particular tool failed, this does not necessarily mean it’s an issue with the tool itself. It’s important to always fully read the error message to identify possible causes.
Tool not found
The most common case is when a user has forgotten to specify a container/environment profile.
If you do not specify one with the -profile
, Nextflow by default looks for all the required tools that will be all manually installed on your machine/cluster and specified $PATH
.
It is not recommended to run without a container/environment system as then your analysis will not be reproducible by others. You should pick one of: docker
, singularity
, podman
and conda
- depending on which your system already has available. See the nf-core Installation documentation for more information.
Error related to Docker
You may have an outdated container. This happens more often when running on the dev
branch of a nf-core pipeline, because Docker will not update the container on each new commit, and thus may not get new tools called within the pipeline code.
To fix, just re-pull the pipeline’s Docker container manually with:
docker pull nfcore/<pipeline>:dev
docker pull nfcore/<pipeline>:dev
If you work in a dev
branch, you may also want to consider putting a request for a pull in each run you do with nextflow, by putting this line of code in your nextflow.config file:
docker {
enabled = true
runOptions = '--pull=always'
}
docker {
enabled = true
runOptions = '--pull=always'
}
Error related to Singularity
If you’re running Singularity, it could be that Nextflow cannot access your Singularity image properly - often due to missing bind paths. See Cannot find input files when using Singularity for more information.
Sometimes, mksquashfs
cannot be found on the login node or workstation that you intend to use, thus the Singularity image build fails unfortunately. See below code snippet that shows such a typical failure:
Caused by:
Failed to pull singularity image
command: singularity pull --name nfcore-rnaseq-1.3.img docker://nfcore/rnaseq:1.3 > /dev/null
status : 255
.....
INFO: Creating SIF file...
FATAL: Unable to pull docker://nfcore/rnaseq:1.3: While searching for mksquashfs: exec: "mksquashfs": executable file not found in $PATH
Caused by:
Failed to pull singularity image
command: singularity pull --name nfcore-rnaseq-1.3.img docker://nfcore/rnaseq:1.3 > /dev/null
status : 255
.....
INFO: Creating SIF file...
FATAL: Unable to pull docker://nfcore/rnaseq:1.3: While searching for mksquashfs: exec: "mksquashfs": executable file not found in $PATH
If this is the case, please install mksquashfs
or ask your IT department to install the package for you.
Error related to HPC Schedulers
If working on a cluster, pipelines can crash if the profile used is not correctly configured for that environment. Typical issues can include missing cluster profile in -profile
, incorrectly specified executor, or incompatible memory/CPU node maximums set in that institutional profile. See nf-core/configs and Nextflow documentation for more information.
These types of error can look like the following:
Caused by:
Failed to submit process to grid scheduler for execution
Command executed:
sbatch .command.run
Command exit status:
1
Command output:
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
Caused by:
Failed to submit process to grid scheduler for execution
Command executed:
sbatch .command.run
Command exit status:
1
Command output:
sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
Cannot find input files when using Singularity
Depending on how you install Singularity on your system several reoccurring issues have been reported. Typically these result in error messages such as these:
Command error:
�[91mERROR : Failed to resolve path to /home/bla/singularity/mnt/container: No such file or directory
�[0m�[31mABORT : Retval = 255
�[0m
Command error:
�[91mERROR : Failed to resolve path to /home/bla/singularity/mnt/container: No such file or directory
�[0m�[31mABORT : Retval = 255
�[0m
You can verify if this is indeed a problem by using a Singularity Shell to access your container, and to check whether the required paths are available inside your container:
singularity shell /path/to/work/singularity/imagename.simg
ls -la /path/to/work
ls -la /path/to/network_storage
singularity shell /path/to/work/singularity/imagename.simg
ls -la /path/to/work
ls -la /path/to/network_storage
If any of these ls -la
commands returns a Not Found
error, you will need to set/update Singularity Bind Paths on your system.
The Singularity installation requires certain host paths to be bound. Please see Singularity Bind Paths Documentation for a more detailed explanation. In many cases this can be resolved by adding these paths to your /etc/singularity/singularity.conf
as highlighted in the documentation:
bind path = /beegfs/work/
bind path = /scratch
bind path = /gpfs
bind path = /home
bind path = /beegfs/work/
bind path = /scratch
bind path = /gpfs
bind path = /home
Alternatively, you can also add Singularity Bind Paths to your Nextflow call, e.g. using autoMounts
and/or runOptions
in the Singularity scope
Warning about sticked on revision
If you get a warning like the following:
Project nf-core/<pipeline> currently is sticked on revision: dev -- you need to specify explicitly a revision with the option -r to use it
Project nf-core/<pipeline> currently is sticked on revision: dev -- you need to specify explicitly a revision with the option -r to use it
This is a Nextflow error, with less-commonly seen Git ‘terminology’. What this means is that you have multiple versions of the pipeline pulled (e.g. 2.0.0, 2.1.0, 2.1.1, dev etc.), and it is not sure which one to use. Therefore, with every nextflow run nf-core/<PIPELINE>
command you should always indicate which version with -r
.
For example:
nextflow run nf-core/<pipeline> -r 2.1.0 --input '/<path>/<to>/data/*_{R1,R2}_*.fq.gz' <...>
nextflow run nf-core/<pipeline> -r 2.1.0 --input '/<path>/<to>/data/*_{R1,R2}_*.fq.gz' <...>
Specifying the version of the run you are using is highly recommended, as it helps in full reproducibility. In the sense that if you explicitly record the whole command with the version for your publication or internal reports, then anyone who wants to check your work can use the exact version you used (including all internal tools).
You can see more information on the Nextflow documentation here.
My pipeline crashes part way through a run at a certain step with a non 0 exit code
Sometimes part way through a run, a particular tool or step of the pipeline will fail. While Nextflow and nf-core pipelines try to solve some of these issues for you, this is not always possible. If the particular eventually tool fails, Nextflow will report the process that failed with a non-0 error code, and print the command that failed.
An example is as follows:
Error executing process > 'NFCORE_SAREK:SAREK:MARKDUPLICATES (DA117)'
Caused by:
Process `NFCORE_SAREK:SAREK:MARKDUPLICATES (DA117)` terminated with an error exit status (137)
Command executed:
picard -Xmx16384M -Xms16384M MarkDuplicates INPUT=DA117.bam OUTPUT=DA117_rmdup.bam REMOVE_DUPLICATES=TRUE AS=TRUE METRICS_FILE="DA117_rmdup.metrics" VALIDATION_STRINGENCY=SILENT
samtools index DA117_rmdup.bam
Command exit status:
137
Error executing process > 'NFCORE_SAREK:SAREK:MARKDUPLICATES (DA117)'
Caused by:
Process `NFCORE_SAREK:SAREK:MARKDUPLICATES (DA117)` terminated with an error exit status (137)
Command executed:
picard -Xmx16384M -Xms16384M MarkDuplicates INPUT=DA117.bam OUTPUT=DA117_rmdup.bam REMOVE_DUPLICATES=TRUE AS=TRUE METRICS_FILE="DA117_rmdup.metrics" VALIDATION_STRINGENCY=SILENT
samtools index DA117_rmdup.bam
Command exit status:
137
Each exit code can mean different things to different tools as well as in different environments. Therefore it is not always easy for developers to predict the exact issue and solution!
Common exit codes and and potential solutions are as follows:
Exit Code | Possible Cause | Solution |
---|---|---|
104 | out of memory | increase memory of process or number of retries in profile: Quick reference, Step By Step |
134 | out of memory | increase memory of process or number of retries in profile: Quick reference, Step By Step |
137 | out of memory | increase memory of process or number of retries in profile: Quick reference, Step By Step |
139 | out of memory | increase memory of process or number of retries in profile: Quick reference, Step By Step |
143 | out of memory | increase memory of process or number of retries in profile: Quick reference, Step By Step |
247 | out of memory | increase memory of process or number of retries in profile: Quick reference, Step By Step |
If in doubt, Google is your friend! Many exit codes are roughly similar across many tools; even if search results don’t mention your tool exactly, you can try a solution similar to the one proposed for the other tool.
If you are still unable to resolve the issue, please make a GitHub issue on the corresponding pipeline repository.
I get a exceeded job memory limit error
While Nextflow tries to make your life easier by automatically retrying jobs that run out of memory with more resources (until a specified max-limit), sometimes you may have such large data that you run out even after the default 3 retries.
To fix this you need to change the default memory requirements for the process that is breaking. We can do this by making a custom profile, which we then provide to the Nextflow run command.
For example, let’s say it’s the MARKDUPLICATES
process that is running out of memory (as displayed on the Nextflow running display).
-
First we need to check to see what default memory value we have. For this you need to go into the pipeline code locally in your local copy (typically:
~/.nextflow/assets/nf-core/<pipeline>/
, or the GitHub repository (https://github.com/nf-core/<pipeline>
) -
Go to the file here
modules/nf-core/<tool_name>/main.nf
, and check thelabel
line. -
Once found, check the line called
label
and note down the corresponding label. In this case the label could beprocess_low
. -
Go back to the main github repository, and open
conf/base.config
. Now, search in your browser forwithLabel:'process_low'
. -
Note what the
memory
field is set to (e.g.4.GB
) on a line like:memory = { check_max( 4.GB * task.attempt, 'memory' )})
. -
Back on your working machine, make a new text file called
custom_resources.conf
. This should be saved somewhere centrally so you can reuse it.If you think this would be useful for multiple people in your lab/institute, we highly recommend you make an institutional profile at nf-core/configs. This will simplify this process in the future.
-
Within this file, add the following. Note we have increased the default
4.GB
to16.GB
.process { withName: MARKDUPLICATES { memory = 16.GB } }
process { withName: MARKDUPLICATES { memory = 16.GB } }
InfoIn some cases you may need to use ‘fully resolved’ paths of the modules, as displayed in the nextflow console. e.g.
'NFCORE_SAREK:SAREK:MARKDUPLICATES'
, or in the pipeline’sconf/modules.config
file, if a module is used multiple times in a workflow and not all instances need the same resources.-
Note that with the above example you will not have the automatic retry mechanism that resubmits jobs with increased resource requests (given appropriate exit codes). The job will still be resubmitted on failure but with
16.GB
each time.-
If you want this, use the following syntax instead:
memory = { check_max( 16.GB * task.attempt, 'memory' ) }
memory = { check_max( 16.GB * task.attempt, 'memory' ) }
-
Next, copy the
check_max()
function from the pipeline’snextflow.config
file (e.g. here) to the bottom of your custom config file. -
16.GB * task.attempt
multiplies the memory request by the index of the retry. So if the job failed and is being tried a second time, it requests32.GB
. -
The
check_max()
function prevents Nextflow requesting excessive resources above what is available on your system. This effectively sets a ceiling on the resources and prevents the pipeline from crashing if it goes too high. Unfortunately because of the order in which pipeline code and Nextflow configs are parsed, this function needs to be defined in your custom config file.
-
-
-
Once saved, modify the original Nextflow run command to supply the custom_resources configuration file:
nextflow run nf-core/<pipeline> -c /<path>/<to>/custom_resources.conf -profile <original> <...>
nextflow run nf-core/<pipeline> -c /<path>/<to>/custom_resources.conf -profile <original> <...>
Crashed pipeline with an error but Nextflow is still running
If this happens, you can either wait until all other already running jobs to safely finish, or if Nextflow still does not stop press ctrl + c
on your keyboard (or equivalent) to stop the Nextflow run.
if you do this, and do not plan to fix the run make sure to delete the work
folder generated that is generated at the same as results
(or specified with the Nextflow variable -w
). Otherwise you may end up a lot of large intermediate files being left! You can clean a Nextflow run of all intermediate files with nextflow clean -f -k
or delete the work/
directory.
A step of a pipeline wasn’t executed
Possible options:
- If an optional step, check for a typo in the parameter name. Nextflow does not check for this (unless you created a workflow with the nf-core template, which provides parameter validation)
- Check that an upstream step/process was turned on (if a step/process requires the output of an earlier process, it will not be activated unless it receives the output of that process)
My pipeline update doesn’t seem to do anything
To download new version of a pipeline, you can use the following.
nextflow pull nf-core/<pipeline> -latest
nextflow pull nf-core/<pipeline> -latest
To download a previous version of a pipeline, you can instead use the following, replacing <version>
to the corresponding version (v3.4 for example)
nextflow pull nf-core/<pipeline> -r `<version>`
nextflow pull nf-core/<pipeline> -r `<version>`
However, in very rare cases, minor fixes to a version will be pushed out without a version number bump. This can confuse Nextflow slightly, as it thinks you already have the ‘broken’ version from your original pipeline download.
This shouldn’t happen with stable versions and normally only happens on
dev
branches.
If when running the pipeline you don’t see any changes in the fixed version when running it, you can try removing your Nextflow’s nf-core pipeline cache typically stored in your home directory with:
rm -r ~/.nextflow/assets/nf-core/<pipeline>
rm -r ~/.config/nfcore/nf-core/modules
rm -r ~/.nextflow/assets/nf-core/<pipeline>
rm -r ~/.config/nfcore/nf-core/modules
And re-pull the pipeline with the command above. This will install a fresh version of the version with the fixes.
Unable to acquire lock error
Errors like the following:
Unable to acquire lock on session with ID 84333844-66e3-4846-a664-b446d070f775
Unable to acquire lock on session with ID 84333844-66e3-4846-a664-b446d070f775
Normally suggest a previous Nextflow run (on the same folder) was not cleanly killed by a user (e.g. using ctrl + z to hard kill a crashed run).
To fix this, you must clean the entirety of the run’s work/
directory e.g. with rm -r work/
and re-running from scratch.
ctrl +z
is not a recommended way of killing a Nextflow job. Runs that take a long time to fail are often still running because other job submissions are still running. Nextflow will normally wait for those processes to complete before cleaning shutting down the run (to allow rerunning of a run with -resume
). ctrl + c
is much safer as it will tell Nextflow to stop earlier but cleanly.
Using a local version of iGenomes
The iGenomes config file uses params.igenomes_base
to make it possible to use a local copy of iGenomes. However, as custom config files are loaded after nextflow.config
and the igenomes.config
has already been imported and parsed, setting params.igenomes_base
in a custom config file has no effect and the pipeline will use the s3 locations by default. To overcome this you can specify a local iGenomes path by either:
- Specifying an
--igenomes_base
path in your execution command.
nextflow run nf-core/<pipeline> --input <input> -c <config> -profile <profile> --igenomes_base <path>/<to>/<data>/igenomes
nextflow run nf-core/<pipeline> --input <input> -c <config> -profile <profile> --igenomes_base <path>/<to>/<data>/igenomes
- Specifying the
igenomes_base
parameter in a parameters file provided with-params-file
inyaml
orjson
format.
nextflow run nf-core/<pipeline> -profile <profile> -params-file params.yml
nextflow run nf-core/<pipeline> -profile <profile> -params-file params.yml
Where the params.yml
file contains the pipeline params:
input: '/<path>/<to>/<data>/input'
igenomes_base: '/<path>/<to>/<data>/igenomes'
input: '/<path>/<to>/<data>/input'
igenomes_base: '/<path>/<to>/<data>/igenomes'
Extra resources and getting help
If you still have an issue with running the pipeline then feel free to contact us via the Slack channel or by opening an issue in the respective pipeline repository on GitHub asking for help.
If you have problems that are directly related to Nextflow and not our pipelines or the nf-core framework tools then checkout the Slack community chat or the discussion forum on GitHub.