One of the strongest features of Nextflow is that it can run on virtually any computational infrastructure. It has built-in support for HPC execution schedulers such as Slurm, SGE, PBS, LSF and more as well as cloud compute infrastructure such as AWS Batch and Google Cloud.
In order to get nf-core pipelines to run properly on your system, you will need to configure Nextflow so that it knows how best to run your analysis jobs. Each nf-core pipeline comes with "sensible defaults" for every configuration option. These are then overwritten as needed.
You do not need to edit the pipeline code to configure nf-core pipelines. If you edit the pipeline defaults then you cannot update to more recent versions of the pipeline without overwriting your changes. You also run the risk of moving away from the canonical pipeline and losing reproducibility.
Pipelines have configuration "profiles" that can be enabled with the command line flag
Multiple profiles can be specified in a comma-separated list.
Alternatively, you can create your own configuration files and supply these to the pipeline when running.
There are three main types of pipeline configuration that you can use:
- Basic pipeline configuration profiles
- Shared nf-core/configs configuration profiles
- Custom configuration files
Note that most users will need to specify an "executor" - telling Nextflow how to submit jobs to a job scheduler (e.g. SGE, LSF, SLURM, PBS, AWS Batch etc.). This can be done within share configuration profiles or in custom configuration files. For more information on how to specify an executor, please refer to the Nextflow documentation.
Every nf-core pipeline comes with the config profiles
Running with these profiles will instruct the pipeline to use the relevant software packaging method.
If you use a shared system with other people, it is best to use a configuration profile from nf-core/configs. These are shared config profiles loaded by all nf-core pipelines at run time.
You may find that your system already has a shared profile available here (see https://github.com/nf-core/configs). If not, please follow the instructions in the repository readme to add your cluster.
If you are the only person to be running this pipeline, you can create a local config file and use this. Nextflow looks for these files in three locations:
- User's home directory:
- Analysis working directory:
- Custom path specified on the command line:
-c path/to/config(multiple can be given)
Configuration parameters are loaded one after another and overwrite previous values.
Hardcoded pipeline defaults are first, then the user's home directory, then the work directory,
-c file in the order supplied, and finally command line
See the Nextflow documentation for more information about configuration syntax and available parameters.