nf-core/configs: Cancer Research UK Manchester Institute Configuration

All nf-core pipelines have been successfully configured for the use on the HPC (griffin) at Cancer Research UK Manchester Institute.

To use, run the pipeline with -profile crukmi. This will download and launch the crukmi.config which has been pre-configured with a setup suitable for the Griffin HPC. Using this profile, Apptainer/Singularity images will be downloaded to run on the cluster and stored in a centralised location.

Before running the pipeline you will need to load Nextflow using the environment module system, for example via:

## Load Nextflow environment modules
module purge
module load apps/nextflow

The pipeline should always be executed inside a workspace on the /scratch/ system. All of the intermediate files required to run the pipeline will be stored in the work/ directory. It is recommended to delete this directory after the pipeline has finished successfully because it can get quite large.

Config file

See config file on GitHub

conf/crukmi
//Profile config names for nf-core/configs
params {
    config_profile_description = 'Cancer Research UK Manchester Institute HPC cluster profile provided by nf-core/configs'
    config_profile_contact     = 'Stephen Kitcatt, Simon Pearce (@skitcattCRUKMI, @sppearce)'
    config_profile_url         = 'http://scicom.picr.man.ac.uk/projects/user-support/wiki'
}
 
singularity {
    cacheDir   = '/lmod/nextflow_software'
    enabled    = true
    autoMounts = true
}
 
process {
    resourceLimits = [
        memory: 4000.GB,
        cpus: 96,
        time: 72.h
    ]
    beforeScript   = 'module load apps/apptainer'
    executor       = 'slurm'
    queue          = { task.memory <= 240.GB ? 'compute' : 'hmem' }
 
    errorStrategy  = { task.exitStatus in [143, 137, 104, 134, 139, 140] ? 'retry' : 'finish' }
    maxErrors      = '-1'
    maxRetries     = 3
 
    withLabel: process_single {
        cpus   = { 1 * task.attempt }
        memory = { 5.GB * task.attempt }
    }
 
    withLabel: process_low {
        cpus   = { 1 * task.attempt }
        memory = { 5.GB * task.attempt }
    }
 
    withLabel: process_medium {
        cpus   = { 4 * task.attempt }
        memory = { 20.GB * task.attempt }
    }
 
    withLabel: process_high {
        cpus   = { 48 * task.attempt }
        memory = { 240.GB * task.attempt }
    }
}
 
executor {
    name         = 'slurm'
    queueSize    = 1000
    pollInterval = '10 sec'
}
 
params {
    max_memory = 4000.GB
    max_cpus   = 96
    max_time   = 72.h
}