nf-core/configs: KU Leuven/UHasselt Tier-2 High Performance Computing Infrastructure (VSC)
NB: You will need an account to use the HPC cluster to run the pipeline.
- Install Nextflow on the cluster
conda create --name nf-core python=3.12 nf-core nextflowA nextflow module is available that can be loaded module load Nextflow but it does not support plugins. So it’s not recommended
- Set up the environment variables in
~/.bashrcor~/.bash_profile:
If you have access to dedicated nodes, you can export these as a command separated list. These queues will only be used if specified task requirements are not available in the normal partitions but they are available in dedicated partitions. AMD is considered a dedicated partition.
export SLURM_ACCOUNT="<your-credential-account>"
# Comma-separated list of available dedicated partitions (if any)
# For example: export VSC_DEDICATED_QUEUES="dedicated_big_bigmem,dedicated_big_gpu"
export VSC_DEDICATED_QUEUES="<available-dedicated-partitions>"
# Needed for running Nextflow jobs
export NXF_HOME="$VSC_SCRATCH/.nextflow"
export NXF_WORK="$VSC_SCRATCH/work"
# Needed for running Apptainer containers
export APPTAINER_CACHEDIR="$VSC_SCRATCH/.apptainer/cache"
export APPTAINER_TMPDIR="$VSC_SCRATCH/.apptainer/tmp"
export NXF_CONDA_CACHEDIR="$VSC_SCRATCH/miniconda3/envs"
# Optional tower key
# export TOWER_ACCESS_TOKEN="<your_tower_access_token>"
# export NXF_VER="<version>" # make sure it's larger then 24.10.1The current config is setup with array jobs. Make sure nextflow version >= 24.10.1, read array jobs in nextflow you can do this in
export NXF_VER=24.10.1- Make the submission script.
NB: you should go to the cluster you want to run the pipeline on. You can check what clusters have the most free space using following command
sinfo --cluster wice|genius.
$ more job.pbs
#!/bin/bash -l
#SBATCH --account=...
#SBATCH --chdir=....
#SBATCH --partition=batch_long
#SBATCH --nodes="1"
#SBATCH --ntasks-per-node="1"
# module load Nextflow # does not support plugins
conda activate nf-core
nextflow run <pipeline> -profile vsc_kul_uhasselt,<CLUSTER> <Add your other parameters>NB: You have to specify your credential account, by setting
export SLURM_ACCOUNT="<your-credential-account>"else the jobs will fail!
Here the cluster options are:
- genius
- genius_gpu
- wice
- wice_gpu
- superdome
NB: The vsc_kul_uhasselt profile is based on a selected amount of SLURM partitions. The profile will select to its best ability the most appropriate partition for the job. Including modules with a label containing
gpuwill be allocated to a gpu partition when the ‘normal’geniusprofile is selected. Select thegenius_gpuorwice_gpuprofile to force the job to be allocated to a gpu partition. NB: If the module does not haveacceleratorset, it will determine the number of GPUs based on the requested resources.
Use the --cluster option to specify the cluster you intend to use when submitting the job:
sbatch --cluster=wice|genius job.slurm All of the intermediate files required to run the pipeline will be stored in the work/ directory. It is recommended to delete this directory after the pipeline has finished successfully because it can get quite large, and all of the main output files will be saved in the results/ directory anyway.
- Optional use nf-co2footprint
You can monitor the CO2 usage of your pipeline using the nf-co2footprint plugin using a nextflow version =>24.10.6. Monitoring the CO2 usage is fully optional and will only be activated when running the following command-line.
nextflow run <pipeline> -profile <CLUSTER> -plugins nf-co2footprint@1.0.0 --outdir your_output_folder <Add your other parameters>Config file
// Default to /tmp directory if $VSC_SCRATCH scratch env is not available,
// see: https://github.com/nf-core/configs?tab=readme-ov-file#adding-a-new-config
// NOTE: All helper variables and functions have been inlined into closures for
// compatibility with Nextflow config parser v2 (Nextflow >= 25.10).
// Perform work directory cleanup when the run has succesfully completed
// cleanup = true
// Reduce the job submit rate to about 50 per minute, this way the server won't be bombarded with jobs
// Limit queueSize to keep job rate under control and avoid timeouts
executor {
submitRateLimit = '50/1min'
queueSize = 50
exitReadTimeout = "10min"
}
// Add backoff strategy to catch cluster timeouts and proper symlinks of files in scratch to the work directory
process {
executor = 'slurm'
stageInMode = "symlink"
stageOutMode = "rsync"
errorStrategy = { sleep(Math.pow(2, task.attempt ?: 1) * 200 as long); return 'retry' }
maxRetries = 3
}
// Specify that singularity should be used and where the cache dir will be for the images
singularity {
enabled = true
autoMounts = true
cacheDir = "${System.getenv('VSC_SCRATCH') ?: '/tmp'}/.singularity"
pullTimeout = "30 min"
}
params {
config_profile_contact = 'GitHub: @Joon-Klaps - Email: joon.klaps@kuleuven.be'
config_profile_url = 'https://docs.vscentrum.be/en/latest/index.html'
}
co2footprint {
traceFile = "${params.get('outdir', "$launchDir")}/pipeline_info/co2footprint_trace_${new java.util.Date().format('yyyy-MM-dd_HH-mm-ss')}.txt"
summaryFile = "${params.get('outdir', "$launchDir")}/pipeline_info/co2footprint_summary_${new java.util.Date().format('yyyy-MM-dd_HH-mm-ss')}.txt"
reportFile = "${params.get('outdir', "$launchDir")}/pipeline_info/co2footprint_report_${new java.util.Date().format('yyyy-MM-dd_HH-mm-ss')}.html"
location = 'BE'
//pue = 1.33 // replace with PUE of your data center
machineType = 'compute cluster' // set to 'compute cluster', 'local', or 'cloud'
}
env {
APPTAINER_TMPDIR="${System.getenv('VSC_SCRATCH') ?: '/tmp'}/.apptainer/tmp"
APPTAINER_CACHEDIR="${System.getenv('VSC_SCRATCH') ?: '/tmp'}/.apptainer/cache"
}
// AWS maximum retries for errors (This way the pipeline doesn't fail if the download fails one time)
aws {
maxErrorRetry = 3
}
/*
* Queue Selection Logic for HPC Environments
* ===========================================
* Each process directive closure below contains inlined queue selection logic
* for compatibility with Nextflow config parser v2 (no top-level def allowed).
*
* Constants used throughout:
* TIME_THRESHOLD = 72.h (threshold for long-running jobs)
* MEMORY_THRESHOLD_GENIUS = 175.GB (bigmem threshold for GENIUS)
* MEMORY_THRESHOLD_WICE = 239.GB (high-memory threshold for WICE)
*
* Queue selection rules:
* GENIUS CPU: memory >= 175GB → bigmem/bigmem_long; else → batch/batch_long
* GENIUS GPU: memory >= 175GB → gpu_v100; else → gpu_p100/amd (with dedicated queue checks)
* WICE CPU: memory >= 239GB → bigmem,hugemem (time capped at 72h); else → batch/batch_icelake
* WICE GPU: memory >= 239GB → gpu_h100; else → gpu_a100,gpu (time capped at 72h unless dedicated)
*/
// Define profiles for each cluster
profiles {
genius {
params.config_profile_description = 'genius profile for use on the genius cluster of the VSC HPC.'
process {
// 768 - 65 so 65GB for overhead, max is 720000MB
resourceLimits = [ memory: 703.GB, cpus: 36, time: 168.h ]
beforeScript = {
// determineGeniusQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 175.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedBigmem = availQueues.contains('dedicated_big_bigmem')
def queueName
if (isHighMemory) {
queueName = isLongRunning ? (hasDedicatedBigmem ? 'dedicated_big_bigmem' : 'bigmem_long') : 'bigmem'
} else {
queueName = isLongRunning ? 'batch_long' : 'batch'
}
'module load cluster/genius/' + queueName.split(',')[0]
}
queue = {
// determineGeniusQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 175.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedBigmem = availQueues.contains('dedicated_big_bigmem')
if (isHighMemory) {
return isLongRunning ? (hasDedicatedBigmem ? 'dedicated_big_bigmem' : 'bigmem_long') : 'bigmem'
}
isLongRunning ? 'batch_long' : 'batch'
}
clusterOptions = {
// determineGeniusQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 175.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedBigmem = availQueues.contains('dedicated_big_bigmem')
def queueName
if (isHighMemory) {
queueName = isLongRunning ? (hasDedicatedBigmem ? 'dedicated_big_bigmem' : 'bigmem_long') : 'bigmem'
} else {
queueName = isLongRunning ? 'batch_long' : 'batch'
}
queueName =~ /dedicated/ ?
"--clusters=genius --account=lp_big_genius_cpu" :
"--clusters=genius --account=${System.getenv('SLURM_ACCOUNT') ?: null}"
}
withLabel: '.*gpu.*'{
resourceLimits = [ memory: 703.GB, cpus: 36 , time: 168.h ]
beforeScript = {
// determineGeniusGpuQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 175.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedGpu = availQueues.contains('dedicated_rega_gpu')
def hasAmdGpu = availQueues.contains('amd')
def queueName
if (isHighMemory) {
queueName = isLongRunning ? 'gpu_v100_long' : 'gpu_v100'
} else if (isLongRunning) {
queueName = hasDedicatedGpu ? 'dedicated_rega_gpu' : (hasAmdGpu ? 'amd_long' : 'gpu_p100_long')
} else {
queueName = hasAmdGpu ? 'amd' : 'gpu_p100'
}
'module load cluster/genius/' + queueName.split(',')[0]
}
apptainer.runOptions = '--containall --cleanenv --nv'
singularity.runOptions = '--containall --cleanenv --nv'
queue = {
// determineGeniusGpuQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 175.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedGpu = availQueues.contains('dedicated_rega_gpu')
def hasAmdGpu = availQueues.contains('amd')
if (isHighMemory) {
return isLongRunning ? 'gpu_v100_long' : 'gpu_v100'
}
if (isLongRunning) {
if (hasDedicatedGpu) return 'dedicated_rega_gpu'
if (hasAmdGpu) return 'amd_long'
return 'gpu_p100_long'
}
hasAmdGpu ? 'amd' : 'gpu_p100'
}
clusterOptions = {
def gpus = task.accelerator?.request ?: Math.max(1, Math.floor((task.cpus ?:1)/9) as int)
"--gres=gpu:${gpus} --clusters=genius --account=${System.getenv('SLURM_ACCOUNT') ?: null}"
}
}
}
}
genius_gpu {
params.config_profile_description = 'genius_gpu profile for use on the genius cluster of the VSC HPC.'
apptainer.runOptions = '--containall --cleanenv --nv'
singularity.runOptions = '--containall --cleanenv --nv'
process {
// 768 - 65 so 65GB for overhead, max is 720000MB
resourceLimits = [ memory: 703.GB, cpus: 36, time: 168.h]
beforeScript = {
// determineGeniusGpuQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 175.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedGpu = availQueues.contains('dedicated_rega_gpu')
def hasAmdGpu = availQueues.contains('amd')
def queueName
if (isHighMemory) {
queueName = isLongRunning ? 'gpu_v100_long' : 'gpu_v100'
} else if (isLongRunning) {
queueName = hasDedicatedGpu ? 'dedicated_rega_gpu' : (hasAmdGpu ? 'amd_long' : 'gpu_p100_long')
} else {
queueName = hasAmdGpu ? 'amd' : 'gpu_p100'
}
'module load cluster/genius/' + queueName.split(',')[0]
}
queue = {
// determineGeniusGpuQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 175.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedGpu = availQueues.contains('dedicated_rega_gpu')
def hasAmdGpu = availQueues.contains('amd')
if (isHighMemory) {
return isLongRunning ? 'gpu_v100_long' : 'gpu_v100'
}
if (isLongRunning) {
if (hasDedicatedGpu) return 'dedicated_rega_gpu'
if (hasAmdGpu) return 'amd_long'
return 'gpu_p100_long'
}
hasAmdGpu ? 'amd' : 'gpu_p100'
}
clusterOptions = {
def gpus = task.accelerator?.request ?: Math.max(1, Math.floor((task.cpus ?:1)/9) as int)
"--gres=gpu:${gpus} --clusters=genius --account=${System.getenv('SLURM_ACCOUNT') ?: null}"
}
}
}
wice {
params.config_profile_description = 'wice profile for use on the Wice cluster of the VSC HPC.'
process {
// max is 2016000
resourceLimits = [ memory: 1968.GB, cpus: 72, time: 168.h ]
beforeScript = {
// determineWiceQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 239.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedQueue = availQueues.contains('dedicated_big_bigmem')
def queueName
if (isHighMemory) {
if (isLongRunning && hasDedicatedQueue) {
queueName = 'dedicated_big_bigmem'
} else {
task.time = task.time > 72.h ? 72.h : task.time
queueName = 'bigmem,hugemem'
}
} else {
queueName = isLongRunning ?
'batch_long,batch_icelake_long,batch_sapphirerapids_long' :
'batch,batch_sapphirerapids,batch_icelake'
}
'module load cluster/wice/' + queueName.split(',')[0]
}
queue = {
// determineWiceQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 239.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedQueue = availQueues.contains('dedicated_big_bigmem')
if (isHighMemory) {
if (isLongRunning && hasDedicatedQueue) {
return 'dedicated_big_bigmem'
}
task.time = task.time > 72.h ? 72.h : task.time
return 'bigmem,hugemem'
}
isLongRunning ?
'batch_long,batch_icelake_long,batch_sapphirerapids_long' :
'batch,batch_sapphirerapids,batch_icelake'
}
clusterOptions = {
// determineWiceQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 239.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedQueue = availQueues.contains('dedicated_big_bigmem')
def queueName
if (isHighMemory) {
if (isLongRunning && hasDedicatedQueue) {
queueName = 'dedicated_big_bigmem'
} else {
task.time = task.time > 72.h ? 72.h : task.time
queueName = 'bigmem,hugemem'
}
} else {
queueName = isLongRunning ?
'batch_long,batch_icelake_long,batch_sapphirerapids_long' :
'batch,batch_sapphirerapids,batch_icelake'
}
queueName =~ /dedicated/ ?
"--clusters=wice --account=lp_big_wice_cpu" :
"--clusters=wice --account=${System.getenv('SLURM_ACCOUNT') ?: null}"
}
withLabel: '.*gpu.*' {
resourceLimits = [ memory: 703.GB, cpus: 64, time: 168.h ]
apptainer.runOptions = '--containall --cleanenv --nv'
singularity.runOptions = '--containall --cleanenv --nv'
beforeScript = {
// determineWiceGpuQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 239.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedQueue = isHighMemory ?
availQueues.contains('dedicated_big_gpu_h100') :
availQueues.contains('dedicated_big_gpu')
if (isLongRunning && !hasDedicatedQueue) {
task.time = task.time > 72.h ? 72.h : task.time
}
def queueName = isHighMemory ?
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') :
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu')
'module load cluster/wice/' + queueName.split(',')[0]
}
queue = {
// determineWiceGpuQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 239.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedQueue = isHighMemory ?
availQueues.contains('dedicated_big_gpu_h100') :
availQueues.contains('dedicated_big_gpu')
if (isLongRunning && !hasDedicatedQueue) {
task.time = task.time > 72.h ? 72.h : task.time
}
isHighMemory ?
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') :
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu')
}
clusterOptions = {
// determineWiceGpuQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 239.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedQueue = isHighMemory ?
availQueues.contains('dedicated_big_gpu_h100') :
availQueues.contains('dedicated_big_gpu')
if (isLongRunning && !hasDedicatedQueue) {
task.time = task.time > 72.h ? 72.h : task.time
}
def gpus = task.accelerator?.request ?: Math.max(1, Math.floor((task.cpus ?:1)/16) as int)
def queueValue = isHighMemory ?
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') :
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu')
queueValue =~ /dedicated_big_gpu_h100/ ? "--clusters=wice --account=lp_big_wice_gpu_h100 --gres=gpu:${gpus}" :
queueValue =~ /dedicated_big_gpu/ ? "--clusters=wice --account=lp_big_wice_gpu --gres=gpu:${gpus}" :
"--clusters=wice --account=${System.getenv('SLURM_ACCOUNT') ?: null} --gres=gpu:${gpus}"
}
}
}
}
wice_gpu {
params.config_profile_description = 'wice_gpu profile for use on the Wice cluster of the VSC HPC.'
apptainer.runOptions = '--containall --cleanenv --nv'
singularity.runOptions = '--containall --cleanenv --nv'
process {
// 768 - 65 so 65GB for overhead, max is 720000MB
resourceLimits = [ memory: 703.GB, cpus: 64, time: 168.h ]
beforeScript = {
// determineWiceGpuQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 239.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedQueue = isHighMemory ?
availQueues.contains('dedicated_big_gpu_h100') :
availQueues.contains('dedicated_big_gpu')
if (isLongRunning && !hasDedicatedQueue) {
task.time = task.time > 72.h ? 72.h : task.time
}
def queueName = isHighMemory ?
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') :
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu')
'module load cluster/wice/' + queueName.split(',')[0]
}
queue = {
// determineWiceGpuQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 239.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedQueue = isHighMemory ?
availQueues.contains('dedicated_big_gpu_h100') :
availQueues.contains('dedicated_big_gpu')
if (isLongRunning && !hasDedicatedQueue) {
task.time = task.time > 72.h ? 72.h : task.time
}
isHighMemory ?
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') :
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu')
}
clusterOptions = {
// determineWiceGpuQueue logic
def availQueues = (System.getenv('VSC_DEDICATED_QUEUES') ?: '').split(',')
def isHighMemory = task.memory >= 239.GB
def isLongRunning = task.time >= 72.h
def hasDedicatedQueue = isHighMemory ?
availQueues.contains('dedicated_big_gpu_h100') :
availQueues.contains('dedicated_big_gpu')
if (isLongRunning && !hasDedicatedQueue) {
task.time = task.time > 72.h ? 72.h : task.time
}
def gpus = task.accelerator?.request ?: Math.max(1, Math.floor((task.cpus ?:1)/16) as int)
def queueValue = isHighMemory ?
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu_h100' : 'gpu_h100') :
((isLongRunning && hasDedicatedQueue) ? 'dedicated_big_gpu' : 'gpu_a100,gpu')
queueValue =~ /dedicated_big_gpu_h100/ ? "--clusters=wice --account=lp_big_wice_gpu_h100 --gres=gpu:${gpus}" :
queueValue =~ /dedicated_big_gpu/ ? "--clusters=wice --account=lp_big_wice_gpu --gres=gpu:${gpus}" :
"--clusters=wice --account=${System.getenv('SLURM_ACCOUNT') ?: null} --gres=gpu:${gpus}"
}
}
}
superdome {
params.config_profile_description = 'superdome profile for use on the genius cluster of the VSC HPC.'
process {
clusterOptions = {"--clusters=genius --account=${System.getenv('SLURM_ACCOUNT') ?: null}"}
beforeScript = 'module load cluster/genius/superdome'
// 6000 - 228 so 228GB for overhead, max is 5910888MB
resourceLimits = [ memory: 5772.GB, cpus: 14, time: 168.h]
queue = { task.time <= 72.h ? 'superdome' : 'superdome_long' }
}
}
}