University of Ghent High Performance Computing Infrastructure (VSC)

*NB:* You will need an account to use the HPC cluster to run the pipeline.

First you should go to the cluster you want to run the pipeline on. You can check what clusters have the most free space on this link. Use the following commands to easily switch between clusters:

module purge  
module swap cluster/<CLUSTER>

Before running the pipeline you will need to create a PBS script to submit as a job.

#!/bin/bash  
  
module load Nextflow  
  
nextflow run <pipeline> -profile vsc_ugent,<CLUSTER> <Add your other parameters>

All of the intermediate files required to run the pipeline will be stored in the work/ directory. It is recommended to delete this directory after the pipeline has finished successfully because it can get quite large, and all of the main output files will be saved in the results/ directory anyway.
The config contains a cleanup command that removes the work/ directory automatically once the pipeline has completed successfully. If the run does not complete successfully then the work/ dir should be removed manually to save storage space. The default work directory is set to $VSC_SCRATCH_VO_USER/work per this configuration

You can also add several TORQUE options to the PBS script. More about this on this link.

To submit your job to the cluster by using the following command:

qsub <script name>.pbs

*NB:* The profile only works for the clusters skitty, swalot, victini, kirlia and doduo.

*NB:* The default directory where the work/ and singularity/ (cache directory for images) is located in $VSC_SCRATCH_VO_USER.

Config file

See config file on GitHub 

// Set up the Tier 1 parameter  
params.validationSchemaIgnoreParams = params.validationSchemaIgnoreParams.toString() + ",tier1_project"  
if (!params.tier1_project) {  
    params.tier1_project = null  
}  
  
// Get the hostname and check some values for tier1  
def hostname = "doduo"  
try {  
    hostname = ['/bin/bash', '-c', 'sinfo --local -N -h | head -n 1 | cut -d " " -f1'].execute().text.trim()  
} catch (java.io.IOException e) {  
    System.err.println("WARNING: Could not run sinfo to determine current cluster, defaulting to doduo")  
}  
  
if(!params.tier1_project && hostname.contains("dodrio")){  
    System.err.println("Please specify your project with --tier1_project in your Nextflow command or with params.tier1_project in your config file.")  
    System.exit(1)  
}  
  
// Define the Scratch directory  
def scratch_dir =   System.getenv("VSC_SCRATCH_PROJECTS_BASE") ? "${System.getenv("VSC_SCRATCH_PROJECTS_BASE")}/${params.tier1_project}" : // Tier 1 scratch  
                    System.getenv("VSC_SCRATCH_VO_USER") ?: // VO scratch  
                    System.getenv("VSC_SCRATCH") // user scratch  
  
// Specify the work directory  
workDir = "$scratch_dir/work"  
  
// Perform work directory cleanup when the run has succesfully completed  
cleanup = true  
  
// Reduce the job submit rate to about 30 per minute, this way the server won't be bombarded with jobs  
// Limit queueSize to keep job rate under control and avoid timeouts  
// Extend the exit read timeout to 3 days to avoid timeouts on tier1 clusters  
executor {  
    submitRateLimit = '30/1min'  
    queueSize = 100  
    exitReadTimeout = "3day"  
}  
  
// Add backoff strategy to catch cluster timeouts and proper symlinks of files in scratch to the work directory  
process {  
    stageInMode = "symlink"  
    stageOutMode = "rsync"  
    errorStrategy = { sleep(Math.pow(2, task.attempt) \* 200 as long); return 'retry' }  
    maxRetries    = 5  
}  
  
// Specify that singularity should be used and where the cache dir will be for the images  
singularity {  
    enabled = true  
    autoMounts = true  
    cacheDir = "$scratch_dir/singularity"  
}  
  
env {  
    SINGULARITY_TMPDIR="$scratch_dir/.singularity"  
    APPTAINER_TMPDIR="$scratch_dir/.apptainer"  
}  
  
// AWS maximum retries for errors (This way the pipeline doesn't fail if the download fails one time)  
aws {  
    maxErrorRetry = 3  
}  
  
// Define profiles for each cluster  
profiles {  
    skitty {  
        params {  
            config_profile_description = 'HPC_SKITTY profile for use on the Skitty cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 177.GB  
            max_cpus = 36  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'skitty'  
        }  
    }  
  
    swalot {  
        params {  
            config_profile_description = 'HPC_SWALOT profile for use on the Swalot cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 116.GB  
            max_cpus = 20  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'swalot'  
        }  
    }  
  
    victini {  
        params {  
            config_profile_description = 'HPC_VICTINI profile for use on the Victini cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 88.GB  
            max_cpus = 36  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'victini'  
        }  
    }  
  
    kirlia {  
        params {  
            config_profile_description = 'HPC_KIRLIA profile for use on the Kirlia cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 738.GB  
            max_cpus = 36  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'kirlia'  
        }  
    }  
  
    doduo {  
        params {  
            config_profile_description = 'HPC_DODUO profile for use on the Doduo cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 250.GB  
            max_cpus = 96  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'doduo'  
        }  
    }  
  
    cpu_rome {  
        params {  
            config_profile_description = 'HPC_DODRIO_cpu_rome profile for use on the Dodrio/cpu_rome cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 256.GB  
            max_cpus = 128  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'dodrio/cpu_rome'  
            clusterOptions = "-A ${params.tier1_project}"  
        }  
    }  
  
    cpu_rome_512 {  
        params {  
            config_profile_description = 'HPC_DODRIO_cpu_rome_512 profile for use on the Dodrio/cpu_rome_512 cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 512.GB  
            max_cpus = 128  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'dodrio/cpu_rome_512'  
            clusterOptions = "-A ${params.tier1_project}"  
        }  
    }  
  
    cpu_milan {  
        params {  
            config_profile_description = 'HPC_DODRIO_cpu_milan profile for use on the Dodrio/cpu_milan cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 256.GB  
            max_cpus = 128  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'dodrio/cpu_milan'  
            clusterOptions = "-A ${params.tier1_project}"  
        }  
    }  
  
    gpu_rome_a100_40 {  
        params {  
            config_profile_description = 'HPC_DODRIO_gpu_rome_a100_40 profile for use on the Dodrio/gpu_rome_a100_40 cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 256.GB  
            max_cpus = 48  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'dodrio/gpu_rome_a100_40'  
            clusterOptions = "-A ${params.tier1_project}"  
        }  
    }  
  
    gpu_rome_a100_80 {  
        params {  
            config_profile_description = 'HPC_DODRIO_gpu_rome_a100_80 profile for use on the Dodrio/gpu_rome_a100_80 cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 512.GB  
            max_cpus = 48  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'dodrio/gpu_rome_a100_80'  
            clusterOptions = "-A ${params.tier1_project}"  
        }  
    }  
  
    debug_rome {  
        params {  
            config_profile_description = 'HPC_DODRIO_debug_rome profile for use on the Dodrio/debug_rome cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 256.GB  
            max_cpus = 48  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'dodrio/debug_rome'  
            clusterOptions = "-A ${params.tier1_project}"  
        }  
    }  
  
    cpu_rome_all {  
        params {  
            config_profile_description = 'HPC_DODRIO_cpu_rome_all profile for use on the Dodrio/cpu_rome_all cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 250.GB  
            max_cpus = 128  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'dodrio/cpu_rome_all'  
            clusterOptions = "-A ${params.tier1_project}"  
        }  
    }  
  
    gpu_rome_a100 {  
        params {  
            config_profile_description = 'HPC_DODRIO_gpu_rome_a100 profile for use on the Dodrio/gpu_rome_a100 cluster of the VSC HPC.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 384.GB  
            max_cpus = 48  
            max_time = "3day"  
        }  
  
        process {  
            executor = 'slurm'  
            queue = 'dodrio/gpu_rome_a100'  
            clusterOptions = "-A ${params.tier1_project}"  
        }  
    }  
  
    stub {  
        params {  
            config_profile_description = 'Stub profile for the VSC HPC. Please also specify the `-stub` argument when using this profile.'  
            config_profile_contact = 'ict@cmgg.be'  
            config_profile_url = 'https://www.ugent.be/hpc/en'  
            max_memory = 2.GB  
            max_cpus = 1  
            max_time = 1.h  
        }  
    }  
}
executor
slurm
contact
cmgg
homepage
https://www.ugent.be/hpc/en
get in touch
Ask a question on Slack Open an issue on GitHub