NCI Gadi HPC Configuration
nf-core pipelines have been successfully configured for use on the Gadi HPC at the National Computational Infrastructure (NCI), Canberra, Australia.
To run an nf-core pipeline at NCI Gadi, run the pipeline with -profile singularity,nci_gadi
. This will download and launch the nci_gadi.config
which has been pre-configured with a setup suitable for the NCI Gadi HPC cluster. Using this profile, a docker image containing all of the required software will be downloaded, and converted to a Singularity image before execution of the pipeline.
Access to NCI Gadi
Please be aware that you will need to have a user account, be a member of an Gadi project, and have a service unit allocation to your project in order to use this infrastructure. See the NCI user guide for details on getting access to Gadi.
Launch an nf-core pipeline on Gadi
Prerequisites
Before running the pipeline you will need to load Nextflow and Singularity, both of which are globally installed modules on Gadi. You can do this by running the commands below:
Execution command
Cluster considerations
Please be aware that as of July 2023, NCI Gadi HPC queues do not have external network access. This means you will not be able to pull the workflow code base or containers if you submit your nextflow run
command as a job on any of the standard job queues. NCI currently recommends you run your Nextflow head job either in a GNU screen or tmux session from the login node or submit it as a job to the copyq. See the nf-core documentation for instructions on running pipelines offline.
This config currently determines which Gadi queue to submit your task jobs to based on the amount of memory required. For the sake of resource and cost (service unit) efficiency, the following rules are applied by this config:
- Tasks requesting less than 128 Gb will be submitted to the normalbw queue
- Tasks requesting more than 128 Gb and less than 190 Gb will be submitted to the normal queue
- Tasks requesting more than 190 Gb and less than 1020 Gb will be submitted to the hugemembw queue
See the NCI Gadi queue limit documentation for details on charge rates for each queue.
Project accounting
This config uses the PBS environmental variable $PROJECT
to assign a project code to all task job submissions for billing purposes. If you are a member of multiple Gadi projects, you should confirm which project will be charged for your pipeline execution. You can do this using:
The version of Nextflow installed on Gadi has been modified to make it easier to specify resource options for jobs submitted to the cluster. See NCI’s Gadi user guide for more details. You can manually override the PROJECT with your project code. For example:
Resource usage
The NCI Gadi config summarises resource usage in a custom trace file that will be saved to your execution directory. However, for accounting or resource benchmarking purposes you may need to collect per-task service unit (SU) charges. Upon workflow completion, you can run the Sydney Informatics Hub’s gadi_nfcore_report.sh script in your workflow execution directory with:
This script will collect resources from the PBS log files printed to each task’s .command.log
. Resource requests and usage for each process is summarised in the output gadi-nf-core-joblogs.tsv
file. This is useful for resource benchmarking and SU accounting.