azurebatch Configuration

To be used with the azurebatch profile by specifying the -profile azurebatch when running nf-core pipelines. Custom queue and storage need to be supplied with params.az_location, params.batch_name, params.batch_key, params.storage_name, params.sas_token.

Required Parameters

--storage_name

Name of Azure blob storage account.

--storage_key

Access key to Azure blob storage account. --storage_key or --storage_sas are required, but not both.

--storage_sas

SAS for access to Azure blob storage account (see relevant permissions on Nextflow documentation). --storage_key or --storage_sas are required, but not both.

--az_location

The Azure Batch region where the computation is executed in VMs. Default (westus2).

--batch_name

The Azure Batch account name.

--batch_key

The Azure Batch account key.

-w

The Azure Blob container to be used as Nextflow work directory (-w az://work).

--vm_type

VM size to use with Nextflow autopool or when creating a worker pool in Azure Batch. Make sure your Azure account has sufficient quota. Defaults to Standard_D8s_v3. See Azure VM Size documentation for more information.

--autopoolmode

Whether to use Nextflow autopool mode which creates an autoscaling pool for running Nextflow jobs. Defaults to true.

--allowpoolcreation

Allow Nextflow to create a pool for running Nextflow jobs. Defaults to true.

--deletejobs

Allow Nextflow to delete pools after completion. Defaults to true.

--az_worker_pool

Select an existing pool by name to run Nextflow jobs on. Defaults to auto (matching --autopoolmode).

--acr_registry

URL to Azure container registry for private docker images.

--acr_username

Username to access private Azure container registry.

--acr_password

Password to access private Azure container registry.

Azure Batch Setup

Please refer to the Nextflow documentation which describe how to setup the Azure Batch environment.

Config file

See config file on GitHub

azurebatch.config
//Nextflow config file for running on Azure batch
params {
  config_profile_description = 'Azure BATCH Cloud Profile'
  config_profile_contact = 'Venkat Malladi (@vsmalladi) & Adam Talbot (@adamrtalbot)'
  config_profile_url = 'https://azure.microsoft.com/services/batch/'
 
  // Storage
  storage_name      = null
  storage_key       = null
  storage_sas       = null
 
  // Batch
  az_location       = "westus2"
  batch_name        = null
  batch_key         = null
 
  vm_type           = "Standard_D8s_v3"
  autopoolmode      = true
  allowpoolcreation = true
  deletejobs        = true
  deletepools       = true
  az_worker_pool    = "auto"
 
  // ACR
  acr_registry      = null
  acr_username      = null
  acr_password      = null
 
}
 
process {
    executor = "azurebatch"
}
 
azure {
  process {
    queue = params.az_worker_pool
  }
  storage {
      accountName = params.storage_name
      accountKey  = params.storage_key
      sasToken    = params.storage_sas
  }
  batch {
      location                = params.az_location
      accountName             = params.batch_name
      accountKey              = params.batch_key
      tokenDuration           = "24h"
      autoPoolMode            = params.autopoolmode
      allowPoolCreation       = params.allowpoolcreation
      deleteJobsOnCompletion  = params.deletejobs
      deletePoolsOnCompletion = params.deletepools
      pools {
          auto {
              vmType    = params.vm_type
              autoScale = true
              vmCount    = 1
              maxVmCount = 12
          }
      }
  }
  registry {
      server   = params.acr_registry
      userName = params.acr_username
      password = params.acr_password
  }
}