Introduction

In nf-core, we’ve been excited to adopt Wave to automate software container builds, and have been looking for the right way to do it. With the announcement of Seqera Containers we felt it was the right time to put in the effort to migrate our containers to be built using Wave, using Seqera Containers to host the container images for our modules. You can read more about our motivation for this change in Part 1 of this blog post.

Here, in Part 2, we will dig into the technical details: how it all works behind the curtain. You don’t need to know or understand any of this as an end-user of nf-core pipelines, or even as a contributor to nf-core modules, but we thought it would be interesting to share the details. It’s mostly to serve as an architectural plan for the nf-core maintainers and infrastructure teams.

The end goal

Before we dig into how the details of how the automation will work, let’s summarise the end goal of this migration.

Glossary

• linux/amd64: Regular intel CPUs (aka x86_64)
• linux/arm64: ARM CPUs (eg. AWS Graviton, aka AArch64). Not Apple Silicon.
• Apptainer: Alternative to Singularity, uses same image format
• Mamba: Alternative to Conda, uses same conda environment files
• Conda lock files: Explicit lists of packages, used to recreate an environment exactly.

Usage summary

Pipeline users will see almost no change in current behaviour, but have several new configuration profiles available.

Conda lock files

Conda lock files were mentioned in Part I of this blog post.

These pin the exact dependency stack used by the build, not just the top-level primary tool being requested. This effectively removes the need for conda to solve the build and also ships md5 hashes for every package. This will greatly improve the reproducibility of the software environments for conda users and the reliability of Conda CI tests.

They look something like this:

# micromamba env export --explicit
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
@EXPLICIT
https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2#d7c89558ba9fa0495403155b64376d81
https://conda.anaconda.org/conda-forge/linux-64/libgomp-13.2.0-h77fa898_7.conda#abf3fec87c2563697defa759dec3d639
https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2#73aaf86a425cc6e73fcf236a5a46396d
https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-13.2.0-h77fa898_7.conda#72ec1b1b04c4d15d4204ece1ecea5978
# .. and so on

Singularity: oras or https?

Unfamiliar with oras://? Don’t worry, it’s relatively new in the field. It’s a new protocol to reference container images, similar to docker:// or shub://. It allows Singularity to interact with any OCI (Open Container Initiative) compliant registry to pull images.

Using oras has some advantages:

• Singularity handles pulls in the process task, rather than in the Nextflow head job
  • This means less resource usage on the head node, and more parallelisation
• Singularity can use authentication to pull from private registries (see Singularity docs for more information).

However, there are some downsides:

• Shared cache Nextflow options such as $NXF_SINGULARITY_CACHEDIR and $NXF_SINGULARITY_LIBRARYDIR are not used
• Singularity must be installed when downloading images for offline use
• oras:// is only supported by recent versions of Singularity / Apptainer

As such, we will continue to use https downloads for Singularity SIF images for now. However, we will start to provide new -profile singularity_oras profiles for anyone who would prefer to fetch images using the newer oras protocol.

If you’d like to know more, check out the amazing bytesize talk by Marco Claudio De La Pierre (@marcodelapierre) from June 2024:

Modules

All nf-core pipelines use a single container per process, and the majority of processes are encapsulated within shared modules in the nf-core/modules repository. As such, we must start with containers at the module level.

Changes to main.nf

With this switch, we simplify the container declaration, listing only the default container image: Docker, for linux/amd64. There will no longer be any string interpolation or logic within the container string.

The container string is never edited by hand and is fully handled by the modules automation.

With the FastQC module as an example:

process FASTQC {
     label 'process_medium'
 
     conda "${moduleDir}/environment.yml"
+    container "fastqc:0.12.1--5cfd0f3cb6760c42" // automatically generated
-    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
-        'https://depot.galaxyproject.org/singularity/fastqc:0.12.1--hdfd78af_0' :
-        'biocontainers/fastqc:0.12.1--hdfd78af_0' }"
 
     input:
     tuple val(meta), path(reads)

We considered removing both conda and container declarations from the module main.nf file entirely. However, we see benefit in keeping these in this form because:

• It’s clearer to those exploring the code about what the module requires.
• The container string is needed to tie the module to the pipeline config files (see Building config files below)

Removing the container logic from the string should be a big win for readability.

Changes to meta.yml

Through the magic of automation, we will append and then validate the following fields within the module’s meta.yml file. Following the FastQC example from above:

# ..existing meta.yml content above
containers:
    docker:
        linux_amd64:
            name: community.wave.seqera.io/library/fastqc:0.12.1--5cfd0f3cb6760c42
            build_id: 5cfd0f3cb6760c42_1
            scan_id: 6fc310277b74
        linux_arm64:
            name: community.wave.seqera.io/library/fastqc:0.12.1--d3caca66b4f3d3b0
            build_id: d3caca66b4f3d3b0_1
            scan_id: d9a1db848b9b
    singularity:
        linux_amd64:
            name: oras://community.wave.seqera.io/library/fastqc:0.12.1--0827550dd72a3745
            https: https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/b2/b280a35770a70ed67008c1d6b6db118409bc3adbb3a98edcd55991189e5116f6/data
            build_id: 0827550dd72a3745_1
        linux_arm64:
            name: oras://community.wave.seqera.io/library/fastqc:0.12.1--b2ccdee5305e5859
            https: https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/76/76e744b425a6b4c7eb8f12e03fa15daf7054de36557d2f0c4eb53ad952f9b0e3/data
            build_id: b2ccdee5305e5859_1
    conda:
        linux_amd64:
            lock file: https://wave.seqera.io/v1alpha1/builds/5cfd0f3cb6760c42_1/condalock
        linux_arm64:
            lock file: https://wave.seqera.io/v1alpha1/builds/d3caca66b4f3d3b0_1/condalock

All images are all built at the same time, avoiding Conda dependency drift. The build and scan IDs allow us to trace back to the build logs and security scans for these images.

The Conda lock files are a new addition to the nf-core ecosystem and will help reproducibility for Conda users. These are generated during the Docker image build and are specific to architecture. The lock files can be accessed remotely via the Wave API, so we can treat them much in the same way that we treat remote container images.

Pipelines

Container information at module-level is great, but it’s not enough. Nextflow doesn’t know about module meta.yml files (they’re an nf-core invention), so we need to tie these into the pipeline code where they will run.

The heart of the solution is to auto-generate a config file for each software packaging type (Docker, Singularity, Conda) and platform (linux/arch64 and linux/arm64). These will be created by the nf-core/tools CLI and never be edited by hand, so no manual merging will be required. They’ll simply be regenerated and overwritten every time a version of a module is updated.

Each config file will specify the container or conda directive for every process in the pipeline:

// AUTOGENERATED CONFIG BELOW THIS POINT - DO NOT EDIT
process { withName: 'NF_PIPELINE:FASTQC'         { container = 'fastqc:0.12.1--5cfd0f3cb6760c42' } }
process { withName: 'NF_PIPELINE:MULTIQC'        { container = 'multiqc:1.25--9968ff4994a2e2d7' } }
process { withName: 'NF_PIPELINE:ANALYSIS_PLOTS' { container = 'express_click_pandas_plotly_typing:58d94b8a8e79e144' } }
//.. and so on, for each process in the pipeline

Likewise, the conda config files will point to the lock files for each process:

// AUTOGENERATED CONFIG BELOW THIS POINT - DO NOT EDIT
process { withName: 'NF_PIPELINE:FASTQC'         { conda = 'https://wave.seqera.io/v1alpha1/builds/5cfd0f3cb6760c42_1/condalock' } }
process { withName: 'NF_PIPELINE:MULTIQC'        { conda = 'https://wave.seqera.io/v1alpha1/builds/9968ff4994a2e2d7_1/condalock' } }
process { withName: 'NF_PIPELINE:ANALYSIS_PLOTS' { conda = 'https://wave.seqera.io/v1alpha1/builds/58d94b8a8e79e144_1/condalock' } }
//.. and so on, for each process in the pipeline

The main nextflow.config file will import these config files, depending on the profile selected by the person running the pipeline.

Singularity will have separate config files and associated -profiles for both oras and https containers, so that users can choose which to use.

Local modules and any edge-case shared modules that cannot use the Seqera Containers automation will need the pipeline developer to hardcode container names and conda lock files manually. These can be added to the above config files as long as they remain above the comment line:

// AUTOGENERATED CONFIG BELOW THIS POINT - DO NOT EDIT

We’re also taking this opportunity to update the apptainer and mamba profiles too, they will import the exact same config files as the singularity and conda profiles.

Here’s roughly how the nextflow.config file with the -profile config includes will look:

// Set container for docker amd64 by default
includeConfig 'config/containers_docker_amd64.config'
 
profiles {
    docker {
        docker.enabled      = true // Use the default config/containers_docker_amd64.config
    }
    docker_arm {
        includeConfig       'config/containers_docker_linux_arm64.config'
        docker.enabled      = true
    }
    // podman, shifter, charliecloud the same as docker - also with _arm versions
    singularity {
        includeConfig       'config/containers_singularity_linux_amd64.config'
        singularity.enabled = true
    }
    singularity_arm {
        includeConfig       'config/containers_singularity_linux_arm64.config'
        singularity.enabled = true
    }
    singularity_oras {
        includeConfig       'config/containers_singularity_oras_linux_amd64.config'
        singularity.enabled = true
    }
    singularity_oras_arm {
        includeConfig       'config/containers_singularity_oras_linux_arm64.config'
        singularity.enabled = true
    }
    apptainer {
        includeConfig       'config/containers_singularity_linux_amd64.config'
        apptainer.enabled = true
    }
    apptainer_arm {
        includeConfig       'config/containers_singularity_linux_arm64.config'
        apptainer.enabled = true
    }
    apptainer_oras {
        includeConfig       'config/containers_singularity_oras_linux_amd64.config'
        apptainer.enabled = true
    }
    apptainer_oras_arm {
        includeConfig       'config/containers_singularity_oras_linux_arm64.config'
        apptainer.enabled = true
    }
    conda {
        includeConfig       'config/conda_lock files_amd64.config'
        conda.enabled       = true
    }
    conda_arm {
        includeConfig       'config/conda_lock files_arm64.config'
        conda.enabled       = true
    }
    conda_env {
        conda.enabled       = true // Use the environment.yml file in the module main.nf
    }
    mamba {
        includeConfig       'config/conda_lock files_amd64.config'
        conda.enabled       = true
        conda.useMamba      = true
    }
    mamba_arm {
        includeConfig       'config/conda_lock files_arm64.config'
        conda.enabled       = true
        conda.useMamba      = true
    }
    mamba_env {
        conda.enabled       = true // Use the environment.yml file in the module main.nf
        conda.useMamba      = true
    }
}
 
docker.registry      = 'community.wave.seqera.io/library'
podman.registry      = 'community.wave.seqera.io/library'
apptainer.registry   = 'oras://community.wave.seqera.io/library'
singularity.registry = 'oras://community.wave.seqera.io/library'

Note that there are a few changes here:

• New profiles with _arm suffixes for linux/arm64 architectures
• New profiles for _oras suffixes for using the oras:// protocol
• The apptainer profiles now uses the singularity config files
• The conda profiles now use Conda lock files instead of environment.yml files
• New conda_env profiles for those wanting to keep the old behaviour
• New mamba profiles, using the conda config files
• Base registries set to Seqera Containers

Because we’re only defining the image name and making use of the base container registry config option, it should still be simple to mirror containers to custom Docker registries and overwrite only docker.registry as before.

Automation - Modules

The nf-core community loves automation. It’s baked into the core of our community from our shared interest in automating workflows. We have linting bots, template updates, slack workflows, pipeline announcements. You name it, we’ve automated it.

In these sections, we’ll cover how we’re going to build all of these shiny new things without manual intervention.

Updating conda packages

The automation begins when a contributor wants to add a piece of software to a container. For instance, they decided that they need samtools installed. The contributor updates the environment.yml and adds a line with samtools:

channels:
    - conda-forge
    - bioconda
dependencies:
    - bioconda::fastqc=0.12.1
    - bioconda::samtools=1.16.1

That will kick off the container image generation factory (it could equally be a change to remove a package, or change a pinned version).

A commit will be pushed automatically with an updated meta.yml file pointing to the new containers, plus new nf-test snapshots for the software version checks.

Only the interesting part needs to be edited by the developer (which tools to use) and all other steps are fully automated.

Container image creation

As with most automation in nf-core, container creation will happen in GitHub Actions. Edits to a module’s environment.yml file will trigger a workflow that uses the wave-cli to build the container images.

1. GitHub Actions identifies changes in the environment.yml file.
2. wave-cli is executed on the updated environment file.
3. Seqera Containers builds new containers for various platforms and architectures.
4. GitHub Actions runs stub tests commits the updated the version snapshot.

Once this GitHub Actions run completes it will push the new commits back to the PR and the regular nf-test CI will run.

nf-test versions snapshot

One of the primary reasons that we were so excited to adopt nf-test was the snapshot functionality. Every test has a snapshot file with the expected outputs, and the outputs are deterministic (not a binary file, and there’s no dates).

In that snapshot, we also capture the versions of the dependencies for the module (example shown for bowtie2):

    "versions": {
        "content": [
            {
                "BOWTIE_ALIGN": {
                    "bowtie": "1.3.0",
                    "samtools": "1.16.1"
                }
            }
        ],
        "meta": {
            "nf-test": "0.9.0",
            "nextflow": "24.04.4"
        },
        "timestamp": "2024-09-27T10:42:58.892298"
    },

This gives a second level of confirmation that the containers were correctly generated.

When updating containers, the nf-test snapshot is parsed and compared to the snapshot from before the new containers were built. The snapshot changes are discarded if anything other than the versions key changed, so it should only vary in the software versions reported. If any other changes are detected, the snapshot change will be rejected and not committed back to the PR. Then PR reviewers will see failing tests and need to manually update the snapshot file.

This means that we can automatically commit the updated snapshot file in the PR if the tool output is unchanged, saving the developer from taking this extra step.

Automatic version bumps with Renovate

We’ve recently adopted Renovate, a tool for automated dependency updates. It’s multi-platform and multi-language and has become pretty popular in the devops space. It’s similar to GitHub’s dependabot, but supports more languages and frameworks, and more importantly for nf-core, enables us to write our own custom dependencies.

Renovate runs on a schedule and automatically updates software versions for us based on the specifications we’ve laid out in a common config.

The magic starts with some nf-core automation to add renovate comments to the environment.yml file:

channels:
    - conda-forge
    - bioconda
dependencies:
    # renovate: datasource=conda depName=bioconda/bwa
    - bioconda::bwa=0.7.18
    # renovate: datasource=conda depName=bioconda/samtools
    - bioconda::samtools=1.20
    # renovate: datasource=conda depName=bioconda/htslib
    - bioconda::htslib=1.20.0

These comments will be added through the batch module updates happening this year. Future modules will have these comments added automatically and linted by the nf-core/tools CLI.

The comments allow some scary regexes to find the conda dependencies and their versions in nf-core/modules, and check if there’s a new version available. If there is a new version available, the Renovate bot will create a PR bumping the version, which in turn will kick off the container creation GitHub Action.

The process will be very similar to the diagram laid out above, however we can go a step further: if the new software versions have no effect on the results of the tests, the PR will be automatically merged:

So: if all tests pass, the pull request is automatically merged without human intervention. In case of test failures, the Renovate bot automatically requests a review from the appropriate module maintainer using the CODEOWNERS file. The maintainer then steps in to fix failing tests and request a final review before merging.

This efficient process ensures that software dependencies stay current with minimal manual oversight, reducing noise and streamlining development workflows. This will hopefully be the end of the “can I get a review on this version bump” requests in #review-requests!

Automation - Pipelines

We now have nice, up to date software packaging for the shared nf-core/modules with minimal manual intervention. However, we need to propagate these changes to the pipelines that use these modules. There are two main areas that we need to address:

Building config files

As described above, pipelines will have a set of config files automatically generated that specify the container or conda environment for each process in the pipeline.

Creation of these files will be triggered whenever installing, updating or removing a module, via the nf-core CLI. The config files will be completely regenerated each time, so there will never be any manual merging required.

The trickiest part of this process is linking the module containers to the pipeline processes. Modules can be imported into pipelines with any alias or scope. We need to match this against the values that we find in the module meta.yml files:

• Run nextflow inspect to generate default Docker linux/arch64 config
• Copy this file and replace the container names with the relevant container for each platform, using the meta.yml files that match the Docker container name

This is why we duplicate the default docker container in both main.nf and meta.yml files - it allows us to link the module to the pipeline.

Edge cases and local modules

There will always be edge cases that won’t fit the automation described above. Not all software tools can be packaged on Bioconda (though we encourage it where possible!). For example, some tools have licensing restrictions that prevent them from being distributed in this way. Other edge-cases include optional usage of container variants for GPU support, or other hardware.

Local modules won’t be able to benefit from the Wave CLI automation to fetch containers from Seqera Containers and will have to be manually updated by the pipeline developer.

For these reasons, we will still support custom container declarations in modules without use of Seqera Containers. It will be up to the module contributors to ensure that these are correctly specified and kept up to date manually.

These can be specified in the main.nf file and added to the autogenerated platform-specific config files, as long as they remain above the comment line:

// AUTOGENERATED CONFIG BELOW THIS POINT - DO NOT EDIT

If at all possible then software should be packaged with Bioconda and Seqera Containers. Failing that, custom containers should be stored under the nf-core account on quay.io. The only time other docker registries / accounts should be used are if there are licensing issues restricting software redistribution.

Custom containers can be also be built using Wave in continuous integration, it’s just that they can’t be pushed to the Seqera Containers registry. However, they can be pushed to quay.io automatically. We can do this using a similar mechanism to the automation used for changing environment.yml files, simply replacing it with Dockerfiles (see nf-core/modules#4940).

Downloads

The nf-core CLI has a download command that downloads pipeline code and software for offline use. This has a lot of hardcoded logic around the previous syntax of container strings and will need a significant rewrite.

By the time we get to running this tool, the pipeline has all containers defined in configuration files. As such, we should be able to run the new and improved nextflow inspect command to return the container names for every process for a given platform. Once collected, we can download the images as before.

The advantage of using this approach is that the download logic can be far simpler. Complex syntax for container strings is not a problem, as we can rely on Nextflow to resolve these to simple strings.

Roadmap

This blog post lays out a future vision for this project. It serves as both a rubber-duck for the authors, a place to request feedback from the community, and as a roadmap for developers.

There are many pieces of work that must come together for its completion, including but not limited to:

• Nextflow
  • Improve nextflow inspect to return all processes
• nf-core/tools
  • Add and lint module Renovate comments
  • Lint for nf-test snapshots
  • Write automation for creating pipeline container config files
  • Rewrite nf-core download
• nf-core/modules
  • Add renovate comments to environment.yml
  • Bulk update modules to use Seqera Containers
  • Build automation for fetching Seqera Containers

If all goes well, we hope to have the majority of this work completed by the end of 2024.

Published on

2 October 2024

Written by

@ewels

@edmundmiller