Creating a new pipeline

The create subcommand makes a new pipeline using the nf-core base template. With a given pipeline name, description and author, it makes a starter pipeline which follows nf-core best practices.

After creating the files, the command initialises the folder as a git repository and makes an initial commit. This first “vanilla” commit which is identical to the output from the templating tool is important, as it allows us to keep your pipeline in sync with the base template in the future. A full description of the file structure and all files created in the initial pipeline setup can be found here. See the nf-core syncing docs for more information on how the syncing works.

Once you have run the command, create a new empty repository on GitHub under your username (not the nf-core organisation, yet) and push the commits from your computer using the example commands in the above log. You can then continue to edit, commit and push normally as you build your pipeline.

Please see the nf-core documentation for a full walkthrough of how to create a new nf-core workflow.

If you prefer, you can supply the pipeline metadata also as command line arguments. See nf-core pipelines create --help for more information.

Customization

The nf-core pipelines create command comes with a number of options that allow you to customize the creation of a pipeline. This can be done in two ways: by using the graphical interface, or by supplying a template.yml file using the --template-yaml <file> option. Both options allow you to specify a custom pipeline prefix to use instead of the common nf-core, as well as selecting parts of the template to be excluded during pipeline creation. The interface will guide you through the pipeline creation process. Both nf-core and non-nf-core pipelines can be customised, but non-nf-core pipelines have more options which can be skipped. See the following example for a template.yml file:

name: coolpipe
description: A cool pipeline
author: me
prefix: myorg
skip:
    - github
    - ci
    - github_badges
    - igenomes
    - nf_core_configs
template:
    author: me
    description: A cool pipeline
    force: false
    is_nfcore: false
    name: coolpipe
    org: myorg
    outdir: .
    skip_features:
        - github
        - ci
        - igenomes
        - nf_core_configs
        - code_linters
        - citations
        - gitpod
        - codespaces
        - multiqc
        - fastqc
        - modules
        - changelog
        - nf_schema
        - license
        - email
        - adaptivecard
        - slackreport
        - documentation
        - test_config
        - seqera_platform
    version: 1.0.0dev

This will create a pipeline called coolpipe in the directory myorg-coolpipe (<prefix>-<name>) with me as the author. It will exclude all possible parts of the template.

A full list of possible features to skip from the template is described here:

github

This will create a GitHub repository for the pipeline.

The repository will include:

• Continuous Integration (CI) tests
• Issues and pull requests templates

The initialisation of a git repository is required to use the nf-core/tools. This means that even if you unselect this option, your pipeline will still contain a .git directory and .gitignore file.

ci

Nf-core provides a set of Continuous Integration (CI) tests for Github. When you open a pull request (PR) on your pipeline repository, these tests will run automatically.

There are different types of tests:

• Linting tests check that your code is formatted correctly and that it adheres to nf-core standards For code linting they will use prettier.
• Pipeline tests run your pipeline on a small dataset to check that it works These tests are run with a small test dataset on GitHub and a larger test dataset on AWS
• Marking old issues as stale

igenomes

Nf-core pipelines are configured to use a copy of the most common reference genome files.

By selecting this option, your pipeline will include a configuration file specifying the paths to these files.

The required code to use these files will also be included in the template. When the pipeline user provides an appropriate genome key, the pipeline will automatically download the required reference files.

For more information about reference genomes in nf-core pipelines, see the nf-core docs.

github_badges

The pipeline README.md will include badges for:

• AWS CI Tests
• Zenodo DOI
• Nextflow
• Conda
• Docker
• Singularity
• Launching on Nextflow Tower

nf_core_configs

Nf-core has a repository with a collection of configuration profiles.

Those config files define a set of parameters which are specific to compute environments at different Institutions. They can be used within all nf-core pipelines. If you are likely to be running nf-core pipelines regularly it is a good idea to use or create a custom config file for your organisation.

For more information about nf-core configuration profiles, see the nf-core/configs repository

is_nfcore

No help text available

code_linters

Pipelines include code linters to check the formatting of your code in order to harmonize code styles between developers. Linters will check all non-ignored files, e.g., JSON, YAML, Nextlow or Python files in your repository. The available code linters are:

• pre-commit (https://pre-commit.com/): used to run all code-linters on every PR and on ever commit if you run pre-commit install to install it in your local repository.
• editor-config (https://github.com/editorconfig-checker/editorconfig-checker): checks rules such as indentation or trailing spaces.
• prettier (https://github.com/prettier/prettier): enforces a consistent style (indentation, quoting, line length, etc).

citations

If adding citations, the pipeline template will contain a CITATIONS.md file to add the citations of all tools used in the pipeline.

Additionally, it will include a YAML file (assets/methods_description_template.yml) to add a Materials & Methods section describing the tools used in the pieline, and the logics to add this section to the output MultiQC report (if the report is generated).

gitpod

Gitpod (https://www.gitpod.io/) provides standardized and automated development environments.

Including this to your pipeline will provide an environment with the latest version of nf-core/tools installed and all its requirements. This is useful to have all the tools ready for pipeline development.

codespaces

The pipeline will include a devcontainer configuration. The devcontainer will create a GitHub Codespaces for Nextflow development with nf-core/tools and Nextflow installed.

Github Codespaces (https://github.com/features/codespaces) is an online developer environment that runs in your browser, complete with VSCode and a terminal.

multiqc

MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline quality control results can be visualised in the report and further statistics are available in the report data directory.

The pipeline will include the MultiQC module and will have special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info.

fastqc

FastQC is a tool which provides quality control checks on raw sequencing data. The pipeline will include the FastQC module.

modules

It is recommended to use this feature if you want to use modules and subworkflows in your pipeline. This will add all required files to use nf-core components or any compatible components from private repos by using nf-core modules and nf-core subworkflows commands.

changelog

Having a CHANGELOG.md file in the pipeline root directory is useful to track the changes added to each version.

You can read more information on the recommended format here: https://keepachangelog.com/en/1.0.0/

nf_schema

nf-schema is used to validate input parameters based on a JSON schema. It also provides helper functionality to create help messages, get a summary of changed parameters and validate and convert a samplesheet to a channel.

license

To protect the copyright of the pipeline, you can add a LICENSE file. This option ads the MIT License. You can read the conditions here: https://opensource.org/license/MIT

email

Enable the option of sending an email which will include pipeline execution reports on pipeline completion.

adaptivecard

This adds an Adaptive Card. A snippets of user interface. This Adaptive Card is used as a template for pipeline update messages and it is compatible with Microsoft Teams.

slackreport

This adds an JSON template used as a template for pipeline update messages in Slack.

documentation

This will add documentation markdown files where you can describe your pipeline. It includes:

• docs/README.md: A README file where you can describe the structure of your documentation.
• docs/output.md: A file where you can explain the output generated by the pipeline
• docs/usage.md: A file where you can explain the usage of the pipeline and its parameters.

These files come with an exemplary documentation structure written.

test_config

This will add two default testing profiles to run the pipeline with different inputs. You can customise them and add other test profiles.

These profiles can be used to run the pipeline with a minimal testing dataset with nextflow run <your_pipeline> -profile test.

The pipeline will include two profiles: test and test_full. In nf-core, we typically use the test profile to run the pipeline with a minimal dataset and the test_full to run the pipeline with a larger dataset that simulates a real-world scenario.

seqera_platform

When launching a pipeline with the Seqera Platform, a tower.yml file can be used to add configuration options.

In the pipeline template, this file is used to specify the output files of you pipeline which will be shown on the reports tab of Seqera Platform. You can extend this file adding any other desired configuration.

rocrate

RO-Crate is a metadata specification to describe research data and software. This will add a ro-crate-metadata.json file to describe the pipeline.

vscode

This will add a VSCode configuration file to render the admonitions in markdown files with the same style as the nf-core website.

Adds the .vscode directory to the pipelinerepository.

To run the pipeline creation silently (i.e. without the graphical interface), you can provide all the arguments. The required ones are --name, --description and --author.

If you have an existing pipeline and you wish to add/exclude one fo the features. Don’t use a template.yml file but modify the .nf-core.yml file from your pipeline and run nf-core pipelines sync to update the template.