nf-core/datasync

Edit

Introduction

[WIP] WORK IN PROGRESS AND NOT YET STABLE - DO NOT USE FOR PRODUCTIVE SETTINGS YET

nf-core/datasync is a system operation pipeline that provides several workflows for handling system operation / automation tasks that are commonly helpful for various tasks in large data processing / analysis facilities. This includes:

• Data Synchronization & Checksum generation
  • Configurable: Can provide YAML file which files to include or exclude from sync
  • Checksum backend: Can configure which backend to use for checksum generation (e.g. sha256sum, md5, …)
  • Configurable whether to include (sub-) folders in the sync or not (search for checkpoint files, e.g. has to have DEMUX_DONE that signals a demultiplexing run was finished & successfully copied)
• Data Integrity validation
  • Provided with a directory to check, can validate that file(s) found are matching checksums from Synchronization subworkflow
• Data Archival & Deletion
  • Can check source and target location for existence of file(s) and decide based on user configurable rules whether files can be considered archived
    • Timestamp older than X days
    • Checksums match Integrity validation report
    • Create empty files to make it obvious that archival was performed
    • Optionally: Delete files or create list of files to be deleted for manual deletion process

The pipeline can be configured by users to execute any of the aforementioned subworkflows and then produces a report using MultiQC custom content that also serves as a report of what was done by the pipeline for documentation purposes.

Usage

Now, you can run the pipeline using:

nextflow run nf-core/datasync \
   -profile <docker/singularity/.../institute> \
   --input samplesheet.csv \
   --outdir <OUTDIR>
   --sync
   --sync_backend 'sha256'
   --sync_done true #Creates SYNC_DONE file when done in each folder

see docs.

For more details and further functionality, please refer to the usage documentation and the parameter documentation.

Pipeline output

To see the results of an example test run with a full size dataset refer to the results tab on the nf-core website pipeline page. For more details about the output files and reports, please refer to the output documentation.

Credits

nf-core/datasync was originally written by Alexander Peltzer.

We thank the following people for their extensive assistance in the development of this pipeline:

Contributions and Support

If you would like to contribute to this pipeline, please see the contributing guidelines.

For further information or help, don’t hesitate to get in touch on the Slack #datasync channel (you can join with this invite).

Citations

An extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.

You can cite the nf-core publication as follows:

The nf-core framework for community-curated bioinformatics pipelines.

Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.

Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.