pipeline / workflow language

Question

[Bogdan]

Dear all,

I would like to have your suggestion/opinions please on which workflow/pipeline description language someone can easily use on a SLURM/SGE cluster, as well as on Amazon cloud computing platform ...

some of the pipeline frameworks are listed here : https://academic.oup.com/bib/article/18/3/530/2562749

many thanks,

-- bogdan

Answer 1

[Devon Ryan]

In general you can use any workflow management system on any cluster/cloud platform, they're pretty much all the same in that regard. Much of the community uses Snakemake, which integrates nicely with things like conda. We use that ourselves and are quite happy with it.

[Bogdan]

Thank you Devon. In addition, if I may ask please : was Snakemake a good choice also for assembling/running most of GATK algorithms ?

[Devon Ryan]

We don't use GATK much, but yes, it works just fine for that (honestly, they're all about the same, it's mostly a matter of how you prefer to structure things). If you're primarily interested in GATK, though, you should consider bcbio-nextgen. Brad Chapman has put a good bit of work into tuning bcbio to make variant calling a bit faster (have a look at his blog, which is generally useful if you're doing variant calling).

[Bogdan]

Thank you again Devon. It is a good recommendation, I did not know which framework is more mature -- Snakemake, a more recent framework WDL (https://software.broadinstitute.org/wdl/) or any other one ?

[Devon Ryan]

Snakemake is quite mature. I don't know anyone actually using wdl, so I can't comment on it.

[Bogdan]

Thanks a lot, Devon. Then ... Snakemake would be a good place to start.