Question: CWL limit number of job instances created through scatter
gravatar for silverspanch
12 months ago by
silverspanch10 wrote:

Is there any way to limit the number of job instances created when scattering a workflow? Right now I have a highly parallelizable workflow that spans into more then 1200 jobs when running in parallel, which is way to many for a single server to handle at once, and I end up getting a bunch of MAX_THREAD errors (each job takes 12 threads, the server maxes out at 4096 threads/user). Unless Im mistaken the reference cwltool doesnt have the ability to span jobs over a cluster or submit to a job queue, does it have any ability to limit the number of concurrent jobs? Ideally I would like it to behave like a FIFO job pool, but any resource management would be useful.

ADD COMMENTlink modified 12 months ago by Michael R. Crusoe1.8k • written 12 months ago by silverspanch10
gravatar for Michael R. Crusoe
12 months ago by
Common Workflow Language project
Michael R. Crusoe1.8k wrote:

Hello silverspanch.

You are correct, cwltool is just a reference runner and not production-hardened code. However recent versions should wait to launch jobs based upon basic resource accounting.

If each instance of one of your tools is using 12 cores, make sure to notate that with the following syntax:

     coresMin: 12

See for the reference

You may also want to consider one of the systems that supports CWL listed at

For example, the toil-cwl-runner can submit jobs to a grid cluster scheduler, or for a full workflow platform experience you may need Arvados.

ADD COMMENTlink written 12 months ago by Michael R. Crusoe1.8k
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1015 users visited in the last hour