If I have a workflow which has a step that performs a scatter over
n samples, what would be the best way to spawn
n EC2 instances so that each instance is responsible for the computation of a single sample?
I realize that for local execution one can specify the
--parallel flag to launch
n threads and each thread will be responsible for the computation of one sample. However, how do I do this on a per-instance level in AWS?
I was looking into Toil and while I couldn't get much information about their parallelization capabilities, my understanding is that Toil, when used with the autoscaling feature, will distribute the workload of a scatter step over
n instances. Is my understanding correct? If not, then what options do I have?