Hi, I'm running a nextflow pipeline and I have a limit of jobs i can submit to the cluster, however nextflow submits jobs for different sections of a pipeline as soon as the required files are available. Is there a way to tell it to fits submit all the tasks for a section (e.g. aligning) without trying to run the next step right away?
Please read the question and answer what is asked. I don't need any notes on what i'm doing.
You can make each alignment process emit a value, then gather/concat this value from all alignment jobs into a new value/variable and let the downstream process read that new one as input. It's not actually used by the downstream process for any operation other than just serving as a serialization instance. The new processes won't start until all alignmrnt jobs are finished. Not aware of an in-built method.
this doesn't answer the question asked, nor is related in the least.
It answers the question exactly, and I would appreciate a different tone. Consider paying a consultant in case a free online community is not serving you appropriately.
hum... yes it is. My answer below is an illustration of what ATpoint said.