Run nextflow sections sequentially
1
0
Entering edit mode
18 months ago
Raygozak ★ 1.4k

Hi, I'm running a nextflow pipeline and I have a limit of jobs i can submit to the cluster, however nextflow submits jobs for different sections of a pipeline as soon as the required files are available. Is there a way to tell it to fits submit all the tasks for a section (e.g. aligning) without trying to run the next step right away?

Thanks

Please read the question and answer what is asked. I don't need any notes on what i'm doing.

nextflow • 1.8k views
ADD COMMENT
2
Entering edit mode

You can make each alignment process emit a value, then gather/concat this value from all alignment jobs into a new value/variable and let the downstream process read that new one as input. It's not actually used by the downstream process for any operation other than just serving as a serialization instance. The new processes won't start until all alignmrnt jobs are finished. Not aware of an in-built method.

ADD REPLY
0
Entering edit mode

this doesn't answer the question asked, nor is related in the least.

ADD REPLY
6
Entering edit mode

It answers the question exactly, and I would appreciate a different tone. Consider paying a consultant in case a free online community is not serving you appropriately.

ADD REPLY
1
Entering edit mode

this doesn't answer the question asked, nor is related in the least.

hum... yes it is. My answer below is an illustration of what ATpoint said.

ADD REPLY
0
Entering edit mode

Why did you delete this post, Raygozak?

ADD REPLY
2
Entering edit mode
18 months ago
workflow {
    mulitple_ch = ....
    step_ch = MANY(mulitple_ch)
    WAIT_FOR_MANY(step_ch.output.collect())
}
ADD COMMENT
0
Entering edit mode

Does the word running has a different meaning of what i'm thinking?

ADD REPLY
6
Entering edit mode

@Raygozak Please be nice to the people who have provided you with answers and have some humility. I understand exactly what they mean, and have learned something here and or already use this pattern in my pipelines. If you do not, then it is not their fault, but yours.

ADD REPLY

Login before adding your answer.

Traffic: 1655 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6