best way to submit hpc job using snakemake
1
0
Entering edit mode
4 months ago
QX ▴ 60

Hi all,

I try to create a snakemake workflow for HPC. I know that snakemake have options for cluster ifself, but can we just put the shell: sbatch --optios in the rules, which is alot easier?

Do you know what is the pros and cons for the 2 approaches?

Best,

snakemake • 530 views
ADD COMMENT
1
Entering edit mode

You should actually learn to do it properly as suggested here and in the docs: https://stackoverflow.com/questions/76166705/how-do-i-set-up-snakemake-on-slurm-properly

I don't think using sbatch or srun in the command is a viable or sensible solution.

ADD REPLY
2
Entering edit mode
4 months ago

If you just put sbatch in the rules, then snakemake will run the command, which will report success almost immediately as sbatch reports that the job is successfully submitted (not successfully completed). Snakemake will them move immediately on to the next rule without waiting for the jobs from the previous one to finish, and will be upset when the files that were the output of rule 1, and the input to rule 2, don't yet exist.

ADD COMMENT
0
Entering edit mode

if output1 from rule1 is the input2 for rule2; is rule2 will while until rule1 output1 finished, or it will execute immediately after input1 is generated (which then could be a problem as they are not whole file)?

also, for analysis with many files, is it better to use snakemake for each file through whole process; or it is better to run all file at the same time for each step?

ADD REPLY
2
Entering edit mode

If run properly snakemake will wait for rule1 output1 is finished, as signaled by SLURM returning a successful completion code.

If you run sbatch in a rule, then rule2 might execute before rule1 has even started, let alone finihsed.

is it better to use snakemake for each file through whole process; or it is better to run all file at the same time for each step?

The purpose of snake make is not to worry about this sort of thing. You tell it you need this 10 or 20 steps done to that 10 or 20 files and it works it out for itself, and you come back to all files having been run for all steps.

ADD REPLY
0
Entering edit mode

thank you!

ADD REPLY

Login before adding your answer.

Traffic: 1562 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6