snakemake job status monitoring
2
0
Entering edit mode
9 weeks ago
Eugene A ▴ 120

Hi everyone, I'm writing the simple linear snakemake pipeline on multiple samples.

I'm runnign snakemake with -k parameters, so failing in one branch does not cause to fail whole pipeline. But I'd like also to have a meaningfull info (possibly via emails) on the pipeline status (which joabs are done, and which are failed). There is a --log-handler-script parameter which supposetly should give me necessary functioonality. BUT, the msg's produced by sankemake rather uninformative:

as one might see the one of the job failed but msg (i just printed it to terminal from the def log_handler(msg):) does not tell me which step went wrong. And I do not see how I can get the jobID of failed job before I the pipeline is over. Any suggestions are appreciated

Best, Eugene

snakemake pipelines • 321 views
0
Entering edit mode

if something crashed it is usually pretty easy to figure out what happened but if a task exited with a normal status code but a file is missing then yes it won't be super helpful

0
Entering edit mode

It is pritty easy to figure out what happend in manual examination. My task was to automatically detect failed rules and report them (via emails or something like that), while the rest of a pipeline goes on (-k option). And _automatic_ detection of failed jobs is surprisengly inconvenientin snakemake

0
Entering edit mode

This allows you to see errors more specifically linked to the jobs submitted.

1
Entering edit mode
9 weeks ago
Eugene A ▴ 120

after some more search I've figured that there is other option for monitoring appart from --log-handler-script, namely panoptes server (https://github.com/panoptes-organization/panoptes, https://snakemake.readthedocs.io/en/stable/executing/monitoring.html). It also has some problems (failed jobs currently displayed as running, so does not solve initial issue), but will suite my purpuses for now.

1
Entering edit mode
5 weeks ago
Eugene A ▴ 120

Some additions, which might be helpfull for someone - it seems that the only way to detect failed jobs in snakemake and automatically keep track of them is to intercept exit codes of the used commands and processe it separetly