snakemake job status monitoring
2
0
Entering edit mode
4 months ago
Eugene A ▴ 130

Hi everyone, I'm writing the simple linear snakemake pipeline on multiple samples.

I'm runnign snakemake with -k parameters, so failing in one branch does not cause to fail whole pipeline. But I'd like also to have a meaningfull info (possibly via emails) on the pipeline status (which joabs are done, and which are failed). There is a --log-handler-script parameter which supposetly should give me necessary functioonality. BUT, the msg's produced by sankemake rather uninformative:

as one might see the one of the job failed but msg (i just printed it to terminal from the def log_handler(msg):) does not tell me which step went wrong. And I do not see how I can get the jobID of failed job before I the pipeline is over. Any suggestions are appreciated

Best, Eugene

snakemake pipelines • 403 views
ADD COMMENT
0
Entering edit mode

if something crashed it is usually pretty easy to figure out what happened but if a task exited with a normal status code but a file is missing then yes it won't be super helpful

ADD REPLY
0
Entering edit mode

It is pritty easy to figure out what happend in manual examination. My task was to automatically detect failed rules and report them (via emails or something like that), while the rest of a pipeline goes on (-k option). And _automatic_ detection of failed jobs is surprisengly inconvenientin snakemake

ADD REPLY
0
Entering edit mode

I assume this something you may have already added, but to be sure, do you have the log directives as part of your rules? https://snakemake.readthedocs.io/en/stable/tutorial/advanced.html?highlight=log%20directive#step-5-logging

This allows you to see errors more specifically linked to the jobs submitted.

As for emails I find the following helpful: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html?highlight=onstart#onstart-onsuccess-and-onerror-handlers

ADD REPLY
1
Entering edit mode
3 months ago
Eugene A ▴ 130

after some more search I've figured that there is other option for monitoring appart from --log-handler-script, namely panoptes server (https://github.com/panoptes-organization/panoptes, https://snakemake.readthedocs.io/en/stable/executing/monitoring.html). It also has some problems (failed jobs currently displayed as running, so does not solve initial issue), but will suite my purpuses for now.

ADD COMMENT
1
Entering edit mode
3 months ago
Eugene A ▴ 130

Some additions, which might be helpfull for someone - it seems that the only way to detect failed jobs in snakemake and automatically keep track of them is to intercept exit codes of the used commands and processe it separetly

ADD COMMENT

Login before adding your answer.

Traffic: 1244 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6