How To Use Perl To Submit Mpiblast Jobs Via Qsub
3
1
Entering edit mode
10.0 years ago
Mchimich ▴ 320

Hello, i'm writing a perl script that use qsub (for mpiblast). My problem is that the perl code don't wait to qsub finish. Did someone have a idea about that problem ?

$command = ( "qsub mpiblast_$VAR.sh" ); system($command);# I would like to wait for qsub process and then use the output to parse the result Thans for your help ! perl clustering • 6.3k views ADD COMMENT 0 Entering edit mode Did you check the options to add dependency to the job with qsub? ADD REPLY 0 Entering edit mode system() just returns the status of the command (1 for success). You may want to capture the output using backticks, or using somthing like Capture::Tiny, and then do something with the output. By the way, you might get more responses on Stack Overflow since this is not really a bioinformatics question. This answer seems relevant if you have multiple jobs to run. ADD REPLY 2 Entering edit mode 10.0 years ago I don't think this is the right approach. The strength of a queing system is that you dispatch the jobs and let them run when the queing system allocates you resources. you don't want your perl script to wait for the job to finish. If you want to analyse the results of that job, you should prepare and run another job that DEPENDS on the previous one. Let's assume you have two jobs: blast and parse. You run the job blast with a particular name (on my system option -N) and then you run the parse job with option that waits for completion of the blast job (on my system option -hold_jid) something like qsub -N blastJob blast.q.sh qsub -hold_jid blastJob parse.q.sh you can, of course, put those options in the file itself ADD COMMENT 1 Entering edit mode just a short note: the system you are using is Sun Grid Engine. For other queue management software (e.g. Torque) the option may be different. ADD REPLY 0 Entering edit mode Thanks for yoyr reply ! I undertrand you Steano. But the parsing job is included in perl script (Bio perl) and it's just a small part of the script. ADD REPLY 1 Entering edit mode 10.0 years ago Fwip ▴ 490 Qsub doesn't block, does it? From my understanding, it submits the job and immediately returns, without waiting for the job to complete. This link: http://stackoverflow.com/questions/5982857/making-qsub-block-until-job-is-done mentions the following workaround: qsub -I -x myscript.sh If the script writes to standard out, you can capture the output by using backticks (   ) instead of the system() call. my$output = qsub -I -x myscript.sh

Edit: There is also the -sync y option, which from the qsub man page,

-sync y causes qsub to wait for the job to complete before exiting. If the job completes successfully, qsub's exit code will be that of the completed job. If the job fails to complete successfully, qsub will print out a error message indicating why the job failed and will have an exit code of 1.

Which does not seem to provide output capturing, but may be more suitable (especially due to the issue mentioned below by Jeremy).

2
Entering edit mode

beware your sysadmin might be unhappy with qsub -I since it launches an interactive session and basically locks out that node

0
Entering edit mode

Thanks, I didn't know this. It's been quite a while since I've worked with qsub.

1
Entering edit mode
10.0 years ago
Sirus ▴ 820

One we have implemented this problem but we did it in a different way. You can lunch your job as you did in you code and then create a function that roughly do as follow (I didn't test it , if it doesn't work just try to play arround with the code)

sub CheckQsub{

my $job_name = shift; my$stop = 1;

while($stop){$command =  "qstat";
my $status=qx[$command];
my @jobsStatus=split(/\n/,$status); my @targetJobStatus = grep(/$job_name/,@jobsStatus);  #Here we get only the status of the of interest

if(scalar(@targetJobStatus )== 0) #if no job is running
{
\$stop=0;
}
sleep(10);
}
}

Roughly this is the code as you can notice we just check if the action finished we don't know if it there was an error or a normal finish you can addition check points after calling this function. hope that helps.

1
Entering edit mode

Rather than checking qsub, I would produce and check a log or result file.

0
Entering edit mode

Yeah; it is a good idea also. But some times some programs don't generate log results and it is hard to tell if they are finished or not from the log file. I think one can combine the two strategies. First check "qsub" when no job is there check the log file to see if every thing worked well. Checking the result file is ok, but if it has a huge size maybe it will be that fast.

0
Entering edit mode

But your queue job could, as last command do something like: echo "Done" > myJob.log