Save Output Into Text File, Whose Name Is A Variable
2
0
Entering edit mode
11.6 years ago
Bioscientist ★ 1.7k

Hi I'm using SUN Grid Engine to submit jobs, written by shell script like this:

#!/bin/bash

#$ -pe single 24
#$ -V
#$ -cwd
#$ -o $HOME/sge_jobs_output/sge_job.$JOB_ID.out -j y
#$ -S /bin/bash
#$ -l mem_free=8G

cd $HOME/scratch/bwa2/

job_number=$SGE_TASK_ID
SAM_NAME_1=`head -"$job_number" master_list_1|tail -1`
/share/bin/bwa aln hg19_index "$SAM_NAME_1".recal.fastq.gz > "$SAM_NAME_1".sai

 #Do sth. here to save the output as text file. Here by output I don't mean
 # .sai file, but the error report like "10000000 sequences have been processed",
 #And text file name is SAM_NAME_1, which is a variable

Maybe I didn't make quite clear. As shown as the end of the script, I want to put some command so that the error report can be saved as .txt file. How can I do that? (I know we can use > .txt at prompt; but here the .txt file name is a variable, so I must put command within script)

WHat I want is similar to $HOME/sgejobsoutput/sgejob.$JOBID.out -j y; but I want name as SAMNAME instead of JOBID. Thanks!

SO actually this turned out as a bioinformatic problem. :)

Edit:

I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..$SAMNAME1 part doesn't act as variable..

Is there any other way, which does NOT rely on this SGE flag? thx

output • 6.2k views
ADD COMMENT
1
Entering edit mode

I think this isn't really bioinformatics related. but you likely want to use ${SAM_NAME_1}.recal.fastq.gz and ${SAM_NAME_1}.sai

ADD REPLY
0
Entering edit mode

Yeah, I know it's NOT bioinformatics-related; I post on stack overflow, but got no reply. So how should I do to save output as text file here? thanks.

ADD REPLY
0
Entering edit mode

Here's a fishing rod: the Advanced Bash-Scripting Guide http://www.tldp.org/LDP/abs/html/ . See Chapter 4. Introduction to Variables and Parameters.

ADD REPLY
3
Entering edit mode
11.6 years ago
Mitch Bekritsky ★ 1.3k

The way to redirect STDERR to a file in SGE is by using the -e flag. The trick for your program is to substitute the SAM file name for the job name in the argument for -e. For you, all you should have to add is

 #! -e "${HOME}/sge_jobs_output/sge_job.${SAM_NAME}.txt"

I haven't tested this out, but I know for certain that redirecting error output from SGE is with -e. Naming the file should be the easy part (I hope...).

Hope that's useful!

ADD COMMENT
0
Entering edit mode

Obviously !! I did not used sge for a long time and forgot this flag ! +1

ADD REPLY
0
Entering edit mode

hi, seems it still doesn't work. It's showing 'sge_job.$SAM_NAME.txt', seems SAM_NAME doesn't work as variable...

ADD REPLY
0
Entering edit mode

I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..$SAM_NAME_1 part doesn't act as variable..

ADD REPLY
0
Entering edit mode

When I set error output using the -e command, I set the filename at the command line (e.g. qsub ... -e [error file name] ...), and I know it will work there, not sure if it will work in the script, but I think it should. Do you know if $SAM_NAME_1 has the the value you want it to? That would be the first place I would check. Unfortunately, I'm away from my grid right now so I can't check it for you. Glad to help you trouble shoot though!

ADD REPLY
0
Entering edit mode

hi, if I do in the command line as: "qsub XXX.sh -e $SAM_NAME_1" ,but here SAM_NAME_1 is a variable. Does that work?

ADD REPLY
0
Entering edit mode

Hi bioscientist, that should work (it always does for me). Although be careful to name it right. In your case, it should be qsub XXX.sh -e $sge_job.{SAM_NAME_1}.txt. You probably had that in mind, but just in case...

ADD REPLY
0
Entering edit mode

Hi bioscientist, that should work (it always does for me). Although be careful to name it right. In your case, it should be qsub XXX.sh -e "$sge_job.{SAM_NAME_1}.txt". You probably had that in mind, but just in case...

ADD REPLY
1
Entering edit mode
11.6 years ago

As brentp notes, you just need to change "$SAM_NAME_1" to ${SAM_NAME_1}. You'll also need to continue the script with:

bwa samse ${SAM_NAME_1} ${SAM_NAME_1}.sai ${SAM_NAME_1}.recal.fastq.gz

I didn't test this, but you should get the idea. The important thing when using a batch system such as SGE is to make sure that your script runs outside of the batch system before trying to use the batch system.

ADD COMMENT
1
Entering edit mode

Yes. You'll need to specify the name directly in the shell script. There is no variable interpolation in the SGE directives.

ADD REPLY
0
Entering edit mode

HI Sean, sorry I don't make clear what I want to do. I don't mean to make .sai file as .txt; but I want the error report like "how many sequences have been processed" saved as .txt. How can I do that?

ADD REPLY
0
Entering edit mode

See the -e flag for SGE.

ADD REPLY
0
Entering edit mode

I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..$SAM_NAME_1 part doesn't act as variable...

ADD REPLY
0
Entering edit mode

Yeah, but how can I redirect the error output in the shell script? ANy clues?

ADD REPLY

Login before adding your answer.

Traffic: 1986 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6