Save Output Into Text File, Whose Name Is A Variable
2
0
Entering edit mode
11.6 years ago
Bioscientist ★ 1.7k

Hi I'm using SUN Grid Engine to submit jobs, written by shell script like this:

#!/bin/bash

#$-pe single 24 #$ -V
#$-cwd #$ -o $HOME/sge_jobs_output/sge_job.$JOB_ID.out -j y
#$-S /bin/bash #$ -l mem_free=8G

cd $HOME/scratch/bwa2/ job_number=$SGE_TASK_ID
SAM_NAME_1=head -"$job_number" master_list_1|tail -1 /share/bin/bwa aln hg19_index "$SAM_NAME_1".recal.fastq.gz > "$SAM_NAME_1".sai #Do sth. here to save the output as text file. Here by output I don't mean # .sai file, but the error report like "10000000 sequences have been processed", #And text file name is SAM_NAME_1, which is a variable  Maybe I didn't make quite clear. As shown as the end of the script, I want to put some command so that the error report can be saved as .txt file. How can I do that? (I know we can use > .txt at prompt; but here the .txt file name is a variable, so I must put command within script) WHat I want is similar to$HOME/sgejobsoutput/sgejob.$JOBID.out -j y; but I want name as SAMNAME instead of JOBID. Thanks! SO actually this turned out as a bioinformatic problem. :) Edit: I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..$SAMNAME1 part doesn't act as variable.. Is there any other way, which does NOT rely on this SGE flag? thx output • 6.2k views ADD COMMENT 1 Entering edit mode I think this isn't really bioinformatics related. but you likely want to use${SAM_NAME_1}.recal.fastq.gz and ${SAM_NAME_1}.sai ADD REPLY 0 Entering edit mode Yeah, I know it's NOT bioinformatics-related; I post on stack overflow, but got no reply. So how should I do to save output as text file here? thanks. ADD REPLY 0 Entering edit mode Here's a fishing rod: the Advanced Bash-Scripting Guide http://www.tldp.org/LDP/abs/html/ . See Chapter 4. Introduction to Variables and Parameters. ADD REPLY 3 Entering edit mode 11.6 years ago Mitch Bekritsky ★ 1.3k The way to redirect STDERR to a file in SGE is by using the -e flag. The trick for your program is to substitute the SAM file name for the job name in the argument for -e. For you, all you should have to add is  #! -e "${HOME}/sge_jobs_output/sge_job.${SAM_NAME}.txt"  I haven't tested this out, but I know for certain that redirecting error output from SGE is with -e. Naming the file should be the easy part (I hope...). Hope that's useful! ADD COMMENT 0 Entering edit mode Obviously !! I did not used sge for a long time and forgot this flag ! +1 ADD REPLY 0 Entering edit mode hi, seems it still doesn't work. It's showing 'sge_job.$SAM_NAME.txt', seems SAM_NAME doesn't work as variable...

0
Entering edit mode

I tried: #$-e$HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..$SAM_NAME_1 part doesn't act as variable..

0
Entering edit mode

When I set error output using the -e command, I set the filename at the command line (e.g. qsub ... -e [error file name] ...), and I know it will work there, not sure if it will work in the script, but I think it should. Do you know if $SAM_NAME_1 has the the value you want it to? That would be the first place I would check. Unfortunately, I'm away from my grid right now so I can't check it for you. Glad to help you trouble shoot though! ADD REPLY 0 Entering edit mode hi, if I do in the command line as: "qsub XXX.sh -e$SAM_NAME_1" ,but here SAM_NAME_1 is a variable. Does that work?

0
Entering edit mode

Hi bioscientist, that should work (it always does for me). Although be careful to name it right. In your case, it should be qsub XXX.sh -e $sge_job.{SAM_NAME_1}.txt. You probably had that in mind, but just in case... ADD REPLY 0 Entering edit mode Hi bioscientist, that should work (it always does for me). Although be careful to name it right. In your case, it should be qsub XXX.sh -e "$sge_job.{SAM_NAME_1}.txt". You probably had that in mind, but just in case...

1
Entering edit mode
11.6 years ago

As brentp notes, you just need to change "$SAM_NAME_1" to${SAM_NAME_1}. You'll also need to continue the script with:

bwa samse ${SAM_NAME_1}${SAM_NAME_1}.sai ${SAM_NAME_1}.recal.fastq.gz  I didn't test this, but you should get the idea. The important thing when using a batch system such as SGE is to make sure that your script runs outside of the batch system before trying to use the batch system. ADD COMMENT 1 Entering edit mode Yes. You'll need to specify the name directly in the shell script. There is no variable interpolation in the SGE directives. ADD REPLY 0 Entering edit mode HI Sean, sorry I don't make clear what I want to do. I don't mean to make .sai file as .txt; but I want the error report like "how many sequences have been processed" saved as .txt. How can I do that? ADD REPLY 0 Entering edit mode See the -e flag for SGE. ADD REPLY 0 Entering edit mode I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..\$SAM_NAME_1 part doesn't act as variable...

0
Entering edit mode

Yeah, but how can I redirect the error output in the shell script? ANy clues?