Question: Naive BASH question on top to bottom programming regarding my alignment script
0
gravatar for datascientist28
6 months ago by
University of Washington
datascientist28350 wrote:

I'm confused as to why the two bowtie lines are running at the same time? I was under the impression that everything in unix is run top to bottom and the second bowtie line wouldn't start running until the first is completed. The reason I ask is these commands are running simultaneously and I'm having memory issues of sorting two large bam files at the same time.. I'm assuming it has something to do with the piping but I am confused. Script below

(I have changed the sample names per privacy)

!/bin/bash
echo BOWTIE LOG > ../BAM/bowtie_log.txt
mkdir ../BAM/
bowtie2 --very-sensitive -x /PATH/TO/genome -X 2000 2>> ../BAM/bowtie_log.txt -p 1 -1 trimmed/Sample1_R1_001_val_1.fq.gz -2 trimmed/Sample1_R2_001_val_2.fq.gz | samtools view -Sb - | samtools sort - -o ../BAM/Sample1.bam
bowtie2 --very-sensitive -x /PATH/TO/genome -X 2000 2>> ../BAM/bowtie_log.txt -p 1 -1 trimmed/Sample2_R1_001_val_1.fq.gz -2 trimmed/Sample2_R2_001_val_2.fq.gz | samtools view -Sb - | samtools sort - -o ../BAM/Sample2.bam
bash unix • 402 views
ADD COMMENTlink modified 6 months ago • written 6 months ago by datascientist28350
1

(Edited my original comment based on @Jean-Karim's comment and some other searches): It may be the pipes in the command that are backgrounding something and allowing both jobs to run. There are no logical/control operators.

Adding { } around the entire command (as suggested by @steve below) may prevent that backgrounding from happening (if it works, my assumption).

(see this, specifically A.2 logical operators).

ADD REPLYlink modified 6 months ago • written 6 months ago by genomax54k

Wow learn something new about bash everyday. Thanks for the input

ADD REPLYlink written 6 months ago by datascientist28350

I don't follow this. This has to be more subtle than that. The newline is a terminator and the shell should wait for the command to complete before continuing on the next one.

I think the OP does not have a new line at all and the text wraps around and shows as if there were a newline.

ADD REPLYlink modified 6 months ago • written 6 months ago by Istvan Albert ♦♦ 77k

I added a comment to Jean's answer? Does this bg have this effect?

ADD REPLYlink written 6 months ago by datascientist28350

does the same thing happen if you write it as { bowtie2 .... | samtools ... ; } && { bowtie2 .... | samtools ... ; } ?

ADD REPLYlink modified 6 months ago • written 6 months ago by steve1.6k

What is making you think that both are running concurrently?

ADD REPLYlink written 6 months ago by jrj.healey6.0k

Reason I thought that was because Sample2 tmp merge files were being made before Sample1 was merged and got a memory error

ADD REPLYlink modified 6 months ago • written 6 months ago by datascientist28350
2
gravatar for Jean-Karim Heriche
6 months ago by
EMBL Heidelberg, Germany
Jean-Karim Heriche16k wrote:

Unless told otherwise (e.g. with &), bash processes lines sequentially waiting for the first to finish before starting the next one. In the case here, if the two bowtie2 commands run concurrently, something in the first line must have forked/backgrounded itself.

ADD COMMENTlink written 6 months ago by Jean-Karim Heriche16k

Oh this is a good point. I did start the command and then put it in the background by control-z and bg, would that have an effect??

ADD REPLYlink modified 6 months ago • written 6 months ago by datascientist28350
1

ok, case closed. with that you've made the programs run in parallel - I also moved the comment to answer.

ADD REPLYlink modified 6 months ago • written 6 months ago by Istvan Albert ♦♦ 77k

Got it. Sorry for the confusion and not saying everything in my original post

ADD REPLYlink modified 6 months ago • written 6 months ago by datascientist28350
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1628 users visited in the last hour