reduce speed terminal command submission

Question

[rank]

Dear All,

I am mapping Illumina reads from fq files of relatively small size (70MB) to mitochondrial genomes. I already filtered these reads using the perl script included with Novoplasty, and I want to map them to the c.a. 12 bp of the mitochondrion outside the control region. I have 200 samples. So each one gets mapped to make the sam file. I am performing this work on a MacBook Pro, running Catalina.

Here is a sample of my commands:

bwa mem ESRC_83_aug2020_noControl.fa BCPL_117.mtgenome_1.fq BCPL_117.mtgenome_2.fq > BCPL_117_mito_NoControl.sam 
bwa mem ESRC_83_aug2020_noControl.fa BCPL_131.mtgenome_1.fq BCPL_131.mtgenome_2.fq > BCPL_131_mito_NoControl.sam 
bwa mem ESRC_83_aug2020_noControl.fa BCPL_133.mtgenome_1.fq BCPL_133.mtgenome_2.fq > BCPL_133_mito_NoControl.sam

I can only get this to run properly with about 10 samples at a time, because terminal submits new commands before the old ones have finished. I believe there is some way to slow down the scrolling of the commands, but can't find anything about this problem or possible solutions by searching anywhere.

I thought about inserting a ls command after each line, so that it performs some minor task before starting on the new mapping command. But that seems dumb to me...any other suggestions would be greatly appreciated.

Answer 1

[Mensur Dlakic]

I can only get this to run properly with about 10 samples at a time, because terminal submits new commands before the old ones have finished.

This is not how things work. Unless you put an ampersand sign (&) after each command, they will be executed one at a time, even if multiple lines flash on the screen at once.

Generally speaking, multiple commands are ran from a script. If you copy all of them into a text file, say something called myscript.sh, you can run it like this:

source myscript.sh &

Alternatively, you can make a script executable, and run it like any other shell script:

chmod +x myscript.sh
./myscript.sh &

Either way, commands will run one at a time.

[Ram]

I think OP is copy-pasting commands and running into clipboard based limitations. @rank, Like Mensur says, save your commands as a script file, then execute it so log messages get written to a file and not to your terminal.

chmod +x myscript.sh
./myscript.sh 2>mylog.txt &

And then you can view the log at your leisure using head or tail or more or less. You can even follow the log as it gets generated using tail -f mylog.txt and if you wish to stop following, hit Ctrl+C.

[Mensur Dlakic]

I am not sure why this has turned into a discussion about output redirection given that the OP already has the output of his commands redirected into appropriate files. Bur since we are doing it, it is worth mentioning that not everyone uses bash (for which the examples were given), just like not everyone uses (t)csh (yours truly would be one of those old-timers). For c-shell users, a good redirection summary is given here.

[Ram]

Mensur,

I was going off of OP's comment saying "I believe there is some way to slow down the scrolling of the commands", which made me think that they're screen is probably filling up with bwa progress messages. I have no idea what vaibhash's comment is supposed to contribute, it seems to be going off on a tangent for no reason.