For loop to parallel run commands
3
0
Entering edit mode
14 months ago
endretoth ▴ 30

After reading two articles/post (I put the links below) I decided to parallelize my commands to be more effective. I would like use a For loop to run multiple commands parallel.

FIRST: For example, I create a file, basically a list of multiple commands that I want to run parallel:

> list
echo $'echo 123\necho 456' > list  so I have this file, in which the "echo 123..." is the one that I will replace with my future command, which is more complex, this is just for test. (maybe I will put here BLAST, BWA mapping or STRUCTURE analysis commands...etc. doesn't matter at the moment) SECOND: I create a bash script which will run them parallel: #!/bin/bash for task in "$list"; do {
$task & } done  However, this bash script doesn't work, I don't clearly understand how and why should work. Please look at these articles/posts where I found this: https://unix.stackexchange.com/questions/103920/parallelize-a-bash-for-loop (look at the comment with 258 vote) https://www.slashroot.in/how-run-multiple-commands-parallel-linux (look at the fourth screenshot) I would like to kindly request help how to design this, since I'm not an expert in bash scripting. (I often face the situation that I have multiple processors/cores and commands run sequentially, one after the other. I see this by hitting the htop command) Unix bash Linux • 724 views ADD COMMENT 1 Entering edit mode 14 months ago endretoth ▴ 30 Dear All, Thank you for your kind help. I managed to create a perfectly functioning bash script. Here it is :) (maybe others will use it) #!/bin/bash #reading the file into an array readarray -t my_array < list #process each lines for line in "${my_array[@]}"; do
$line done  123 456 or alternatively, #!/bin/bash readarray -t my_array < list for line in "${my_array[@]}"; do
$line rc=$?
if [[ 0 != $rc ]]; then echo Failed command:${line}; break; fi
done


I know about parallel and Xargs, however I wanted to do it a different way, with simple programing, and without using any program. It was quite a challenge. :)

2
Entering edit mode
14 months ago

a few things are going on here:

• you do not need to use the { (curly brackets) in that for loop, correct syntax should be:

for task in "$list"; do$task &; done

• secondly you need to 'access' your list as follows: for task in \$( cat list); do you need to list the items from that list file not only that file

• lastly: doing a for loop with items that have spaces in them is notriously difficult (not impossible but certainly not straightforward) , that is because 'for' will spit the items from list on each whitespace (regardless of on same line or not. SO in your case you will get echo / 123 / echo / 456 in your loop.

if you want to parallelize your commands you can have a look at the command parallel perhaps even in combination with Xargs