For loop to parallel run commands
3
0
Entering edit mode
14 days ago
endretoth ▴ 20

After reading two articles/post (I put the links below) I decided to parallelize my commands to be more effective. I would like use a For loop to run multiple commands parallel.

FIRST: For example, I create a file, basically a list of multiple commands that I want to run parallel:

> list
echo $'echo 123\necho 456' > list

so I have this file, in which the "echo 123..." is the one that I will replace with my future command, which is more complex, this is just for test. (maybe I will put here BLAST, BWA mapping or STRUCTURE analysis commands...etc. doesn't matter at the moment)

SECOND: I create a bash script which will run them parallel:

#!/bin/bash
for task in "$list"; do {
  $task &
} done

However, this bash script doesn't work, I don't clearly understand how and why should work.

Please look at these articles/posts where I found this:

https://unix.stackexchange.com/questions/103920/parallelize-a-bash-for-loop (look at the comment with 258 vote) https://www.slashroot.in/how-run-multiple-commands-parallel-linux (look at the fourth screenshot)

I would like to kindly request help how to design this, since I'm not an expert in bash scripting. (I often face the situation that I have multiple processors/cores and commands run sequentially, one after the other. I see this by hitting the htop command)

Unix bash Linux • 137 views
ADD COMMENT
1
Entering edit mode
13 days ago
endretoth ▴ 20

Dear All,

Thank you for your kind help. I managed to create a perfectly functioning bash script. Here it is :) (maybe others will use it)

#!/bin/bash
#reading the file into an array
readarray -t my_array < list
#process each lines
for line in "${my_array[@]}"; do
  $line
done

123 456

or alternatively,

#!/bin/bash
readarray -t my_array < list
for line in "${my_array[@]}"; do
  $line
  rc=$?
  if [[ 0 != $rc ]]; then echo Failed command: ${line}; break; fi
done

I know about parallel and Xargs, however I wanted to do it a different way, with simple programing, and without using any program. It was quite a challenge. :)

ADD COMMENT
2
Entering edit mode
13 days ago

a few things are going on here:

  • you do not need to use the { (curly brackets) in that for loop, correct syntax should be:

    for task in "$list"; do $task &; done

  • secondly you need to 'access' your list as follows: for task in $( cat list); do you need to list the items from that list file not only that file

  • lastly: doing a for loop with items that have spaces in them is notriously difficult (not impossible but certainly not straightforward) , that is because 'for' will spit the items from list on each whitespace (regardless of on same line or not. SO in your case you will get echo / 123 / echo / 456 in your loop.

if you want to parallelize your commands you can have a look at the command parallel perhaps even in combination with Xargs

ADD COMMENT

Login before adding your answer.

Traffic: 2574 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6