Results of two sequential bash pipes into a variable instead of a file?
0
0
Entering edit mode
2.0 years ago

Hi!

I would like to put the result of a pipe into a variable instead of a file. But it does not work as intended.

The well working script with the pipe resulting in a file is like that:

for sample in *.csv;
do
pr -m -t -s\  $sample  Ldha.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g'| cat >Ldha/${input}.tsv
done

I tried to funnel it into a variable called "input" with these ways:

read input <(pr -m -t -s\  $sample  Ets1.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g')
input=`pr -m -t -s\  $sample  Ldha.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g'`
input=$(pr -m -t -s\  $sample  Ets1.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g')
pr -m -t -s\  $sample  Ldha.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g' | read input

Nothing worked. If i run it with saving to files, it works. But saving well 50 000 files for each run... In the end, it should look like that (using a perl script for pearson correlations from the web):

for sample in *.csv;
do
input=$(pr -m -t -s\  $sample  Ets1.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g')
perl corr.pl -i $input  -col 1 2 | cat >>correl_Ets1.csv
done

But i fail to get the output from the pipe into a variable. Any ideas?

Best wishes,

Norman

bash pipe variable shell script • 532 views
ADD COMMENT
2
Entering edit mode

Some comments:

  • You should provide small example files to allow us reproduce the issue
  • What "nothing works" means? What is stored in $input with each command? Did you check with echo "$input"?
  • It should be awk, not Awk

The following simple example shows no problems assigning piped stdout to a bash variable:

$ awk '{print $2"\t"$13}' relAbun.tsv | head
U2  U13
0   0
0.1 0
0   0
0   0
0   0
0.3 0.2
0   0
0   0
0   0

Now put the result in a variable:

$ input=$(cat relative_abundance_raw.tsv | awk '{print $2"\t"$13}' | head}' relAbun.tsv | head)
$ echo "$input"
U2  U13
0   0
0.1 0
0   0
0   0
0   0
0.3 0.2
0   0
0   0
0   0
ADD REPLY
0
Entering edit mode

Dear h.mon,

thank you very much for your reply and the brillant idea with echo $input. Echo input looks good. But now what happens is the following: The perl script is taking the first value of the first file as a filename. Then it takes the first value from the second file as a filename. So, the input is transferred indeed to the variable input. But the filename, which would be taken from $sample is now lost and not fed to the perl script. I am not sure if this can be fixed.... Maybe i need to feed this as an array? But still, how would i pass the filename....

Best regards and thank you very much again,

Norman

ADD REPLY
0
Entering edit mode

Without knowing the internals of the perl script, I don't even know if this can be "fixed". But my gut feeling is the perl scripts expects the data to be stored in a file, and -i something is interpreted as "read the data from file something" - thus, save the results to a file and subsequently feed this file to the perl script in the same loop.

ADD REPLY
0
Entering edit mode

Hi!

Your gut feeling is very reliable. I implemented your suggestion to just open the file. I almost accepted to produce 50k nonsensical files, but then it came to me that i also could delete them in that loop. 😂

So, this thing is now using two files to compute the pearson correlation, using the script from Didier Gonze. http://homepages.ulb.ac.be/~dgonze/SCRIPTS/PERL/correlation.pl

for sample in *.csv;
do
comparator=Csf3
pr -m -t -s\  $sample $comparator.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g'| cat >${sample}.tsv
perl corr.pl -i "$sample".tsv -col 1 2 | cat >>results$comparator.tsv
rm "$sample".tsv
done

Thank you very much for your help! :-)

Best regards,

Norman

ADD REPLY

Login before adding your answer.

Traffic: 1473 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6