Results of two sequential bash pipes into a variable instead of a file?
0
0
Entering edit mode
2.0 years ago

Hi!

I would like to put the result of a pipe into a variable instead of a file. But it does not work as intended.

The well working script with the pipe resulting in a file is like that:

for sample in *.csv;
do
pr -m -t -s\  $sample Ldha.csv | Awk '{print$6"\t"$13}' | sed 's/\./,/g'| cat >Ldha/${input}.tsv
done


I tried to funnel it into a variable called "input" with these ways:

read input <(pr -m -t -s\  $sample Ets1.csv | Awk '{print$6"\t"$13}' | sed 's/\./,/g') input=pr -m -t -s\$sample  Ldha.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g'
input=$(pr -m -t -s\$sample  Ets1.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g')
pr -m -t -s\  $sample Ldha.csv | Awk '{print$6"\t"$13}' | sed 's/\./,/g' | read input  Nothing worked. If i run it with saving to files, it works. But saving well 50 000 files for each run... In the end, it should look like that (using a perl script for pearson correlations from the web): for sample in *.csv; do input=$(pr -m -t -s\  $sample Ets1.csv | Awk '{print$6"\t"$13}' | sed 's/\./,/g') perl corr.pl -i$input  -col 1 2 | cat >>correl_Ets1.csv
done


But i fail to get the output from the pipe into a variable. Any ideas?

Best wishes,

Norman

bash pipe variable shell script • 532 views
2
Entering edit mode

• You should provide small example files to allow us reproduce the issue
• What "nothing works" means? What is stored in $input with each command? Did you check with echo "$input"?
• It should be awk, not Awk

The following simple example shows no problems assigning piped stdout to a bash variable:

$awk '{print$2"\t"$13}' relAbun.tsv | head U2 U13 0 0 0.1 0 0 0 0 0 0 0 0.3 0.2 0 0 0 0 0 0  Now put the result in a variable: $ input=$(cat relative_abundance_raw.tsv | awk '{print$2"\t"$13}' | head}' relAbun.tsv | head)$ echo "$input" U2 U13 0 0 0.1 0 0 0 0 0 0 0 0.3 0.2 0 0 0 0 0 0  ADD REPLY 0 Entering edit mode Dear h.mon, thank you very much for your reply and the brillant idea with echo$input. Echo input looks good. But now what happens is the following: The perl script is taking the first value of the first file as a filename. Then it takes the first value from the second file as a filename. So, the input is transferred indeed to the variable input. But the filename, which would be taken from $sample is now lost and not fed to the perl script. I am not sure if this can be fixed.... Maybe i need to feed this as an array? But still, how would i pass the filename.... Best regards and thank you very much again, Norman ADD REPLY 0 Entering edit mode Without knowing the internals of the perl script, I don't even know if this can be "fixed". But my gut feeling is the perl scripts expects the data to be stored in a file, and -i something is interpreted as "read the data from file something" - thus, save the results to a file and subsequently feed this file to the perl script in the same loop. ADD REPLY 0 Entering edit mode Hi! Your gut feeling is very reliable. I implemented your suggestion to just open the file. I almost accepted to produce 50k nonsensical files, but then it came to me that i also could delete them in that loop. 😂 So, this thing is now using two files to compute the pearson correlation, using the script from Didier Gonze. http://homepages.ulb.ac.be/~dgonze/SCRIPTS/PERL/correlation.pl for sample in *.csv; do comparator=Csf3 pr -m -t -s\$sample $comparator.csv | Awk '{print$6"\t"$13}' | sed 's/\./,/g'| cat >${sample}.tsv
perl corr.pl -i "$sample".tsv -col 1 2 | cat >>results$comparator.tsv
rm "\$sample".tsv
done


Thank you very much for your help! :-)

Best regards,

Norman