1
0
Entering edit mode
7.6 years ago
clm406 • 0

I am trying to download ~100 public amplicon sets from MG RAST I am using a slightly modified script from a previous entry here on BioStars (Downloading Data From Mg-Rast) that looks like this

cat names.csv | while read line; do curl http://api.metagenomics.anl.gov/1/download/"$line"?file=100.2 >$line.fna; done


where names.csv is a list of the numbers of the files in which I am interested.

This will write all the files for the entries names.csv like it is supposed to, but each of those files only contain an error message:

<html>
<body bgcolor="white">
<hr><center>nginx</center>
</body>
</html>


I can download the files if I replace "$line" with the actual number in my browser or if I run the command curl http://api.metagenomics.anl.gov/1/download/metagenomenumber?file=100.2 > metagenomenumber.fna  from my terminal window, but I would much prefer to not have to do each one manually. Any ideas on why my shell script may be failing? Thanks! Christy MG-RAST • 3.8k views ADD COMMENT 0 Entering edit mode Try replacing $line with ${line} maybe? Just an idea since I don't have access to your CSV file. ADD REPLY 0 Entering edit mode 7.6 years ago hpmcwill ★ 1.2k The URL of the resource to be downloaded should generally be completely enclosed in quotes to prevent any of its contents being interpreted by the shell as additional arguments or file patterns. In your case this means something like the following will work as you expect: cat names.csv | while read line; do curl "http://api.metagenomics.anl.gov/1/download/$line?file=100.2" > "$line.fna"; done  As noted by Dan D, you might want to protect yourself further by making the variable boundaries explicit by using something like: cat names.csv | while read line; do curl "http://api.metagenomics.anl.gov/1/download/${line}?file=100.2" > "\${line}.fna"; done


Although this is unnecessary in this case.

0
Entering edit mode

Is this question and answer about the Unix language for cURL? Is there a similar function for 'while' and 'do' for cURL in a windows command prompt? I found that 'type' is a similar command to 'cat'.