Downloading from MG-RAST
1
0
Entering edit mode
7.0 years ago
clm406 • 0

I am trying to download ~100 public amplicon sets from MG RAST I am using a slightly modified script from a previous entry here on BioStars (Downloading Data From Mg-Rast) that looks like this 

cat names.csv|while read line;do curl http://api.metagenomics.anl.gov/1/download/"$line"?file=100.2 >$line.fna;done

where names.csv is a list of the numbers of the files in which I am interested.

This will write all the files for the entries names.csv like it is supposed to, but each of those files only contain an error message:

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

I can download the files if I replace "$line" with the actual number in my browser or if I run the command 

 curl http://api.metagenomics.anl.gov/1/download/metagenomenumber?file=100.2 >metagenomenumber.fna

from my terminal window, but I would much prefer to not have to do each one manually.

Any ideas on why my shell script may be failing? 

Thanks!

Christy

 

MG-RAST • 3.6k views
ADD COMMENT
0
Entering edit mode

Try replacing $line with ${line} maybe? Just an idea since I don't have access to your CSV file.

ADD REPLY
0
Entering edit mode
7.0 years ago
hpmcwill ★ 1.2k

The URL of the resource to be downloaded should generally be completely enclosed in quotes to prevent any of its contents being interpreted by the shell as additional arguments or file patterns. In your case this means something like the following will work as you expect:

cat names.csv | while read line; do curl "http://api.metagenomics.anl.gov/1/download/$line?file=100.2" > "$line.fna"; done

As noted by Dan D, you might want to protect yourself further by making the variable boundaries explicit by using something like:

cat names.csv | while read line; do curl "http://api.metagenomics.anl.gov/1/download/${line}?file=100.2" > "${line}.fna"; done

Although this is unnecessary in this case.

 

ADD COMMENT
0
Entering edit mode

Is this question and answer about the Unix language for cURL? Is there a similar function for 'while' and 'do' for cURL in a windows command promt? I found that 'type' is a similar command to 'cat'. 

ADD REPLY

Login before adding your answer.

Traffic: 1739 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6