Batching fastq download w/sra-toolkit: Won't recognize IDs

Question

[harka14]

Hi all,

I'm new to bash scripting, and I'm trying to create a simple script to download fastq files from SRA using a list of SRR IDs in a txt file. (Eventually I'd like to parallelize this process, but for now I just need it to work.) I've used fastq-dump from sra-toolkit numerous times, but now I'm having issues with the batching that don't make sense to me.

I have successfully used the following command to download one of the datasets in this list:

~/bin/sratoolkit/bin/fastq-dump -O ~/output/location/01_fastq --gzip --skip-technical  --readids --read-filter pass --dumpbase --split-spot --clip SRRXXXXXXXX

My script, named 01_SRA, is:

BATCH=${1?Error: No batch ID given.}
LOC=${2?Error: No directory given.}

INPUT=${LOC}00_SRAlists/${BATCH}.txt
OUTLOC=${LOC}${BATCH}/01_fastq/

cat $INPUT | while read line
do
        echo Starting $line
        ~/bin/sratoolkit/bin/fastq-dump -O $OUTLOC --gzip --skip-technical --readids --read-filter pass --dumpbase --split-spot --clip $line
done

When I run

~/location/01_SRA batch002 ~/location/seq_data/

I get:

Starting SRRXXXXXXXX
2020-08-27T14:21:16 fastq-dump.2.10.8 err: error unexpected while resolving query within virtual file system module - No accession to process ( 500 )
Failed to call external services.

For each of the IDs in my txt file. This dumps out almost immediately. So based on the echo line, I know it is reading the ID, but somehow fastq-dump isn't able to process it.

I thought it might be a problem with my txt file. I created another file called batchtest.txt that just has the test dataset from the wiki three times:

SRR390728
SRR390728
SRR390728

When I run it:

~/location/01_SRA batchtest ~/tmp/

I don't get an error... (it pauses on the first ID to download)

As far as I can tell, nothing has changed except for the SRR IDs in the text file.

So, in summary:

• Command line + test ID = works fine
• Command line + my IDs = works fine
• Script + test ID = works fine
• Script + my IDs = ERROR

Does anybody know what might be causing this and what I can do to fix it?

Software info:

• Ubuntu v18.04.3 LTS (Bionic Beaver)
• GNU bash, v4.4.20(1)
• sra-toolkit v2.10.8

Thanks

Answer 1

[Ram]

Can you examine the IDs using

echo "Starting $line" | cat -te

Maybe there are some invisible characters messing up your code.

[harka14]

This was it! I've had to convert dos2unix in another context before, so I feel like I should have caught that 🙄 Re-creating the file in nano fixed it. Thank you!

[Ram]

I've moved my comment to an answer. Please go ahead and accept it using the green check mark on the left.