SRA Toolkit Prefetch error when downloading dbGaP SRR files

Question

[Anthony.Knox]

I am trying to download SRR files that requires authorization through dbGaP, so I have to download it through command line. I downloaded the Aspera Connect CLI as well as the Aspera Connect app and imported the repository key file correctly. I am using prefetch-2.8.2. I am reading in a text file with each line as a different SRR file and calling prefetch for that file with /path/to/prefetch-2.8.2 SRRXXXXXX (this is the SRR_download_ascp.sh shell script). The . . . is where it successfully downloaded 124 out of 299 dependencies. It seems to not be able to get past 124 dependency downloads.

Why am I getting this file descriptor error after 124 dependencies and then the download fails? What are these dependencies and is it integral that they download correctly? Hard disk space is not an issue.

Here is my SRR_download_acsp.sh shell script, and I called chmod +x SRR_download_ascp.sh (not shown)

while IFS= read -r line || [[ -n "$line" ]]; do
    echo $line
        /Users/anthonyknox/Documents/AKnox/RNA-seq_analysis/Tuxedo_tools/sratoolkit.2.8.2-1-mac64/bin/prefetch.2.8.2 $line
done < "$1"

Here is my code:

otp1423081ots:~ anthonyknox$ PATH=$PATH:/Users/anthonyknox/Documents/AKnox/RNA-seq_analysis/Tuxedo_tools/sratoolkit.2.8.2-1-mac64/bin 
otp1423081ots:~ anthonyknox$ cd /Users/anthonyknox/ncbi/dbGaP-16260 
otp1423081ots:dbGaP-16260 anthonyknox$ vi SRR_download_ascp.sh 
otp1423081ots:dbGaP-16260 anthonyknox$ ./SRR_download_ascp.sh /Users/anthonyknox/ncbi/Downloading\ SRR/RV_LV_SRR_1.txt 
SRR1768659

2017-12-05T18:55:06 prefetch.2.8.2: 1) Downloading 'SRR1768659'...
2017-12-05T18:55:06 prefetch.2.8.2:  Downloading via fasp...
2017-12-05T18:55:51 prefetch.2.8.2:  fasp download succeed
2017-12-05T18:55:51 prefetch.2.8.2: 1) 'SRR1768659' was downloaded successfully
2017-12-05T18:57:31 prefetch.2.8.2: 'SRR1768659' has 299 unresolved dependencies
2017-12-05T18:57:32 prefetch.2.8.2: 2) Downloading 'ncbi-acc:CM000663.2?vdb-ctx=refseq'...
2017-12-05T18:57:32 prefetch.2.8.2:  Downloading via fasp...
2017-12-05T18:57:37 prefetch.2.8.2:  fasp download succeed
2017-12-05T18:57:37 prefetch.2.8.2: 2) 'ncbi-acc:CM000663.2?vdb-ctx=refseq' was downloaded successfully
.
.
.
2017-12-05T19:06:26 prefetch.2.8.2: 124) Downloading 'ncbi-acc:KI270529.1?vdb-ctx=refseq'...
2017-12-05T19:06:26 prefetch.2.8.2:  Downloading via fasp...
pipe() from: Too many open files
2017-12-05T19:06:26 prefetch.2.8.2 err: file descriptor failed while creating file descriptor - while pipe
pipe() to: Too many open files
2017-12-05T19:06:26 prefetch.2.8.2 err: file descriptor failed while creating file descriptor - while pipe
2017-12-05T19:06:26 prefetch.2.8.2:  fasp download failed
2017-12-05T19:06:26 prefetch.2.8.2:  Downloading via http...
2017-12-05T19:06:26 prefetch.2.8.2 int: connection not found while validating within network system module - ncbi-acc:KI270529.1?vdb-ctx=refseq: Cannot resolve remote
2017-12-05T19:06:26 prefetch.2.8.2: 124) failed to download ncbi-acc:KI270529.1?vdb-ctx=refseq
2017-12-05T19:06:30 prefetch.2.8.2: 'SRR1768659' has no remote vdbcache

Thank you in advance!

Answer 1

[Anthony.Knox]

For anyone interested, it turns out that this is a bug where prefetch is leaking file descriptors. As a current workaround (for 2.8.2), just use this command before calling prefetch:

ulimit -n 10000

This will set the maximum number of open file descriptors to 10000 (mine was at 256 as a default).

[GenoMax]

Thank you for coming back and providing closure to this thread.