Quick download of FASTQ

Question

[Thanh]

I'm figuring out how to download fastq files from either SRA or ENA in a quick manner. I have previously done it on a HPC computer using prefetch/fastq-dump. However, when I tried this on my personal laptop, the process took a very long time.

I have a vague idea that Aspera Connect can provide high-speed downloading of fastq file. However, I'm quite perplexed as to which application to be downloaded (whether it is Aspera Connect or Aspera CLI or both?) and installed and the specific process of downloading fastq files with aspera.

[GenoMax]

Download aspera client from: https://www.ibm.com/aspera/connect/

(from @ATPoint tutorial found elsewhere on biostars, not linking directly since it is deprecated in parts):

For Linux, it is a tarball (use tar zxvf to unpack) with an installer batch script and for Mac, a standard disk image.

After installation, there now will be these executables/files in their default locations:

Linux:

$HOME/.aspera/connect/bin/ascp --- the executable

$HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh --- openssh file

Mac:

$HOME/Applications/Aspera\ Connect.app/Contents/Resources/ascp --- the executable

$HOME/Applications/Aspera\ Connect.app/Contents/Resources/asperaweb_id_dsa.openssh --- openssh file

In any case, make sure you add the folder with the ascp executable to your $PATH.

Then use sra-explorer to get the aspera command lines.

Edit: Going to move this to a comment since the installation on linux does not seem to be functional. As noted by ATpoint the SSH key file is missing from installer.

[ATpoint]

Just realized that with the current version of Aspera Connect (>4.2) they removed that openssh key from the tarball one can download -- and without that the command does not work anymore, older versions are behind a login wall, and I do not know how to work around that for now. If anyone knows please share.

Or just use wget links...

[GenoMax]

IBM appears to have made this more difficult than necessary. Now the openssh keys may be provided by individual clients. So unless you download the client from a place that licenses aspera this may no longer work.

[ATpoint]

Found a way -- writing a tutorial, hang on.

[Carambakaracho]

Not a direct reply to your question: maybe you can give gget a try? I never tried it, and I'd be surprised if it was a drop in replacement for aspera. For myself I had not the patience to setup aspera on my side.

[GenoMax]

I don't think gget can download data from SRA.

Answer 1

[lieven.sterck]

What I usually do is to go via sra-explorer.info , that site is excellent to search for datasets and to then get several options on how to download them ... ftp, wget, ...

Answer 2

[ATpoint]

Setting up Aspera Connect (ascp) on Linux and macOS