3
70
Entering edit mode
2.8 years ago
ATpoint 48k

Edit 06.05.20: I actually recommend to use sra-explorer these days to get download links. This is the most convenient and fastest way. It implements the creation of download links from both NCBI and ENA including links for Aspera-driven downloads based on this tutorial. Step 1 of this tutorial can help setting up Aspera. The other steps are redundant if using sra-explorer, but this tutorial has been written before that tool was published. For more details see also sra-explorer : find SRA and FastQ download URLs in a couple of clicks and visit sra-explorer.info. It is self-explanatory.

TUTORIAL

Published sequencing data are commonly stored at NCBI and questions on how to efficiently download these data are posted frequently at Biostars. While NCBI relies on its SRA format to share these data it is possible to directly download them in fastq format from the European Nucleotide Archive (ENA[1]) which mirrors NCBI.

In this tutorial, we will examplarily download an entire dataset of ChIP-seq and ATAC-seq data, requiring minimal preprocessing work. We will use the Aspera client for download rates of several tens of Mb/s up to few hundred Mb/s (depending on the connection, I/O capacity and distance to the download location). This example code should work on Linux and Mac.

Step-1: Get the Aspera client

Go to https://downloads.asperasoft.com/en/downloads/8?list and get the most recent installer for your system. For Linux, it is a tarball (use tar zxvf to unpack) with an installer batch script and for Mac, a standard disk image.

After installation, there now will be these executables/files in their default locations:

Linux:

$HOME/.aspera/connect/bin/ascp --- the executable $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh --- openssh file that we'll need later

Mac:

$HOME/Applications/Aspera\ Connect.app/Contents/Resources/ascp --- the executable $HOME/Applications/Aspera\ Connect.app/Contents/Resources/asperaweb_id_dsa.openssh --- openssh file that we'll need later

In any case, make sure you add the folder with the ascpexecutable to your PATH. If PATH is a new word to you, please google it ;-)

We have multiple options here.

Use sra-explorer: The sra-explorer is a convenient GUI to get browse NCBI for datasets. In our case we enter our accession number PRJNA288801 into the search field, then select the desired samples, add them to the cart and then copy the Aspera download links it produces. I will not add further details here as the tool is self-explainatory. It includes many (download) options, check it out. Thanks to Phil Ewels for this awesome tool!

As you'll see in accessions.txt, the download paths direct you to the ENA ftp-server, which is rather slow. We want to download with the Aspera client (up to 200Mb/s at my workplace). Therefore, we awk around a bit to change the download paths to the era-fasp server. As you'll see in case of paired-end data, the paths to the two mate fastq files in accessions.txt are separated by semicolon, which we take into account. The output of this snippet is download.txt.

Linux:

awk 'FS="\t", OFS="\t" { gsub("ftp.sra.ebi.ac.uk", "era-fasp@fasp.sra.ebi.ac.uk:"); print }' accessions.txt | cut -f3 | awk -F ";" 'OFS="\n" {print $1,$2}' | awk NF | awk 'NR > 1, OFS="\n" {print "ascp -QT -l 300m -P33001 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh" " "$1 " ."}' > download.txt


Mac:

awk 'FS="\t", OFS="\t" { gsub("ftp.sra.ebi.ac.uk", "era-fasp@fasp.sra.ebi.ac.uk:"); print }' accessions.txt | cut -f3 | awk -F ";" 'OFS="\n" {print $1,$2}' | awk NF | awk 'NR > 1, OFS="\n" {print "ascp -QT -l 300m -P33001 -i $HOME/Applications/Aspera\\ Connect.app/Contents/Resources/asperaweb_id_dsa.openssh" " "$1 " ."}' > download.txt


Edit (23.07.18): The download paths are always like era-fasp@fasp.sra.ebi.ac.uk:/vol1(...). I point that out because of a recent post (328182) where OP accidentally forgot the ":" after the .ac.uk and used fasp@ instead of era-fasp@.

## Either by a simple loop:
$LIST; done < download.txt ## or by using GNU parallel to have things parallelized: cat download.txt | parallel "{}"  Once the download is complete, one can play around using the accessions.txt to rename the files with e.g. information from the Experiment title field (column 2), or other metadata you may retrieve from ENA. Edit 28.2.19: For matters of completeness, I also add a suggestion on how to get the same data from NCBI using prefetch and parallel-fastq-dump, a wrapper for fastq-dump from Renan Valieris for parallelized fastq conversion from sra files. Say one has a file IDs.txt which contains the SRA file IDs like: SRRXXXXXX1 SRRXXXXXX2 (...) SRRXXXXXXn  one can use this simple function to download SRA files via prefetch, followed by fastq conversion with parallel-fastq-dump. Note In the past prefetch could be coupled with Aspera for fast downloads, this is no longer the case, see https://github.com/ncbi/sra-tools/issues/255. Now prefetch will always download via the standard https connection. function LoadDump { prefetch -O ./ -X 999999999$1

if [[ -e ${1}.sra ]]; then parallel-fastq-dump -s${1}.sra -t 8 -O ./ --tmpdir ./ --split-3 --gzip && rm ${1}.sra else echo '[ERROR]'$1 'apparently not successfully loaded' && exit 1
fi

cat IDs.txt | parallel -j 2 "LoadDump {}"


This would use 8 threads for fastq conversion and run two SRA files at a time via GNU parallel, hence requiring 16 threads. As always, scale up or down based on the available resources and potential I/O bottlenecks on your system.

ENA Fastq fastq-dump aspera Tutorial • 24k views
2
Entering edit mode

Good work! I once had the pleasure to use fastq-dump on whole-genome data. I was cursing in multiple languages! :D

2
Entering edit mode

I recently was downloading the data form CCLE experiment and it was taking ages (also crushing more than once) with sra-toolkit and fastq-dump. I used your approach, slightly it modifying and it worked wonders! Thanks a lot!

My modification below:

esearch -db sra -query PRJNA523380 | efetch --format runinfo |  grep ${tissue_of_interest} | grep${experiment} | cut -f1 -d',' | xargs | sed 's/ / OR /g' | xclip -selection c


where ${tissue_of_interest} and ${experiment} where variables I set up specifically to my needs (i.e. CERVIX, RNA-seq). I copied this into sra-explorer. The Project has too many files to directly search sra-explorer with its ID.

1
Entering edit mode

Glad to hear it is used productively :)

0
Entering edit mode

I am sorry. I am considering switch from SRA because the files I want to download are too big and I got unkown error using SRA for some time. I am trying to use aspera client to download the fastq files. From the link above I downloaded the ibm-aspera-connect-3.9.9.177872-linux-g2.12-64.tar.gz and decompressed it to ibm-aspera-connect-3.9.9.177872-linux-g2.12-64.sh.

I also set the path to the directory I am working:

PATH=$PATH:/home/caiqi/.aspera/connect/bin/ascp export PATH  everything looks fine until the next step; ascp -QT -l 300m -P33001 -i$HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/SRR504/007/SRR5048027/SRR5048027_1.fastq.gz .


Which part did I made the mistake

1
Entering edit mode

PATH is the path to the folder containing the binaries, not the binary itself, please try:

PATH=\$PATH:/home/caiqi/.aspera/connect/bin/

1
Entering edit mode

Thank you for explanation. It is working and it is faster than I expected

0
Entering edit mode

Good work dude. Will use this next time I need to get data from ENA.

0
Entering edit mode

Thank you Sir, glad to help!

0
Entering edit mode

I finally got this to work. Do you know why I could be getting variable download speeds? The frist time I ran this, I was getting speeds of ~230 MB/s and now I'm getting 1.7 MB/s at best. Is there an option I can choose to increase the speed? What could be affecting this you think?

0
Entering edit mode

Which of the method is usually the fastest to download fastq files? I know it depends a lot on network yet over a stable connection (but low bandwidth) which one of wget, curl, aspera or SRA toolkit performs better?

0
Entering edit mode

Aspera will be the fastest , instead of curl or wget use Aria2 for a faster download.

2
Entering edit mode

SRAtoolkit downloads via http but it performs some internal validation checks and only will produce the final SRA file on disk if download finished successfully afaik, so for downloads from SRA I prefer it. NCBI does not support Aspera anymore.

For ENA it would be Aspera, this is why I cover it as the default method in this tutorial. Still, unless you download hundreds of GB or even TB standard download via wget is fine as well. With our HPC I tyically get download rates via wget from ENA in the ~50-60Mb/s range. Sufficient for most applications unless you are going to download entire cohorts.

10
Entering edit mode
20 months ago
benjwoodcroft ▴ 170

Hi,

This is a very helpful post - thanks a lot for writing it. I wrote a simple Python script based on this which automates things so you only need to provide a run identifier as an argument and it works out the rest - hopefully someone will find it useful.

2
Entering edit mode

Cool, very useful if you want to batch-query many accession numbers. You might want to add an option like --linux and --osx to output the correct path to the default aspera openssh file as default paths are a bit different in both operating systems. If you want to search NCBI also check out sra-explorer : find SRA and FastQ download URLs in a couple of clicks from Phil Ewels which has an option to print ENA links directly.

1
Entering edit mode

Good idea - I added an --ssh_key option along those lines. I've not had a chance to test it on OSX though - if that is straightforward for you would you mind giving it a crack please?

0
Entering edit mode

Works like a charm! Just a small thing, maybe add a kind of exists(--ssh_key) an isinPATH (ascp) (no clue what the command in python are :-D ) option that stops the run right away if the key does not exist. Everything else works really nice!

0
Entering edit mode

0
Entering edit mode

great tool, super useful and works like a charm, kudos!

1
Entering edit mode
8 months ago
Brunox13 ▴ 30

The sra-explorer is currently not working for me, unfortunately.

However, when selecting the columns at the ENA website (above in Step-2: Choose your dataset), it is now possible to select fastq_aspera links directly, as shown below.

As far as I can tell, he rest of this tutorial is still up to date. Thank you for saving me from fasterq-dump that has been just giving me headache so far!!