How to download all searched SRA result data at once?
1
0
Entering edit mode
4.6 years ago

Dear Biostars,

There has been a lot of question and tutorial here teaching people how to download SRA data using SRA toolkits, R Bioconductor program, NCBI ENTREZ and so on, and I have learnt a lot from them.

It is easy to search on the SRA website to find specific data using keywords like (METAGENOMIC[Source]) AND (RNA-Seq[Strategy]).

I usually download SRA data using the SRA tookilts with fastq_dump and can easily get one specific data given the accession number, i.e. SRRXXXXXX.

However, when it comes to the situation that I would like to download all the searched SRA data resulted from the keywords, fastq_dump does not seem to be a good choice...

Is there any other method in downloading all the searched data from SRA? i.e. Is there any way to download ALL the data from the following link: https://www.ncbi.nlm.nih.gov/sra/?term=(RNA-Seq%5BStrategy%5D)

Thank you very much!

sra download data-mining data mining • 1.9k views
ADD COMMENT
0
Entering edit mode
4.6 years ago
timstuart ▴ 30

Hi, I had a similar problem so wrote this script to do it, which I find is better than fastq-dump: https://github.com/timoast/ngs-tools/blob/master/download_from_sra.sh

You just need to get the list of accession names from the SRA by going to that page you linked, select send to > file > format = accession list

ADD COMMENT
0
Entering edit mode

Thanks Tim! So now I just need to make a list of accession number from the searched result.

Is there any restriction on the list? Say does it have to be in .txt file? Is there any restriction on the delimiter too?

I am sorry for asking these, may seem silly, question as I am actually still a bit new to programming ...

Thank you very much!

ADD REPLY
0
Entering edit mode

You just need one line per SRA accession, exactly as the file list is downloaded from the SRA. Then run the program like this:

sh download_from_sra.sh -f SraAccList.txt

ADD REPLY
0
Entering edit mode

Thanks Tim! This helped a lot!

ADD REPLY
0
Entering edit mode

Hi Tim, I am sorry that this might seem to be a silly question but I am still pretty new to linux.

So, I downloaded your script and had made the accession list and tried to run with the following command:

$sh download_from_sra.sh -f ./test/test2.txt Where the are some error popped out as followings: download_from_sra.sh: line 2:$'\r': command not found download_from_sra.sh: line 4: $'\r': command not found download_from_sra.sh: line 9:$'\r': command not found download_from_sra.sh: line 11: $'\r': command not found download_from_sra.sh: line 17:$'\r': command not found download_from_sra.sh: line 24: syntax error near unexpected token $'in\r'' 'ownload_from_sra.sh: line 24: case$opt in

Did I missed anything or did I did something wrong?

ADD REPLY
0
Entering edit mode

Did you by chance open this using Windows and saved it again? Looks like Windows line endings. Try dos2unix download_from_sra.sh to fix the line endings.

ADD REPLY
0
Entering edit mode

Oh yes... I did... Thanks Wouter!

Would you mind if I ask one more thing.

As I am using cygwin64, it is somehow strange that I cannot run SRAtool on bash... but I can do so when using window command terminal.

It seems that SRAtoolkit for Linux is not applicable on cygwin64 ... I know that virtual machine is one of the solution. However, is there other way to execute Tim's script on Window beside using virtual machine?

Thanks a lot!

ADD REPLY
0
Entering edit mode

I don't know - I'm not a Windows guy :-)

ADD REPLY
0
Entering edit mode

I see ... Thanks Wouter anyway! Maybe I should start using virtual machine or even simply purchase a new desktop using linux ...

ADD REPLY
0
Entering edit mode

Admittedly it sounds scary at first to change your operating system you got so used to, but I have no regrets of switching to Linux (Ubuntu).

ADD REPLY
0
Entering edit mode

Hi Tim, do you think there is any way to run your script on a window based terminal?

ADD REPLY
0
Entering edit mode

I don't use windows myself so I'm not 100% sure, but I think you can now run bash on windows 10...or maybe try starting docker container with ubuntu bash: docker run -t -i ubuntu bash https://www.docker.com/docker-windows

You'll also need pigz so run apt-get update && apt-get install pigz

ADD REPLY
0
Entering edit mode

Thanks Tim, I will try it out !

ADD REPLY

Login before adding your answer.

Traffic: 2583 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6