Question: How to download all searched SRA result data at once?
0
gravatar for sratestimon
4 months ago by
sratestimon0 wrote:

Dear Biostars,

There has been a lot of question and tutorial here teaching people how to download SRA data using SRA toolkits, R Bioconductor program, NCBI ENTREZ and so on, and I have learnt a lot from them.

It is easy to search on the SRA website to find specific data using keywords like (METAGENOMIC[Source]) AND (RNA-Seq[Strategy]).

I usually download SRA data using the SRA tookilts with fastq_dump and can easily get one specific data given the accession number, i.e. SRRXXXXXX.

However, when it comes to the situation that I would like to download all the searched SRA data resulted from the keywords, fastq_dump does not seem to be a good choice...

Is there any other method in downloading all the searched data from SRA? i.e. Is there any way to download ALL the data from the following link: https://www.ncbi.nlm.nih.gov/sra/?term=(RNA-Seq%5BStrategy%5D)

Thank you very much!

ADD COMMENTlink modified 4 months ago by timstuart30 • written 4 months ago by sratestimon0
0
gravatar for timstuart
4 months ago by
timstuart30
Australia
timstuart30 wrote:

Hi, I had a similar problem so wrote this script to do it, which I find is better than fastq-dump: https://github.com/timoast/ngs-tools/blob/master/download_from_sra.sh

You just need to get the list of accession names from the SRA by going to that page you linked, select send to > file > format = accession list

ADD COMMENTlink written 4 months ago by timstuart30

Thanks Tim! So now I just need to make a list of accession number from the searched result.

Is there any restriction on the list? Say does it have to be in .txt file? Is there any restriction on the delimiter too?

I am sorry for asking these, may seem silly, question as I am actually still a bit new to programming ...

Thank you very much!

ADD REPLYlink written 4 months ago by sratestimon0

You just need one line per SRA accession, exactly as the file list is downloaded from the SRA. Then run the program like this:

sh download_from_sra.sh -f SraAccList.txt

ADD REPLYlink written 4 months ago by timstuart30

Thanks Tim! This helped a lot!

ADD REPLYlink written 4 months ago by sratestimon0

Hi Tim, I am sorry that this might seem to be a silly question but I am still pretty new to linux.

So, I downloaded your script and had made the accession list and tried to run with the following command:

$ sh download_from_sra.sh -f ./test/test2.txt

Where the are some error popped out as followings:

download_from_sra.sh: line 2: $'\r': command not found download_from_sra.sh: line 4: $'\r': command not found download_from_sra.sh: line 9: $'\r': command not found download_from_sra.sh: line 11: $'\r': command not found download_from_sra.sh: line 17: $'\r': command not found download_from_sra.sh: line 24: syntax error near unexpected token $'in\r'' 'ownload_from_sra.sh: line 24: case $opt in

Did I missed anything or did I did something wrong?

ADD REPLYlink modified 4 months ago • written 4 months ago by sratestimon0

Did you by chance open this using Windows and saved it again? Looks like Windows line endings. Try dos2unix download_from_sra.sh to fix the line endings.

ADD REPLYlink written 4 months ago by WouterDeCoster23k

Oh yes... I did... Thanks Wouter!

Would you mind if I ask one more thing.

As I am using cygwin64, it is somehow strange that I cannot run SRAtool on bash... but I can do so when using window command terminal.

It seems that SRAtoolkit for Linux is not applicable on cygwin64 ... I know that virtual machine is one of the solution. However, is there other way to execute Tim's script on Window beside using virtual machine?

Thanks a lot!

ADD REPLYlink written 4 months ago by sratestimon0

I don't know - I'm not a Windows guy :-)

ADD REPLYlink written 4 months ago by WouterDeCoster23k

I see ... Thanks Wouter anyway! Maybe I should start using virtual machine or even simply purchase a new desktop using linux ...

ADD REPLYlink written 4 months ago by sratestimon0

Admittedly it sounds scary at first to change your operating system you got so used to, but I have no regrets of switching to Linux (Ubuntu).

ADD REPLYlink written 4 months ago by WouterDeCoster23k

Hi Tim, do you think there is any way to run your script on a window based terminal?

ADD REPLYlink written 4 months ago by sratestimon0

I don't use windows myself so I'm not 100% sure, but I think you can now run bash on windows 10...or maybe try starting docker container with ubuntu bash: docker run -t -i ubuntu bash https://www.docker.com/docker-windows

You'll also need pigz so run apt-get update && apt-get install pigz

ADD REPLYlink written 4 months ago by timstuart30

Thanks Tim, I will try it out !

ADD REPLYlink written 4 months ago by sratestimon0
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1474 users visited in the last hour