How To Write Bash Script To Download Files From Specific Databse
3
1
Entering edit mode
7.8 years ago
jack ▴ 490

Hi all,

I want to download some files from online database. but it does not allow to download all the files at once, instead it's offer to download file for searched keyword, and because my keywords are more than 20000, so in principal it's not feasible for me for example, I want to download whole information about miRNA-mRNA interaction from SarBase but it does not offer option to download all of them at once. I wonder , how can I download it by writing some scripts. can anybody help me ?

script bioinformatician • 18k views
ADD COMMENT
2
Entering edit mode
ADD REPLY
2
Entering edit mode
7.8 years ago

For static files, you could use either wget or curl in a bash script, e.g.:

#!/bin/bash
wget https://www.example.com/foo/bar.baz -O /path/to/destination/bar.baz
curl https://www.example.com/baz/foo.bar --output /path/to/destination/foo.bar
...

If it is a web form, you could emulate submitting a form with curl -F and key-value pairs representing form fields, using the URL or address of the form CGI, e.g.:

#!/bin/bash
curl -F "foo=bar;baz=foo" https://www.example.com/some_form.cgi --output /path/to/destination/some_result

You can also use wget --post-data; take a look at the man pages for the two tools.

If you put textual form data into a script, make sure its contents are percent-encoded. If you put binary data into a script, you'll need to do some other encoding, such as base64-encoding. There are numerous CLI tools to do these and other kinds of encoding.

ADD COMMENT
1
Entering edit mode
7.8 years ago
Zhaorong ★ 1.3k

Is this what you want? http://starbase.sysu.edu.cn/browseClipSeq.php Click "export" at the upper left corner of the table.

ADD COMMENT
1
Entering edit mode
7.8 years ago
zhxq09 ▴ 140

In miRNA-mRNA query webpage, if you do not input/select anything, you can browse and download all miRNA-mRNA interactions. You only click "export" at the upper left corner of the table and download all results.

ADD COMMENT

Login before adding your answer.

Traffic: 2384 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6