How to transfer big sequencing data abroad on internet?
1
1
Entering edit mode
4.8 years ago
hsu ▴ 40

I have a very big database almost150G, how could I transfer it to abroad on internet? Can you recommend some methods? Thank you very much!

transfer database sequence • 3.2k views
ADD COMMENT
1
Entering edit mode

You can always upload it to SRA and make it public.

ADD REPLY
0
Entering edit mode

*assuming its sequence data...

ADD REPLY
0
Entering edit mode

Compress the files and upload them to institute FTP server or any cloud storage platform.

ADD REPLY
0
Entering edit mode

Take your pick from an FTP/HTTP webserver, Aspera, S3, torrents, or the good old fashioned post-them-a-harddrive.

Which of these is best might also depend on their data download speeds. If they have a poor connection, something with innate integrity checking that has the file broken up in to parts, like torrenting, might be best.

ADD REPLY
0
Entering edit mode

Could you explain more details?.......

ADD REPLY
0
Entering edit mode

About which bit exactly?

ADD REPLY
0
Entering edit mode

Or setup an FTP server yourself.

ADD REPLY
1
Entering edit mode

One issue I nearly always had is the local network is behind NAT and the local IP address is not static/globally_routeable which makes such servers only available locally. It is nearly always the case in both home and office. Although I got a static IP when I paid handsomely to my ISP :) .

ADD REPLY
0
Entering edit mode

About the FTP/HTTP webserve and torrents, thank you very much

ADD REPLY
6
Entering edit mode
4.8 years ago
Joe 21k

Aspera (IBM) and S3 (Amazon) are cloud storage options that allow for high speed uploads and downloads over a simple link that you can share. This is functionally no different than dropping your data into DropBox, or Google Drive etc. They are however, not typically free.

Setting up an FTP server is somewhat like hosting an S3 or Aspera server. You yourself are holding the data on a server you are running (rather than using a middleman like Amazon etc). The person you're trying to send the data to can then log in to the server and access the data. A common usecase is the NCBI FTP for instance: ftp://ftp.ncbi.nlm.nih.gov/

Your user would get a similar experience at the other end. This might be preferable if you need to do this long term, but you need to maintain the webserver too.

Torrents are used a little less these days (unless you're downloading movies illegally), but a torrent is basically a way of sharing 'bits' of a file. A torrent is a description of where to get the data, and the torrent handler will break up your file in to small peices, which are then put back together at the other end. What's good about this is it means that the file is checked for corruptions etc as its put back together at the other end, and the download can be paused or resumed at any time. Typically, torrents work by gathering different bits of the same file from many "seeders", but in this case you'd be providing the whole thing.

The torrenting approach probably won't be the fastest, but it would avoid issues with the end user downloading 140GB of your 150GB, before the file being corrupted, and them having to start all over again.

ADD COMMENT

Login before adding your answer.

Traffic: 1716 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6