How to keep my data and analysis up to date across new genome releases
2
0
Entering edit mode
7.5 years ago
Sheila ▴ 270

What are the best methods to maintain up-to-date biological database?

I used a gnome from NCBI which was a large gz file. Now there is a newer release available in gz format. Every-time there is a change in database I have to download it completely and reanalysis the data. Same is the problem with some other databases. Is there a way to just only get the information which have been updated only?

pdb database uniprotKB ncbi • 1.7k views
ADD COMMENT
3
Entering edit mode
7.5 years ago
t.candelli ▴ 60

It depends on which kind of analysis you do. if your outputs are simple data representation such as .bed or .bedgraph files the liftOver tool from UCSC can help keep everything up to date. it might require more work in generating the chain files if you work with unusual organisms. this method works for almost any form of feature-based annotation (or even .wig files), the only thing you need is to convert it into a .bed or .bedgraph.

A brief word on how liftOver works. the program is designed to convert a set of genomic coordinates (with respective values if present) between different assemblies of the same organism. in order to do this, liftOver requires a special "chain file", a file containing the differencese between the two target assemblies.

a number of pre-computed chainfiles for a number of different organisms, along with liftOver itself, is downloadable here.

ADD COMMENT
0
Entering edit mode

Thanks. I am interested in gz files or any flat file data. Is there any tool/method of just only downloading changes!

ADD REPLY
0
Entering edit mode

No, it doesn't look like UCSC publishes diffs. It is probably easier for them to publish compressed files. You may need to download the entire file.

ADD REPLY
0
Entering edit mode

UCSC does publish a number of "diffs", or in this context chain files, between a number of different assemblies in different organisms here. i will update the answer to be more detailed.

ADD REPLY
0
Entering edit mode
7.5 years ago

For UniProtKB, you could explore the advanced query mechanisms by date and use them programmatically (cf http://www.uniprot.org/faq/28 and http://www.uniprot.org/help/query-fields ), e.g.

lwp-mirror "http://www.uniprot.org/uniprot/?query=reviewed:yes+AND+created:[current+TO+current]&format=txt" new_seq.dat
lwp-mirror "http://www.uniprot.org/uniprot/?query=reviewed:yes+AND+sequence_modified:[current+TO+current]&format=txt" upd_seq.dat
lwp-mirror "http://www.uniprot.org/uniprot/?query=reviewed:yes+AND+modified:[current+TO+current]&format=txt" upd_ann.dat
ADD COMMENT

Login before adding your answer.

Traffic: 2269 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6