Question: Load data from NCBI .gpff files into a SQL database. Looking for a quicker way.
gravatar for azotov
2.6 years ago by
UAB, Birmingham AL, USA
azotov10 wrote:

I'm trying to load NCBI reference sequence data (from f into an SQL server for further processing. I'm using a straightforward method with BioPython of parsing the text file with seqIO.parse() and then loading it into SQL with BioSeqDatabase.load(). That converts data at the rate of around 10 entries/second if I have an SQL server on the same computer or slightly slower if SQL is remote, what seems too slow. I was not able to run multiple python scripts in parallel. It runs into error 1205 : "Lock wait timeout exceeded", so it looks like only 1 BioSeqDatabase.load() function at a time can access the SQL tables. I have access to a massive cluster, but I don't see a way of utilizing it for this task.

What is a good way to speed up the process by parallelizing or else?

bioseqdatabase sql biopython • 1.0k views
ADD COMMENTlink modified 2.5 years ago • written 2.6 years ago by azotov10
gravatar for Pierre Lindenbaum
2.6 years ago by
France/Nantes/Institut du Thorax - INSERM UMR1087
Pierre Lindenbaum128k wrote:

normalize your data to create a set of flat files (creating your primary keys by hand) and then use mysqlimport

ADD COMMENTlink written 2.6 years ago by Pierre Lindenbaum128k

Works like a charm! Thanks Pierre

ADD REPLYlink written 2.5 years ago by azotov10
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 897 users visited in the last hour