This question was inspired from a thread on stackoverflow regarding BerkeleyDB-JE :
Inserting data into BerkeleyDB-JE is getting slower and slower
and a conversation on twitter with Jerven@uniprot:
I'm currently inserting a large number of data (dbsnp) into a BDB and I'd like to deploy it into a web-server (glassfish) to implement some web services. The structure of uniprot has been briefly described in :
Infrastructure for the life sciences: design and implementation of the UniProt website
I think biostar would be a nice place to ask Jerven for some specific questions about the implementation of uniprot. Here are my questions :-)
- WebContainer: wich one do you use ? tomcat ?
- Do you use a specific framework to wrap BerkeleyDB (something like JPA) ?
- How do you manage the BDB-Environment in the web-container ? I would open it in a ContextListener and put it in the WebContext.. but is it safe for the Multi-threaded access ?
- Do you use a 'high-availability' environment ?
- in the web-context: Is your BDB-Env read-only ? do you use transaction. Do you use a copy of your database for the web-access ? or do people at uniprot use the same BDB-ENV for working/inserting/updating ?
- How many database does your BDB-env contain ? do you open all those databases in the BDB.env on startup ?
- Do you handle your data like a SQL database (column-oriented ?) or do you put some complex structures in the DatabaseEntry (like a whole XML record) UML doesn't work for NOSQL. How do you write the documentation about the schema for those complex objects.
- Did you write some specific JSP-Tags form BDB or used a known jsp library for BDB ?
- REST services: did you use Jersey (JAX-RS) or did you implement your own web-services ?
- In the paper you said SOAP was a bad idea ... " necessitated introducing limitations such as the maximum number of entries that could be retrieved in one go"... What's the difference with a REST solution. I mean you can stream the result with REST but, as far as I known, you could also do the same with SOAP:
- Entry bindings: do you use things like @Annotation for BDB-JE our did you write your own EntryBinding<T> ?
- does BDB sometimes broke ? do you have to sometimes rebuild the whole BDB ? How long does it take ? How much space do you need ?
- Any regret about your current architecture ?
Thank you :-)
Many thanks Jerven ! :-)