R And Sqlite Db Locked Error

Question

[polarise]

I've become bald pulling out my hair because of this problem. Time and time again I need to run an R package that needs to use RSQLite but fails because of NFS locking. Well, that's the setup on our cluster. Curiously, I don't see other people complaining about this problem but I know that a lot of bioinformatics is done on Sun Grid Engine installations, which I assume employ NFS in one way or another.

Any tips on how to alleviate this pain? Are my assumptions right (about SGE and NFS being inseparable)?

Thanks!

PK

[Michael]

This is definitely off topic, but I thank that on serverfault.com you will receive a good answer if you give some more details: e.g. http://serverfault.com/questions/71871/best-system-to-serve-sqlite-databases-to-webservers-over-a-network

[Pierre Lindenbaum]

my experience: sqlite3 doesn't work with SGE/NFS. I tried to log the steps of my NGS workflow with sqlite3, I had to switch to a java standalone sql engine (derby)

[polarise]

Perhaps if I stated that I was running a Bioconductor package it might draw it in... ;-)

[Michael]

If you could also say which package, that may make it bioinformatics

[Sudeep]

Which R/Bioconductor package are you using and when do you get this message ? (I mean are you trying to parallelize calls to a function that uses RSQLite? )

Answer 1

[Sean Davis]

This is a SQLite FAQ.

http://www.sqlite.org/faq.html#q5

Answer 2

[Michael]

Honestly, what I would do in this situation is probably to migrate the package to a server based DBMS by patching the code, e.g. Migrate to RPostgreSQL. The SQL should be mostly compatible and run updates in a transaction.

And did I say already that it would really help to know which package you use? Giving sufficient level of details often helps solving problems, just saying...

[polarise]

I'm using the following package: http://bioconductor.org/packages/release/data/annotation/html/hugene10stprobeset.db.html

The current solution is to install the package on the master node (if accessible) since that will have its own disk hence no NFS lock.