Hi,
I've got a 12 core CPU and 32 GB RAM. I'm trying to optimise my searches by running small batches of queries (say 100 queries each, totalling anywhere between 1 MB and 20 MB) against a database that is 750 MB (the .nsq file, that is) which is about 3 GB in raw FASTA format.
My assumption was that the amount of memory that a single instance of blastn would take up wouldn't be much bigger than the size of the database - I've seen estimates that say you should allow for 20% bigger than the database itself (i.e. 4 GB, max).
But what I'm seeing is some of my blastn processes taking up obscene amounts of RAM - like 12 GB - and eventually, the Linux OOM Killer will kill the offending process. And that makes me sad.
Is this a normal amount of RAM usage .. ?
ETA: Version is 2.14.0
Can you add the exact command you are using? If you are running out of memory using less number of threads is likely the only option in your case since you have a fixed amount of hardware resources.
Instead of starting multiple parallel processes it may be better to start one with the max number of cores you can manage to use and then chunk through the data that way.
Thanks for replying!
Here's the command:
time blastn -task blastn -query $FILENAME -db $DBNAME -num_threads 4 -out $FILENAME.csv \ -outfmt "10 qseqid qstart qend sseqid sstart send pident nident length qlen evalue" \ -dust no -soft_masking false -evalue 0.1 -word_size 11 -max_target_seqs 1 -max_hsps 1 \ -gapopen 3 -gapextend 3 -perc_identity 50
So I started with 5 instances running 2 threads each, then 4 x 3, then 3 x 4 .. now I may as well just run one with maybe ten threads shrug I don't have flexibility with the parameters here (e.g. word_size 11) because I am replicating someone else's work.
The question is more about whether this memory usage is normal - just seems a lot for a relatively small database.
Perhaps but it is what it is. You are running it a certain way and the program seems to be using what it needs. Looks like you have tried doing possible variations so you are doing all the right things.