Error when running hhblits: Cannot allocate memory/Could not read index file...

Question

[nkuhuangyan]

Hi all,

I tried to run hhblits but it gives errors as: Could not read index file...

The command I run is:

hhblits -i input.fa -d /home/yanhuang/Desktop/hhsuite/database/uniclust30_2018_08_hhsuite/uniclust30_2018_08 -o output.hhr -norealign -n 3 -cov 80 -e 0.001

And the output is:

ffindex.c:452 ffindex_index_parse: mlock: Cannot allocate memory

    11:23:20.049 WARNING: In /build/hhsuite-rhvmkl/hhsuite-3.0~beta2+dfsg/src/hhdatabase.cpp:50: FFindexDatabase:

    11:23:20.049 WARNING: Could not read index file/home/yanhuang/Desktop/hhsuite/database/uniclust30_2018_08_hhsuite/uniclust30_2018_08_cs219.ffindex. Is the file empty or corrupted?

ffindex.c:452 ffindex_index_parse: mlock: Cannot allocate memory

    11:23:21.125 WARNING: In /build/hhsuite-rhvmkl/hhsuite-3.0~beta2+dfsg/src/hhdatabase.cpp:50: FFindexDatabase:

    11:23:21.125 WARNING: Could not read index file/home/yanhuang/Desktop/hhsuite/database/uniclust30_2018_08_hhsuite/uniclust30_2018_08_a3m.ffindex. Is the file empty or corrupted?

ffindex.c:452 ffindex_index_parse: mlock: Cannot allocate memory

    11:23:21.148 WARNING: In /build/hhsuite-rhvmkl/hhsuite-3.0~beta2+dfsg/src/hhdatabase.cpp:50: FFindexDatabase:

    11:23:21.148 WARNING: Could not read index file/home/yanhuang/Desktop/hhsuite/database/uniclust30_2018_08_hhsuite/uniclust30_2018_08_hhm.ffindex. Is the file empty or corrupted?

Segmentation fault (core dumped)

And when I tried uniclust30_2018_08 instead of uniclust30_2018_08_hhsuite, similar problem occurs:

ffindex.c:452 ffindex_index_parse: mlock: Cannot allocate memory
- 12:24:37.459 WARNING: In /build/hhsuite-rhvmkl/hhsuite-3.0~beta2+dfsg/src/hhdatabase.cpp:50: FFindexDatabase:

- 12:24:37.459 WARNING:     Could not read index file/home/yanhuang/Desktop/hhsuite/database/uniclust30_2018_08/uniclust30_2018_08_cs219.ffindex. Is the file empty or corrupted?

- 12:24:37.459 ERROR: In /build/hhsuite-rhvmkl/hhsuite-3.0~beta2+dfsg/src/hhdatabase.cpp:40: FFindexDatabase:

- 12:24:37.459 ERROR:   could not open file '/home/yanhuang/Desktop/hhsuite/database/uniclust30_2018_08/uniclust30_2018_08_a3m.ffindex'

The problem could be reproduced on multiple computers, from my desktop computer to my laptop. But one thing I noticed is that when running the same command on other database, everything will be OK.

For example:

hhblits -i input.fa -d /home/yanhuang/Desktop/hhsuite/database/pfamA_32.0/pfam -o output.hhr -norealign -n 3 -cov 80 -e 0.001

gives me everything I need. uniclust30_2018_08 and uniclust30_2018_08_hhsuite are the only ones that would give this problem.

For your information, I am using hhsuite version 3.0.0 (15-03-2015) on Ubuntu 18.04 and my RAM is 32GB. And I have installed it by git clone. Memory overcommit is enabled.

Really appreciate it for your help, thank you.

Answer 1

[Mensur Dlakic]

The two errors are not the same: you have memory warnings only with the first database you tried, while the second gives you ERROR: could not open file. Beyond the non-existing file I think your problem is literally what the warning says: 32 Gb may not enough to open recent HHblits databases. I know for sure that any of my 16 Gb computers can't be used with newest HHblits any more. Speaking of newest version, it may be a good idea to update to hhsuite that is more recent than the database you are trying to use.

Answer 2

[h.uzzaman]

Hi, I'm actually seeing the exact same error. The output just contains a bunch of lines saying "Cannot allocate memory" and "count not read index file...empty or corrupted?" exactly as the OP.

For the memory issue: I'm running hhblits_mpi on a compute cluster using 4 to 10 nodes with 96 GB memory each. For the "could not read index file" issue, I have no idea why the file can't be detected, because I'm using absolute paths, and using "ls" on that address does lead to the file.

Here's the code I'm running:

./hh-suite/bin/hhblits_mpi -i /path/to/pfam_databases/sample_fas -d /path/to/pfam_databases/UniRef30_2020_06 -oa3m sample_a3m_wo_ss -n 2 -cpu 68

The input file has just one amino acid sequence of moderate length.

Would be grateful for any help in this regard!

[Dunois]

You really just should be asking this over at their GitHub repo's issue tracker. Might have even been addressed already.

[h.uzzaman]

Thanks so much for your reply! I'll post my issues under the GitHub link (the comments that I see there haven't really addressed the issue yet).

[Mensur Dlakic]

Did you see that updating to a higher hhsuite version solved the problem?

I think this problem happens because I was used older version of hh-suite(3.0). I have been solved my problem by re-install this software with version 3.1

[h.uzzaman]

Yes I did- the hh-suite I'm running is version 3.3 (compiled from source), which is why I thought this solution doesn't apply in my case.

I've since run some additional experiments, and it seems the problem (memory allocation + unable to use index files) only happens when I run the MPI version of hhblits. Vanilla hhblits or hhblits_OMP don't show this error (even with nothing else changed). I'll make a post in the github forum after checking out a couple other things. But thanks so much for following up thus far!

[Dunois]

Did you set up the MPI properly following these suggestions?