Question: malt-build and Java heap Space
0
gravatar for stu111392
4.3 years ago by
stu11139230
Germany/Kiel
stu11139230 wrote:

Hey everybody,

at the moment I try to index a bunch of sequences with malt. I've indexed all viral proteins from refeseq without a problem. But now I wanted to index all bacterial ref-seq sequences at once. The file is about 9.8gb great and contains 5242 sequences. I used the command:

./malt-build -L /ifs/data/nfs_share/sukmb241/home.julian/malt/MEGAN5-academic-license.txt --input references/bacteriall_nucleotide --sequenceType DNA --index index/bacteriall_nr/

But after allocating the hash table the program spits out the following error: 

Allocating: 75.6 GB
java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapByteBuffer.<init>(Unknown Source)
        at java.nio.ByteBuffer.allocate(Unknown Source)
        at jloda.E.H.<init>(Unknown Source)
        at jloda.E.D.<init>(Unknown Source)
        at malt.data.O.A(Unknown Source)
        at malt.MaltBuild.run(Unknown Source)
        at malt.MaltBuild.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
        at com.install4j.runtime.launcher.UnixLauncher.main(Unknown Source)

I changed the -Xmx parameter in the vmoptions files as described in the manual and my last try to get this to work was with 120g but it crashed with the exact same error message as above. At the moment I have no way to increase the memory level. Has anybody experience with the indexing of greater files in Malt? Did you encounter the same problem? 

With best regards and thanks in advance,

Julian

ADD COMMENTlink modified 4.3 years ago • written 4.3 years ago by stu11139230
0
gravatar for RamRS
4.3 years ago by
RamRS23k
Houston, TX
RamRS23k wrote:

The manual here: http://ab.inf.uni-tuebingen.de/data/software/malt/download/manual.pdf

says that maximum memory availability is set at setup time. 

Quote from manual:

The most important performance-related option is the maximum amount of memory that malt-build is allowed to use. This cannot be set from within the program but rather is set during installation of the software

There also seems to be another option:

The installation dialog will ask how much memory the program may use. Please set this variable carefully. If the amount needs to be changed after installation, then this can be done by editing the files ending on vmoptions in the installation directory. 

Please, always read the manual before asking on forums.

ADD COMMENTlink written 4.3 years ago by RamRS23k
0
gravatar for stu111392
4.3 years ago by
stu11139230
Germany/Kiel
stu11139230 wrote:

I read the manual carefully and adjusted the memory parameters  in vmop files up to 100gb RAM. This was allready done and I therefor I asked the question here. So has anyone out there a helpfull answer? 

ADD COMMENTlink written 4.3 years ago by stu11139230
  1. You did not mention that you already tried this. If you did, I would not have suggested this.
  2. There's no need to be dismissive. Even if my answer were not relevant (which is not the case here because of point #1), you should acknowledge it better.
  3. Your response belongs in a comment to my answer, not an answer of its own.
ADD REPLYlink written 4.3 years ago by RamRS23k
0
gravatar for Brian Bushnell
4.3 years ago by
Walnut Creek, USA
Brian Bushnell16k wrote:

If you have enough memory, you might try something this:

export JAVA_OPTIONS="-Xmx200g"

Not sure about malt but in general it will override any subsequent attempt to set the -Xmx flag.

ADD COMMENTlink written 4.3 years ago by Brian Bushnell16k

Thank you for that. I´ll try it out tomorrow. 

ADD REPLYlink written 4.3 years ago by stu11139230
0
gravatar for stu111392
4.3 years ago by
stu11139230
Germany/Kiel
stu11139230 wrote:

And maybe let me rephrase the question a bit. I´am not entierly sure if this is just a pure problem of RAM or a Java error and therefor I postet the question here. Because even if I give the task 100gb of RAM it says after the hash table that 75,6GB are allocated. And therefor I thought it has another reason. 

ADD COMMENTlink written 4.3 years ago by stu11139230

Java does not use 100% of the memory for data; a large fraction is reserved for garbage-collection.  Being able to use 75% sounds pretty typical.

ADD REPLYlink written 4.3 years ago by Brian Bushnell16k

So your suggestion would be to increase the overall amount of memory?

ADD REPLYlink written 4.3 years ago by stu11139230

Unless you have a reason not to, just give Java all the physical memory of the machine. "java.lang.OutOfMemoryError: Java heap space" means that it was not given enough memory.

ADD REPLYlink written 4.3 years ago by Brian Bushnell16k

Edit your question and add all relevant information in there please. People will then not have to read through all the answers and comments to give you appropriate input.

ADD REPLYlink written 4.3 years ago by RamRS23k

I changed it. Hope that the problem is clearer now.

ADD REPLYlink written 4.3 years ago by stu11139230
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1541 users visited in the last hour