Question: Trinity: how to avoid out of memory error in butterfly
1
gravatar for Michael Dondrup
21 months ago by
Bergen, Norway
Michael Dondrup46k wrote:

I am running Trinity on a high-mem/multi CPU server and am getting a large number of out-of memory errors in the butterfly step, the machine has 2TB of RAM and 143 CPU's.

This is how I am invoking Trinity:

 $TRINITYCMD  --trimmomatic --SS_lib_type RF --seqType fq --max_memory 1500G --left $1 --right $2 --CPU 140 --output ./Trinity-trimmomatic

As a work-around, I have to re-run the pipeline with lower CPU settings, sometimes several times, to clean up the remaining jobs, but that does not seem like the best way of handling it.

The errors I am getting look like out of memory, which is strange given the machine has 2TB of RAM:

We are sorry, commands in file: [failed_butterfly_commands.8603.txt] failed.  :-(

Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
...
Error, cmd: /net/apps/cbu/stow/trinityrnaseq-Trinity-v2.4.0/trinity-plugins/parafly/bin/ParaFly -c recursive_trinity.cmds -CPU 140 -v  died with ret 256 at /net/apps/cbu/stow/trinityrnaseq-Trinity-v2.4.0/Trinity line 2462.

## this is the txt file:

java -Xmx4G -Xms1G -Xss1G -XX:ParallelGCThreads=2  -jar /net/apps/cbu/stow/trinityrnaseq-Trinity-v2.4.0/Butterfly/Butterfly.jar -N 100000 -L 200 -F 500 -C /export/jonassenfs/michaeld/licebase/genomedata/Halicryptus_spinulosus/Trinity-trimmomatic/read_partitions/Fb_0/CBin_863/c86329.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c2.graph  --path_reinforcement_distance=25
ADD COMMENTlink modified 21 months ago • written 21 months ago by Michael Dondrup46k

Are you sure that the problem comes from how much RAM is required? I believe you ask too many cores to your server: unable to create new native thread (https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread). Most Trinity problems can be resolved by lowering CPU requirement (16-32 cores). Also, is the RAM free on your cluster? I mean you have 2Tb but are you the only one to use it?

ADD REPLYlink modified 21 months ago • written 21 months ago by Tom_L310

Hi Tom, thank you very much, It works with 50, though the the server has 143 entries in /proc/cpuinfo, that means maybe in reality half the size of cores exist. I will check what the maximum number of threads is that I can start. However, I think this should be determined by the pipeline master, because I am providing the information on how many CPUs I have. From the link you sent it says:

64-bit Ubuntu Linux, Java 1.7.0_45 – JVM dies after #31893 threads have been created

ADD REPLYlink modified 21 months ago • written 21 months ago by Michael Dondrup46k

A common recommendation when running Trinity is to set the stack size to unlimited:

ulimit -s unlimited

ADD REPLYlink written 21 months ago by h.mon26k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 2138 users visited in the last hour