Trinity: how to avoid out of memory error in butterfly

Question

[Michael]

I am running Trinity on a high-mem/multi CPU server and am getting a large number of out-of memory errors in the butterfly step, the machine has 2TB of RAM and 143 CPU's.

This is how I am invoking Trinity:

 $TRINITYCMD  --trimmomatic --SS_lib_type RF --seqType fq --max_memory 1500G --left $1 --right $2 --CPU 140 --output ./Trinity-trimmomatic

As a work-around, I have to re-run the pipeline with lower CPU settings, sometimes several times, to clean up the remaining jobs, but that does not seem like the best way of handling it.

The errors I am getting look like out of memory, which is strange given the machine has 2TB of RAM:

We are sorry, commands in file: [failed_butterfly_commands.8603.txt] failed.  :-(

Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
...
Error, cmd: /net/apps/cbu/stow/trinityrnaseq-Trinity-v2.4.0/trinity-plugins/parafly/bin/ParaFly -c recursive_trinity.cmds -CPU 140 -v  died with ret 256 at /net/apps/cbu/stow/trinityrnaseq-Trinity-v2.4.0/Trinity line 2462.

## this is the txt file:

java -Xmx4G -Xms1G -Xss1G -XX:ParallelGCThreads=2  -jar /net/apps/cbu/stow/trinityrnaseq-Trinity-v2.4.0/Butterfly/Butterfly.jar -N 100000 -L 200 -F 500 -C /export/jonassenfs/michaeld/licebase/genomedata/Halicryptus_spinulosus/Trinity-trimmomatic/read_partitions/Fb_0/CBin_863/c86329.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c2.graph  --path_reinforcement_distance=25

[Tom_L]

Are you sure that the problem comes from how much RAM is required? I believe you ask too many cores to your server: unable to create new native thread (https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread). Most Trinity problems can be resolved by lowering CPU requirement (16-32 cores). Also, is the RAM free on your cluster? I mean you have 2Tb but are you the only one to use it?

[Michael]

Hi Tom, thank you very much, It works with 50, though the the server has 143 entries in /proc/cpuinfo, that means maybe in reality half the size of cores exist. I will check what the maximum number of threads is that I can start. However, I think this should be determined by the pipeline master, because I am providing the information on how many CPUs I have. From the link you sent it says:

64-bit Ubuntu Linux, Java 1.7.0_45 – JVM dies after #31893 threads have been created

[h.mon]

A common recommendation when running Trinity is to set the stack size to unlimited:

ulimit -s unlimited