I ran into this problem/error but I can't figure out the problem. We have 237G left of memory on our attached storage volume (where all the output goes). Is this a RAM limitation?
If it indicates bad_alloc(), then Inchworm ran out of memory. You'll need to either reduce the size of your data set or run Trinity on a server with more memory available.
The inchworm process failed.warning, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/../../Trinity --single "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_51/c5142.trinity.reads.fa" --output "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_51/c5142.trinity.reads.fa.out" --CPU 1 --max_memory 1G --run_as_paired --SS_lib_type F --seqType fa --trinity_complete --full_cleanup --group_pairs_distance 1000 failed with ret: 256, going to retry.
warning, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/../../Trinity --single "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_129/c12904.trinity.reads.fa" --output "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_129/c12904.trinity.reads.fa.out" --CPU 1 --max_memory 1G --run_as_paired --SS_lib_type F --seqType fa --trinity_complete --full_cleanup --group_pairs_distance 1000 failed with ret: 256, going to retry.
succeeded(0), failed(2) 100% completed.
We are sorry, commands in file: [FailedCommands] failed. :-(
Error, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/trinity-plugins/BIN/ParaFly -c recursive_trinity.cmds -CPU 64 -v -shuffle died with ret 256 at Trinity line 2581.
main::process_cmd("/home/ubuntu/programs/Trinityrnaseq-v2.6.6/trinity-plugins/BI"...) called at Trinity line 3244
main::run_partitioned_cmds("recursive_trinity.cmds") called at Trinity line 2239
main::run_recursive_trinity("/media/trinity_out/squid_trinity_out/squid_trinity_out_2/chry"...) called at Trinity line 2001
main::run_chrysalis("/media/trinity_out/squid_trinity_out/squid_trinity_out_2/inch"..., "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/both.fa", 200, 1000, "RF", "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/both.fa", "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/both.fa") called at Trinity line 1664
main::run_Trinity() called at Trinity line 1317
eval {...} called at Trinity line 1316
Trinity run failed. Must investigate error above.
ubuntu@ip-172-31-4-70:/media/scripts$
Inchworm is a RAM hog. Trinity FAQs state that. They also recommend 1G RAM per 1M reads if I recall correctly.
237G of storage space or 237G of memory (as in RAM)? Please be specific.
You'll also need to specify the exact command you ran as well as the volume of your data, and the specs on the compute node you ran it on.
Here's the script:
We have paired-end Illumina reads, 12 samples so 24 files in all. They are each around 20 million reads per file. We ran it on an AWS EC2, x1.16xlarge instance with 976G RAM and 64 CPUs. We have an attached storage volume where dump all of our output (500G) and that is what we have 237G of space left on. It's possible we're running out of RAM it just seems crazy. Especially since normalization has already completed.
I'd recommend you edit your question and add this information in there. Also, for context, point to your previous question as it has quite a bit of relevant information.