Trinity Inchworm Error - what does it mean?

Question

[sierraallinone]

I ran into this problem/error but I can't figure out the problem. We have 237G left of memory on our attached storage volume (where all the output goes). Is this a RAM limitation?

If it indicates bad_alloc(), then Inchworm ran out of memory. You'll need to either reduce the size of your data set or run Trinity on a server with more memory available.

The inchworm process failed.warning, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/../../Trinity --single "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_51/c5142.trinity.reads.fa" --output "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_51/c5142.trinity.reads.fa.out" --CPU 1 --max_memory 1G --run_as_paired --SS_lib_type F --seqType fa --trinity_complete --full_cleanup --group_pairs_distance 1000   failed with ret: 256, going to retry.
warning, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/../../Trinity --single "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_129/c12904.trinity.reads.fa" --output "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_129/c12904.trinity.reads.fa.out" --CPU 1 --max_memory 1G --run_as_paired --SS_lib_type F --seqType fa --trinity_complete --full_cleanup --group_pairs_distance 1000   failed with ret: 256, going to retry.
succeeded(0), failed(2)   100% completed.    

We are sorry, commands in file: [FailedCommands] failed.  :-( 

Error, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/trinity-plugins/BIN/ParaFly -c recursive_trinity.cmds -CPU 64 -v -shuffle  died with ret 256 at Trinity line 2581.
        main::process_cmd("/home/ubuntu/programs/Trinityrnaseq-v2.6.6/trinity-plugins/BI"...) called at Trinity line 3244
        main::run_partitioned_cmds("recursive_trinity.cmds") called at Trinity line 2239
        main::run_recursive_trinity("/media/trinity_out/squid_trinity_out/squid_trinity_out_2/chry"...) called at Trinity line 2001
        main::run_chrysalis("/media/trinity_out/squid_trinity_out/squid_trinity_out_2/inch"..., "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/both.fa", 200, 1000, "RF", "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/both.fa", "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/both.fa") called at Trinity line 1664
        main::run_Trinity() called at Trinity line 1317
        eval {...} called at Trinity line 1316

Trinity run failed. Must investigate error above.
ubuntu@ip-172-31-4-70:/media/scripts$

[Ram]

Inchworm is a RAM hog. Trinity FAQs state that. They also recommend 1G RAM per 1M reads if I recall correctly.

We have 237G left of memory on our attached storage volume

237G of storage space or 237G of memory (as in RAM)? Please be specific.

You'll also need to specify the exact command you ran as well as the volume of your data, and the specs on the compute node you ran it on.

[sierraallinone]

Here's the script:

 #!/bin/bash
#This is a program to execute Trinity v2.6.6 using previously output trimmed files 
cd /home/ubuntu/programs/Trinityrnaseq-v2.6.6

perl Trinity --seqType fq --max_memory 976G \
    --left /media/trinity_out/squid_trinity_out/Arm_WW_6_1_ACAGTG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Funnel_WW_6_1_CGATGT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Head_WW_6_1_TGACCA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_TT_7_1_CAGATC.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_UU_7_1_ACTTGA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_WW_7_1_CTTGTA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_TT_7_1_TGACCA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_UU_7_1_ACAGTG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_WW_7_1_GCCAAT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_TT_7_1_ATCACG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_UU_7_1_CGATGT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_WW_7_1_TTAGGC.fastq.gz.P.qtrim.gz \
    --right /media/trinity_out/squid_trinity_out/Arm_WW_6_2_ACAGTG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Funnel_WW_6_2_CGATGT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Head_WW_6_2_TGACCA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_TT_7_2_CAGATC.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_UU_7_2_ACTTGA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_WW_7_2_CTTGTA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_TT_7_2_TGACCA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_UU_7_2_ACAGTG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_WW_7_2_GCCAAT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_TT_7_2_ATCACG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_UU_7_2_CGATGT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_WW_7_2_TTAGGC.fastq.gz.P.qtrim.gz \
    --SS_lib_type RF --CPU 64 --normalize_by_read_set \
    --group_pairs_distance 1000 \
    --output /media/trinity_out/squid_trinity_out/squid_trinity_out_2

We have paired-end Illumina reads, 12 samples so 24 files in all. They are each around 20 million reads per file. We ran it on an AWS EC2, x1.16xlarge instance with 976G RAM and 64 CPUs. We have an attached storage volume where dump all of our output (500G) and that is what we have 237G of space left on. It's possible we're running out of RAM it just seems crazy. Especially since normalization has already completed.

[Ram]

I'd recommend you edit your question and add this information in there. Also, for context, point to your previous question as it has quite a bit of relevant information.

Answer 1

[sierraallinone]

So it turns out that this was a different error (Didn't scroll up far enough to see where it began--was using tmux)

Here is the beginning of the error message:

Number of Commands: 2
Use of uninitialized value $pm_temp in division (/) at /home/ubuntu/programs/Trinityrnaseq-v2$
6.6/util/support_scripts/../../Trinity line 1506.
Use of uninitialized value $pm_temp in division (/) at /home/ubuntu/programs/Trinityrnaseq-v2$
6.6/util/support_scripts/../../Trinity line 1506.
sh: 1: cannot open /media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/F$
_0/CBin_51/c5142.trinity.reads.fa.out/single.fa: No such file
Argument "" isn't numeric in division (/) at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/$
upport_scripts/../../Trinity line 1529.
sh: 1: cannot open /media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/F$
_0/CBin_129/c12904.trinity.reads.fa.out/single.fa: No such file
Argument "" isn't numeric in division (/) at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/$
upport_scripts/../../Trinity line 1529.
mv: cannot stat '/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_$
/CBin_51/c5142.trinity.reads.fa.out/inchworm.K25.L25.fa.tmp': No such file or directory
Error, cmd: mv /media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/$
Bin_51/c5142.trinity.reads.fa.out/inchworm.K25.L25.fa.tmp /media/trinity_out/squid_trinity_ou$
/squid_trinity_out_2/read_partitions/Fb_0/CBin_51/c5142.trinity.reads.fa.out/inchworm.K25.L25$
fa 2>tmp.81843.stderr died with ret 256 at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/PerlLib$
Pipeliner.pm line 166.
        Pipeliner::run(Pipeliner=HASH(0x2a3d428)) called at /home/ubuntu/programs/Trinityrnas$
q-v2.6.6/util/support_scripts/../../Trinity line 2409 eval {...} called at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/.
./../Trinity line 2404
        main::run_inchworm("/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read"...,
 "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read"..., "F", "") called at /home/
ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/../../Trinity line 1608
        main::run_Trinity() called at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_
scripts/../../Trinity line 1317
        eval {...} called at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/.
./../Trinity line 1316

and there were two CBin folders that it mentioned

"cannot open /media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/F$
_0/CBin_51/c5142.trinity.reads.fa.out/single.fa: No such file" so file "/c5142.trinity.reads.fa.out: within the CBin_51 folder and then "sh: 1: cannot open /media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/F$
_0/CBin_129/c12904.trinity.reads.fa.out/single.fa: No such file" so file "c12904.trinity.reads.fa.out/single.fa:"

within the CBin_129 folder. We went ahead and deleted those two file folders and restarted trinity and was able to complete it successfully. Turns out there was a corruption in those two output files.