Question: Trinity Inchworm Error - what does it mean?
0
gravatar for sierraallinone
4 weeks ago by
sierraallinone10 wrote:

I ran into this problem/error but I can't figure out the problem. We have 237G left of memory on our attached storage volume (where all the output goes). Is this a RAM limitation?

If it indicates bad_alloc(), then Inchworm ran out of memory. You'll need to either reduce the size of your data set or run Trinity on a server with more memory available.

The inchworm process failed.warning, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/../../Trinity --single "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_51/c5142.trinity.reads.fa" --output "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_51/c5142.trinity.reads.fa.out" --CPU 1 --max_memory 1G --run_as_paired --SS_lib_type F --seqType fa --trinity_complete --full_cleanup --group_pairs_distance 1000   failed with ret: 256, going to retry.
warning, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/../../Trinity --single "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_129/c12904.trinity.reads.fa" --output "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/CBin_129/c12904.trinity.reads.fa.out" --CPU 1 --max_memory 1G --run_as_paired --SS_lib_type F --seqType fa --trinity_complete --full_cleanup --group_pairs_distance 1000   failed with ret: 256, going to retry.
succeeded(0), failed(2)   100% completed.    

We are sorry, commands in file: [FailedCommands] failed.  :-( 

Error, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/trinity-plugins/BIN/ParaFly -c recursive_trinity.cmds -CPU 64 -v -shuffle  died with ret 256 at Trinity line 2581.
        main::process_cmd("/home/ubuntu/programs/Trinityrnaseq-v2.6.6/trinity-plugins/BI"...) called at Trinity line 3244
        main::run_partitioned_cmds("recursive_trinity.cmds") called at Trinity line 2239
        main::run_recursive_trinity("/media/trinity_out/squid_trinity_out/squid_trinity_out_2/chry"...) called at Trinity line 2001
        main::run_chrysalis("/media/trinity_out/squid_trinity_out/squid_trinity_out_2/inch"..., "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/both.fa", 200, 1000, "RF", "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/both.fa", "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/both.fa") called at Trinity line 1664
        main::run_Trinity() called at Trinity line 1317
        eval {...} called at Trinity line 1316

Trinity run failed. Must investigate error above.
ubuntu@ip-172-31-4-70:/media/scripts$
error inchworm trinity rnaseq • 169 views
ADD COMMENTlink modified 27 days ago • written 4 weeks ago by sierraallinone10
1

Inchworm is a RAM hog. Trinity FAQs state that. They also recommend 1G RAM per 1M reads if I recall correctly.

We have 237G left of memory on our attached storage volume

237G of storage space or 237G of memory (as in RAM)? Please be specific.

You'll also need to specify the exact command you ran as well as the volume of your data, and the specs on the compute node you ran it on.

ADD REPLYlink written 4 weeks ago by Ram17k

Here's the script:

 #!/bin/bash
#This is a program to execute Trinity v2.6.6 using previously output trimmed files 
cd /home/ubuntu/programs/Trinityrnaseq-v2.6.6

perl Trinity --seqType fq --max_memory 976G \
    --left /media/trinity_out/squid_trinity_out/Arm_WW_6_1_ACAGTG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Funnel_WW_6_1_CGATGT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Head_WW_6_1_TGACCA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_TT_7_1_CAGATC.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_UU_7_1_ACTTGA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_WW_7_1_CTTGTA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_TT_7_1_TGACCA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_UU_7_1_ACAGTG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_WW_7_1_GCCAAT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_TT_7_1_ATCACG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_UU_7_1_CGATGT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_WW_7_1_TTAGGC.fastq.gz.P.qtrim.gz \
    --right /media/trinity_out/squid_trinity_out/Arm_WW_6_2_ACAGTG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Funnel_WW_6_2_CGATGT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Head_WW_6_2_TGACCA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_TT_7_2_CAGATC.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_UU_7_2_ACTTGA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Keel_WW_7_2_CTTGTA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_TT_7_2_TGACCA.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_UU_7_2_ACAGTG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Mantle_WW_7_2_GCCAAT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_TT_7_2_ATCACG.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_UU_7_2_CGATGT.fastq.gz.P.qtrim.gz,/media/trinity_out/squid_trinity_out/Nuchal_WW_7_2_TTAGGC.fastq.gz.P.qtrim.gz \
    --SS_lib_type RF --CPU 64 --normalize_by_read_set \
    --group_pairs_distance 1000 \
    --output /media/trinity_out/squid_trinity_out/squid_trinity_out_2

We have paired-end Illumina reads, 12 samples so 24 files in all. They are each around 20 million reads per file. We ran it on an AWS EC2, x1.16xlarge instance with 976G RAM and 64 CPUs. We have an attached storage volume where dump all of our output (500G) and that is what we have 237G of space left on. It's possible we're running out of RAM it just seems crazy. Especially since normalization has already completed.

ADD REPLYlink modified 29 days ago by Ram17k • written 4 weeks ago by sierraallinone10

I'd recommend you edit your question and add this information in there. Also, for context, point to your previous question as it has quite a bit of relevant information.

ADD REPLYlink written 29 days ago by Ram17k
0
gravatar for sierraallinone
27 days ago by
sierraallinone10 wrote:

So it turns out that this was a different error (Didn't scroll up far enough to see where it began--was using tmux)

Here is the beginning of the error message:

Number of Commands: 2
Use of uninitialized value $pm_temp in division (/) at /home/ubuntu/programs/Trinityrnaseq-v2$
6.6/util/support_scripts/../../Trinity line 1506.
Use of uninitialized value $pm_temp in division (/) at /home/ubuntu/programs/Trinityrnaseq-v2$
6.6/util/support_scripts/../../Trinity line 1506.
sh: 1: cannot open /media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/F$
_0/CBin_51/c5142.trinity.reads.fa.out/single.fa: No such file
Argument "" isn't numeric in division (/) at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/$
upport_scripts/../../Trinity line 1529.
sh: 1: cannot open /media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/F$
_0/CBin_129/c12904.trinity.reads.fa.out/single.fa: No such file
Argument "" isn't numeric in division (/) at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/$
upport_scripts/../../Trinity line 1529.
mv: cannot stat '/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_$
/CBin_51/c5142.trinity.reads.fa.out/inchworm.K25.L25.fa.tmp': No such file or directory
Error, cmd: mv /media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/Fb_0/$
Bin_51/c5142.trinity.reads.fa.out/inchworm.K25.L25.fa.tmp /media/trinity_out/squid_trinity_ou$
/squid_trinity_out_2/read_partitions/Fb_0/CBin_51/c5142.trinity.reads.fa.out/inchworm.K25.L25$
fa 2>tmp.81843.stderr died with ret 256 at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/PerlLib$
Pipeliner.pm line 166.
        Pipeliner::run(Pipeliner=HASH(0x2a3d428)) called at /home/ubuntu/programs/Trinityrnas$
q-v2.6.6/util/support_scripts/../../Trinity line 2409 eval {...} called at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/.
./../Trinity line 2404
        main::run_inchworm("/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read"...,
 "/media/trinity_out/squid_trinity_out/squid_trinity_out_2/read"..., "F", "") called at /home/
ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/../../Trinity line 1608
        main::run_Trinity() called at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_
scripts/../../Trinity line 1317
        eval {...} called at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/support_scripts/.
./../Trinity line 1316

and there were two CBin folders that it mentioned

"cannot open /media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/F$
_0/CBin_51/c5142.trinity.reads.fa.out/single.fa: No such file" so file "/c5142.trinity.reads.fa.out: within the CBin_51 folder and then "sh: 1: cannot open /media/trinity_out/squid_trinity_out/squid_trinity_out_2/read_partitions/F$
_0/CBin_129/c12904.trinity.reads.fa.out/single.fa: No such file" so file "c12904.trinity.reads.fa.out/single.fa:"

within the CBin_129 folder. We went ahead and deleted those two file folders and restarted trinity and was able to complete it successfully. Turns out there was a corruption in those two output files.

ADD COMMENTlink modified 27 days ago by genomax54k • written 27 days ago by sierraallinone10
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1457 users visited in the last hour