Question: Trinity Error encountered with thread, terminated abnormally, not all specified records have been retrieved.
0
gravatar for sierraallinone
29 days ago by
sierraallinone10 wrote:

We had raw read files (20 million reads per file, paired end, 12 samples so 24 files in total) that we processed through trimmomatic within trinity and we are coming up with this error. From what I have read it seems to be a RAM issue. We have it set to 976G of RAM and 64CPU. We have tried looking at memory requirements and get differing opinions. One says we should have enough the other if true, says we would need 1,200G.

Done converting input files.Thread 3 terminated abnormally: Error, not all specified records have been retrieved (missing 107265) from /media/trinity_out/squid_trinity_out/Arm_WW_6_1_ACAGTG.fastq.gz.PwU.qtrim.fq, see file: /media/trinity_out/squid_trinity_out/norm_for_read_set_1/Arm_WW_6_1_ACAGTG.fastq.gz.PwU.qtrim.fq.normalized_K25_maxC50_minC1_pctSD10000.fq.missing_accs for list of missing entries at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/insilico_read_normalization.pl line 552.
Error encountered with thread.
Thread 4 terminated abnormally: Error, not all specified records have been retrieved (missing 107265) from /media/trinity_out/squid_trinity_out/Arm_WW_6_2_ACAGTG.fastq.gz.PwU.qtrim.fq, see file: /media/trinity_out/squid_trinity_out/norm_for_read_set_1/Arm_WW_6_2_ACAGTG.fastq.gz.PwU.qtrim.fq.normalized_K25_maxC50_minC1_pctSD10000.fq.missing_accs for list of missing entries at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/insilico_read_normalization.pl line 552.
Error encountered with thread.
Error, at least one thread died at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/insilico_read_normalization.pl line 423.
Error, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/insilico_read_normalization.pl --seqType fq --JM 976G  --max_cov 50 --min_cov 1 --CPU 64 --output /media/trinity_out/squid_trinity_out/norm_for_read_set_1   --max_pct_stdev 10000  --SS_lib_type RF  --left Arm_WW_6_1_ACAGTG.fastq.gz.PwU.qtrim.fq --right Arm_WW_6_2_ACAGTG.fastq.gz.PwU.qtrim.fq --pairs_together --PARALLEL_STATS   died with ret 7424 at Trinity line 2581.
        main::process_cmd("/home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/insilico_read"...) called at Trinity line 3127
        main::normalize("/media/trinity_out/squid_trinity_out/norm_for_read_set_1", 50, ARRAY(0x2375c20), ARRAY(0x2375f98)) called at Trinity line 3051
        main::run_normalization(50, ARRAY(0x2150d98), ARRAY(0x2150dc8)) called at Trinity line 1294

script:

perl Trinity --seqType fq --max_memory 976G \
    --left /media/reads/SquidRawReads/Arm_WW_6_1_ACAGTG.fastq.gz,/media/reads/SquidRawReads/Funnel_WW_6_1_CGATGT.fastq.gz,/media/reads/SquidRawReads/Head_WW_6_1_TGACCA.fastq.gz,/media/reads/SquidRawReads/Keel_TT_7_1_CAGATC.fastq.gz,/media/reads/SquidRawReads/Keel_UU_7_1_ACTTGA.fastq.gz,/media/reads/SquidRawReads/Keel_WW_7_1_CTTGTA.fastq.gz,/media/reads/SquidRawReads/Mantle_TT_7_1_TGACCA.fastq.gz,/media/reads/SquidRawReads/Mantle_UU_7_1_ACAGTG.fastq.gz,/media/reads/SquidRawReads/Mantle_WW_7_1_GCCAAT.fastq.gz,/media/reads/SquidRawReads/Nuchal_TT_7_1_ATCACG.fastq.gz,/media/reads/SquidRawReads/Nuchal_UU_7_1_CGATGT.fastq.gz,/media/reads/SquidRawReads/Nuchal_WW_7_1_TTAGGC.fastq.gz \
    --right /media/reads/SquidRawReads/Arm_WW_6_2_ACAGTG.fastq.gz,/media/reads/SquidRawReads/Funnel_WW_6_2_CGATGT.fastq.gz,/media/reads/SquidRawReads/Head_WW_6_2_TGACCA.fastq.gz,/media/reads/SquidRawReads/Keel_TT_7_2_CAGATC.fastq.gz,/media/reads/SquidRawReads/Keel_UU_7_2_ACTTGA.fastq.gz,/media/reads/SquidRawReads/Keel_WW_7_2_CTTGTA.fastq.gz,/media/reads/SquidRawReads/Mantle_TT_7_2_TGACCA.fastq.gz,/media/reads/SquidRawReads/Mantle_UU_7_2_ACAGTG.fastq.gz,/media/reads/SquidRawReads/Mantle_WW_7_2_GCCAAT.fastq.gz,/media/reads/SquidRawReads/Nuchal_TT_7_2_ATCACG.fastq.gz,/media/reads/SquidRawReads/Nuchal_UU_7_2_CGATGT.fastq.gz,/media/reads/SquidRawReads/Nuchal_WW_7_2_TTAGGC.fastq.gz \
    --SS_lib_type RF --CPU 64 \
    --trimmomatic --quality_trimming_params "ILLUMINACLIP:$TRIMMOMATIC_DIR/adapters/TruSeq3-PE-2.fa:2:30:10 SLIDINGWINDOW:4:10 LEADING:5 TRAILING:5 MINLEN:25" \
    --normalize_by_read_set --group_pairs_distance 1000 --output /media/trinity_out/squid_trinity_out
ADD COMMENTlink modified 12 days ago by Biostar ♦♦ 20 • written 29 days ago by sierraallinone10

EDIT: Ignore that, my knowledge is outdated - turns out Trinity normalizes by default these days (if you're using recent versions)

If I recall correctly, trinity has a normalization algorithm that extracts a representative set of reads from your overall reads and uses that to extrapolate metrics, thereby reducing computational costs a lot. Have you considered using that?

ADD REPLYlink modified 28 days ago • written 28 days ago by Ram17k

This error is often associated with badly formatted fastq headers. Could you post the output of:

for i in /media/reads/SquidRawReads/*_1_??????.fastq.gz
do
    echo $i:
    zcat $i | head -n1
done

For more background, search for Error, not all specified records have been retrieved.

ADD REPLYlink modified 28 days ago • written 28 days ago by h.mon18k

I was able to get it going again after I deleted some files off of the volume. The trimming was done within Trinity.

ADD REPLYlink written 26 days ago by sierraallinone10
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1570 users visited in the last hour