Trinity Error encountered with thread, terminated abnormally, not all specified records have been retrieved.

Question

[sierraallinone]

We had raw read files (20 million reads per file, paired end, 12 samples so 24 files in total) that we processed through trimmomatic within trinity and we are coming up with this error. From what I have read it seems to be a RAM issue. We have it set to 976G of RAM and 64CPU. We have tried looking at memory requirements and get differing opinions. One says we should have enough the other if true, says we would need 1,200G.

Done converting input files.Thread 3 terminated abnormally: Error, not all specified records have been retrieved (missing 107265) from /media/trinity_out/squid_trinity_out/Arm_WW_6_1_ACAGTG.fastq.gz.PwU.qtrim.fq, see file: /media/trinity_out/squid_trinity_out/norm_for_read_set_1/Arm_WW_6_1_ACAGTG.fastq.gz.PwU.qtrim.fq.normalized_K25_maxC50_minC1_pctSD10000.fq.missing_accs for list of missing entries at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/insilico_read_normalization.pl line 552.
Error encountered with thread.
Thread 4 terminated abnormally: Error, not all specified records have been retrieved (missing 107265) from /media/trinity_out/squid_trinity_out/Arm_WW_6_2_ACAGTG.fastq.gz.PwU.qtrim.fq, see file: /media/trinity_out/squid_trinity_out/norm_for_read_set_1/Arm_WW_6_2_ACAGTG.fastq.gz.PwU.qtrim.fq.normalized_K25_maxC50_minC1_pctSD10000.fq.missing_accs for list of missing entries at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/insilico_read_normalization.pl line 552.
Error encountered with thread.
Error, at least one thread died at /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/insilico_read_normalization.pl line 423.
Error, cmd: /home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/insilico_read_normalization.pl --seqType fq --JM 976G  --max_cov 50 --min_cov 1 --CPU 64 --output /media/trinity_out/squid_trinity_out/norm_for_read_set_1   --max_pct_stdev 10000  --SS_lib_type RF  --left Arm_WW_6_1_ACAGTG.fastq.gz.PwU.qtrim.fq --right Arm_WW_6_2_ACAGTG.fastq.gz.PwU.qtrim.fq --pairs_together --PARALLEL_STATS   died with ret 7424 at Trinity line 2581.
        main::process_cmd("/home/ubuntu/programs/Trinityrnaseq-v2.6.6/util/insilico_read"...) called at Trinity line 3127
        main::normalize("/media/trinity_out/squid_trinity_out/norm_for_read_set_1", 50, ARRAY(0x2375c20), ARRAY(0x2375f98)) called at Trinity line 3051
        main::run_normalization(50, ARRAY(0x2150d98), ARRAY(0x2150dc8)) called at Trinity line 1294

script:

perl Trinity --seqType fq --max_memory 976G \
    --left /media/reads/SquidRawReads/Arm_WW_6_1_ACAGTG.fastq.gz,/media/reads/SquidRawReads/Funnel_WW_6_1_CGATGT.fastq.gz,/media/reads/SquidRawReads/Head_WW_6_1_TGACCA.fastq.gz,/media/reads/SquidRawReads/Keel_TT_7_1_CAGATC.fastq.gz,/media/reads/SquidRawReads/Keel_UU_7_1_ACTTGA.fastq.gz,/media/reads/SquidRawReads/Keel_WW_7_1_CTTGTA.fastq.gz,/media/reads/SquidRawReads/Mantle_TT_7_1_TGACCA.fastq.gz,/media/reads/SquidRawReads/Mantle_UU_7_1_ACAGTG.fastq.gz,/media/reads/SquidRawReads/Mantle_WW_7_1_GCCAAT.fastq.gz,/media/reads/SquidRawReads/Nuchal_TT_7_1_ATCACG.fastq.gz,/media/reads/SquidRawReads/Nuchal_UU_7_1_CGATGT.fastq.gz,/media/reads/SquidRawReads/Nuchal_WW_7_1_TTAGGC.fastq.gz \
    --right /media/reads/SquidRawReads/Arm_WW_6_2_ACAGTG.fastq.gz,/media/reads/SquidRawReads/Funnel_WW_6_2_CGATGT.fastq.gz,/media/reads/SquidRawReads/Head_WW_6_2_TGACCA.fastq.gz,/media/reads/SquidRawReads/Keel_TT_7_2_CAGATC.fastq.gz,/media/reads/SquidRawReads/Keel_UU_7_2_ACTTGA.fastq.gz,/media/reads/SquidRawReads/Keel_WW_7_2_CTTGTA.fastq.gz,/media/reads/SquidRawReads/Mantle_TT_7_2_TGACCA.fastq.gz,/media/reads/SquidRawReads/Mantle_UU_7_2_ACAGTG.fastq.gz,/media/reads/SquidRawReads/Mantle_WW_7_2_GCCAAT.fastq.gz,/media/reads/SquidRawReads/Nuchal_TT_7_2_ATCACG.fastq.gz,/media/reads/SquidRawReads/Nuchal_UU_7_2_CGATGT.fastq.gz,/media/reads/SquidRawReads/Nuchal_WW_7_2_TTAGGC.fastq.gz \
    --SS_lib_type RF --CPU 64 \
    --trimmomatic --quality_trimming_params "ILLUMINACLIP:$TRIMMOMATIC_DIR/adapters/TruSeq3-PE-2.fa:2:30:10 SLIDINGWINDOW:4:10 LEADING:5 TRAILING:5 MINLEN:25" \
    --normalize_by_read_set --group_pairs_distance 1000 --output /media/trinity_out/squid_trinity_out

[Ram]

EDIT: Ignore that, my knowledge is outdated - turns out Trinity normalizes by default these days (if you're using recent versions)

If I recall correctly, trinity has a normalization algorithm that extracts a representative set of reads from your overall reads and uses that to extrapolate metrics, thereby reducing computational costs a lot. Have you considered using that?

[h.mon]

This error is often associated with badly formatted fastq headers. Could you post the output of:

for i in /media/reads/SquidRawReads/*_1_??????.fastq.gz
do
    echo $i:
    zcat $i | head -n1
done

For more background, search for Error, not all specified records have been retrieved.

[sierraallinone]

I was able to get it going again after I deleted some files off of the volume. The trimming was done within Trinity.