BBDuk problem: maskMiddle was disabled because useShortKmers=true

Question

[raniai]

Hi,

I'm trying to trim adapter from Illumina metagenomics reads using bbduk.sh. The code worked well with some samples, but then it gave me an error with others.

The command I used was (my reads are paired):

./bbduk.sh in1=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R1.fastq in2=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R2.fastq out1=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R1_clean1.fastq out2=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R2_clean2.fastq ref=./resources/adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo

And here is the message I got:

java -ea -Xmx141665m -Xms141665m -cp /data/bbmap/current/ jgi.BBDuk in1=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R1.fastq in2=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R2.fastq out1=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R1_clean1.fastq out2=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R2_clean2.fastq ref=./resources/adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo
Executing jgi.BBDuk [in1=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R1.fastq, in2=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R2.fastq, out1=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R1_clean1.fastq, out2=/data/KC12a_HW3M3DSX3_GTCTAATGGC-CCTGACCACT_L002_R2_clean2.fastq, ref=./resources/adapters.fa, ktrim=r, k=23, mink=11, hdist=1, tpe, tbo]
Version 39.01

maskMiddle was disabled because useShortKmers=true
0.017 seconds.
Initial:
Memory: max=148579m, total=148579m, free=148511m, used=68m

Added 217135 kmers; time:   0.100 seconds.
Memory: max=148579m, total=148579m, free=148360m, used=219m

Input is being processed as paired
Started output streams: 0.006 seconds.
Exception in thread "Thread-17" java.lang.RuntimeException: java.io.IOException: Disk quota exceeded
    at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:32)
Caused by: java.io.IOException: Disk quota exceeded
    at java.base/java.io.FileOutputStream.writeBytes(Native Method)
    at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
    at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
    at stream.ReadStreamByteWriter.writeFastq(ReadStreamByteWriter.java:468)
    at stream.ReadStreamByteWriter.processJobs(ReadStreamByteWriter.java:97)
    at stream.ReadStreamByteWriter.run2(ReadStreamByteWriter.java:42)
    at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:28)
Exception in thread "Thread-16" java.lang.RuntimeException: java.io.IOException: Disk quota exceeded
    at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:32)
Caused by: java.io.IOException: Disk quota exceeded
    at java.base/java.io.FileOutputStream.writeBytes(Native Method)
    at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
    at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
    at stream.ReadStreamByteWriter.writeFastq(ReadStreamByteWriter.java:452)
    at stream.ReadStreamByteWriter.processJobs(ReadStreamByteWriter.java:97)
    at stream.ReadStreamByteWriter.run2(ReadStreamByteWriter.java:42)
    at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:28)
Exception in thread "Thread-25" java.lang.RuntimeException: Writing to a terminated thread.
    at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:202)
    at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:197)
    at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
    at jgi.BBDuk$ProcessThread.run(BBDuk.java:3131)
Exception in thread "Thread-21" java.lang.RuntimeException: Writing to a terminated thread.
    at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:202)
    at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:197)
    at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
Nodes            : 1

Would you please help me to solve this issue?

Many thanks Rania

Answer 1

[Matthias Zepper]

The maskMiddle was disabled because useShortKmers=true is purely informative and tells you, that you have chosen incompatible parameters.

The true error is

Java.io.IOException: Disk quota exceeded at stream.ReadStreamByteWriter [...]

So you exceeded your hard drive capacity, possibly because of large temporary files.

[raniai]

Thanks, Matthias. I've checked my memory quota and it turned out that I exceeded it.

[colindaven]

Disk is NOT memory, RAM is memory. Please learn the difference, it will help you a lot.