Question: BBTools - Repair.sh bug (?)
0
gravatar for cacampbell
3.4 years ago by
cacampbell40
United States
cacampbell40 wrote:

Hello Brian Bushnell,

I get an ambiguous stack trace in 1/8 - 1/10 of repair.sh jobs (but not other tools), and I cannot seem to reliably reproduce the problem. Running the same jobs again on the same files only produces errors in the same files by rare coincidence, if at all.

Here is the full output:

java -ea -Xmx38G -cp /home/cacampbe/.prog/bbmap/current/ jgi.SplitPairsAndSingles rp in1=ErrCorrect/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R1.fq.gz in2=ErrCorrect/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R2.fq.gz out1=ErrCorrect_Repair/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R1.fq.gz out2=ErrCorrect_Repair/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R2.fq.gz -Xmx38G t=9
Executing jgi.SplitPairsAndSingles [rp, in1=ErrCorrect/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R1.fq.gz, in2=ErrCorrect/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R2.fq.gz, out1=ErrCorrect_Repair/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R1.fq.gz, out2=ErrCorrect_Repair/SM13_200/FCC5V31ANXX_L1_wHAXPI019566-83_R2.fq.gz, -Xmx38G, t=9]

Set threads to 9
Set INTERLEAVED to false
Exception in thread "main" java.lang.NoClassDefFoundError: java/lang/ProcessBuilder$Redirect
    at fileIO.ReadWrite.getOutputStreamFromProcess(ReadWrite.java:619)
    at fileIO.ReadWrite.getPigzStream(ReadWrite.java:550)
    at fileIO.ReadWrite.getGZipOutputStream(ReadWrite.java:521)
    at fileIO.ReadWrite.getOutputStream(ReadWrite.java:382)
    at fileIO.ReadWrite.getOutputStream(ReadWrite.java:346)
    at stream.ReadStreamWriter.<init>(ReadStreamWriter.java:69)
    at stream.ReadStreamByteWriter.<init>(ReadStreamByteWriter.java:17)
    at stream.ConcurrentGenericReadOutputStream.<init>(ConcurrentGenericReadOutputStream.java:39)
    at stream.ConcurrentReadOutputStream.getStream(ConcurrentReadOutputStream.java:53)
    at stream.ConcurrentReadOutputStream.getStream(ConcurrentReadOutputStream.java:27)
    at jgi.SplitPairsAndSingles.process2(SplitPairsAndSingles.java:287)
    at jgi.SplitPairsAndSingles.process(SplitPairsAndSingles.java:230)
    at jgi.SplitPairsAndSingles.main(SplitPairsAndSingles.java:45)
Caused by: java.lang.ClassNotFoundException: java.lang.ProcessBuilder$Redirect
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
    ... 13 more

Some (relevant?) background information:

  • Java 1.8

  • pigz

  • BBMap 35.92

The process then hangs seemingly indefinitely.

bbtools • 1.6k views
ADD COMMENTlink modified 3.4 years ago • written 3.4 years ago by cacampbell40

That's really strange, and I can't seem to replicate it. Are you using OpenJDK or the Oracle version of Java?

Either way, if you add these flags:

pigz=f unpigz=f

...the problem should go away. Please let me know if it doesn't.

It kind of sounds to me like sometimes an old version of Java is being used. For example, if you are on a cluster and the job is running on some random node, then there is one node with perhaps Java 1.6 and all the others have Java 1.8; or one node has pigz installed and the others don't. Basically, they only explanation I can imagine is that the environment is different for the jobs that fail compared to the ones that succeed.

ADD REPLYlink modified 3.4 years ago • written 3.4 years ago by Brian Bushnell17k

Hi Brian, I believe that the java problem was the issue.

We do use an environment module system, and I was loading java 1.8 for each of these jobs. Nonetheless, it appears that three nodes are unable to find the modulepath and thus default to java 1.6. I have notified our system administrators.

I didn't see this issue with other processes because the three nodes in question all have very little RAM. Reformat doesn't take much RAM, so it is the only process that ever ran on these nodes (of the BBTools that I have been running so far)

Thanks for the help!

ADD REPLYlink modified 3.4 years ago • written 3.4 years ago by cacampbell40

Ah. Modules are wonderful but when individual nodes misbehave it can be hard to track down .. specially when you are a regular user.

ADD REPLYlink written 3.4 years ago by genomax75k
2
gravatar for cacampbell
3.4 years ago by
cacampbell40
United States
cacampbell40 wrote:

Java version error due to nodes not finding modulepath -- see comments

ADD COMMENTlink written 3.4 years ago by cacampbell40
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1247 users visited in the last hour