Question: GATK JointDiscovery Error - Java Heap Space?
gravatar for vctrm67
5 months ago by
vctrm6710 wrote:

I was running the JointDiscovery pipeline as a part of the GATK Best Practices pipeline. I am running this on many vcf files (~150) called by the HaplotypeCaller. I am getting this error:

23:32:16.900 INFO  ProgressMeter - Traversal complete. Processed 19816687 total variants in 613.7 minutes.
23:32:17.434 INFO  VariantDataManager - QD:      mean = 22.14    standard deviation = 6.21
23:32:18.366 INFO  VariantDataManager - MQRankSum:   mean = -0.01    standard deviation = 0.25
23:32:19.385 INFO  VariantDataManager - ReadPosRankSum:      mean = 0.08     standard deviation = 0.47
23:32:20.172 INFO  VariantDataManager - FS:      mean = 1.38     standard deviation = 3.42
23:32:20.947 INFO  VariantDataManager - MQ:      mean = 59.89    standard deviation = 2.23
23:32:21.711 INFO  VariantDataManager - SOR:     mean = 0.70     standard deviation = 0.22
23:32:22.473 INFO  VariantDataManager - DP:      mean = 6835.64  standard deviation = 3532.38
01:01:24.432 INFO  VariantDataManager - Annotations are now ordered by their information content: [DP, MQ, QD, SOR, MQRankSum, FS, ReadPosRankSum]
01:02:53.662 INFO  VariantDataManager - Training with 6999956 variants after standard deviation thresholding.
01:02:53.662 WARN  VariantDataManager - WARNING: Very large training set detected. Downsampling to 2500000 training variants.
01:05:04.968 INFO  VariantRecalibrator - Shutting down engine
[September 18, 2019 1:05:04 AM EDT] done. Elapsed time: 706.54 minutes.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.broadinstitute.hellbender.engine.GATKTool.doWork(
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(
    at org.broadinstitute.hellbender.Main.runCommandLineProgram(
    at org.broadinstitute.hellbender.Main.mainEntry(
    at org.broadinstitute.hellbender.Main.main(

I believe this is derived from an error earlier in the log, since the stderr gives the same Java heap space error:

[2019-09-16 19:05:59,50] [error] WorkflowManagerActor Workflow 9f7a01a4-0632-4817-8622-aa51e520abf1 failed (during ExecutingWorkflowState): Job JointGenotyping.SNPsVariantRecalibratorClassic:NA:1 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /path/to/stderr.

I have read past issues ( regarding this that may suggest it is a bug. It has pointed me to increasing the available heap memory through the primary command of -Xmx. Is this the way to do it? java -Xmx600G -Dconfig.file=' + re.sub('input.json', 'overrides.conf', input_json) + ' -jar ' + args.cromwell_path + ' run ' + re.sub('input.json', 'joint-discovery-gatk4.wdl', input_json) + ' -i ' + input_json

where I substitute in the corresponding config, json, and wdl files. Is 600G enough? Each vcf is around 6G large and since I have 150, does that mean I should be allocating more than 900G (6G x 150)?

By the way, I would post this to the GATK forums but it's taking too long to get a verified account.

gatk • 221 views
ADD COMMENTlink modified 5 months ago • written 5 months ago by vctrm6710

'600G' do you have a server with 600G of memory ?

ADD REPLYlink written 5 months ago by Pierre Lindenbaum126k

Yes. But I'm not sure where the error is originating from (ie. is it a subprocess call in the wdl file?). I'm confused because it says Runtime.totalMemory()=3208118272, which is only ~3G, but I definitely specified more memory allocation than that.

I also looked at the task SNPsVariantRecalibrator memory allocation and it's more than 3G as well, but I'm not sure if this is the problem or how high I should be making it.

ADD REPLYlink modified 5 months ago • written 5 months ago by vctrm6710

Probably does not need 600G, since it only used about 3G. I would start with specifying 32G~64G.

ADD REPLYlink written 5 months ago by Vitis2.3k

But why would it error out even if I overspecify?

ADD REPLYlink written 5 months ago by vctrm6710

Did it give you the same "heap space" message and "RuntimeTotalMemory" after you specified 600G and ran the command?

ADD REPLYlink written 5 months ago by Vitis2.3k

Yes, I got the same message

ADD REPLYlink written 5 months ago by vctrm6710

Then I don't understand. Could you post this to GATK help forum?

ADD REPLYlink written 5 months ago by Vitis2.3k
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1301 users visited in the last hour