I'm working on the last step of our lab's well established variant calling pipeline, running GATK GenotypeGVCFs on 4392 whole exome sequenced individuals. In the past I haven't had any problems with this sort of thing, but on this last run the job would be killed on the supercomputer cluster for using too much memory. Now it appears that even with allocating 16 threads and 64 GB of memory the log file predicts nearly 8 weeks of runtime remaining! I am using GATK 3.3 with the following arguments:
-T GenotypeGVCFs -R /projects/resources/Homo_sapiens_assembly19.fasta --variant /projects/combinedgvcfs/combined_gvcfs.list --dbsnp /projects/resources/gatk_bundle/dbsnp_138.b37.vcf -o /04_15_2016/genotype_gvcfs/04_15_2016_raw.vcf -log /04_15_2016/genotype_gvcfs/04_15_2016_raw.log -L /projects/resources/bed.and.interval.files/b37_refseqplus50_clean.bed -nt 16 --max_alternate_alleles 6
If anyone has any ideas please let me know because in the past this would take no more than 72 hours to run to completion. Let me know if I can provide any additional information to help.