Picard SamFormatConverter gives JVM OutOfMemoryError

Question

[stachele]

I am getting an error upon running Picard SamFormatConverter: Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit

Looking in GATK and Biostars threads, this issue seems to come up frequently. The .sam file I start with is 67Gb, which I think is not unreasonably large.

I don't understand the memory requirements for java or where capacity is needed. I believe I have specified near the max memory for java processes by giving the command 20 gigs of memory ("-java -Xmx20g"). I am running all this on an Oracle-VM with Ubuntu 16.04. Windows and Ubuntu both say the computer has ~21 gigs of free RAM

Could there simply be an error in the script that is causing it to think there is not enough memory? Or is there some way to test how much memory is needed? How much memory would likely be enough to run Exome pipelines through Bwa, Picard, GATK, etc?

Thank you very much,

Answer 1

[patel.ravip]

Are you sure that you have allocated that much memory for your VM? i.e, you can have a machine with large memory, but if you don't allocate any of it to the VM in the settings, you could have this issue.

Picard is designed to use 2gb of ram for each task, so the -Xmx20g flag for java probably won't have an effect. Instead, if you are sure that you have given enough RAM to your VM, then you can try decreasing the number of MAX_RECORDS_IN_RAM (Picardtools flag) from the default.

EDIT: I misread your original post where you said your VM is Ubuntu and it also says 21GB of ram. I would recommend that you try changing the MAX_RECORDS_IN_RAM option to a smaller number for now, although if the machine and VM have access to 21GB of memory, this is probably not causing the problem. I will also caution that your VM usually shouldn't have access to all the RAM on a machine, as the background host OS still needs to perform tasks to keep the VM running. I have found about 75-80% of the total RAM (on big memory machines; should be less on smaller machines) to be a good upper limit for the VM.