Memory usage of picard Samsort
4
0
Entering edit mode
7.2 years ago
boymin2020 ▴ 80

I want to use SortSam of picard to coordinate sort and index the individual WGS data (e.g. 73G). The memory size of one node in our cluster is near 126.2G. I even set the memory size in qsub command to 110G and distributed 100G to java, it still gave me the following error message : Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: java.io.IOException: No space left on device

What should I do ? Any advice will be appreciated.

Picard SortSam • 5.4k views
ADD COMMENT
0
Entering edit mode

I have fixed the problem by adding the optional parameter TMP_DIR. Thanks for help.

ADD REPLY
1
Entering edit mode
7.2 years ago
  • use the classical samtools
  • or enhance the memory of java using -Xmx
  • or/and change the parameters of picard like MAX_RECORDS_IN_RAM

When writing SAM files that need to be sorted, this will specify the number of records stored in RAM before spilling to disk. Increasing this number reduces the number of file handles needed to sort a SAM file, and increases the amount of RAM needed. Default value: 500000. This option can be set to 'null' to clear the default value.

ADD COMMENT
0
Entering edit mode

, it still gave me the following error message : Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: java.io.IOException:

what @dariober said.

Use the standard option

TMP_DIR=

to change the default dir

ADD REPLY
1
Entering edit mode
7.2 years ago

java.io.IOException: No space left on device

I don't think the problem is with the memory but rather with the disk space. Picard writes temporary files to /tmp which usually is not a very large partition and it gets filled up easily. Try setting the TMP_DIR option to somewhere else, e.g. the current working directory (which is where samtools sort writes its tmp files).

ADD COMMENT
0
Entering edit mode
7.2 years ago

Simple not-very-clever solution...use grep or something to split out the files by chromosome, sort those individually, then cat them back together (with the orignal headers)

ADD COMMENT
0
Entering edit mode
5 weeks ago

For me, setting TMP_DIR (--TMP_DIR /home/tmp) didn't work in picard 2.21.1. What solved the issue was setting the environmental variable : export _JAVA_OPTIONS=-Djava.io.tmpdir=/home/tmp .

ADD COMMENT
0
Entering edit mode

Guys, just use samtools sort which won't get you into trouble with any of that.

ADD REPLY

Login before adding your answer.

Traffic: 1234 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6