Samtools Sort Set Max Memory , Value In Bytes, Kb, Mb Or Gb ?

Question

[William]

Does the samtools sort maximum memory parameter (m- ) require / accept values given in bytes, kB, MB of GB?

I would like to set the maximum memory parameter to 32 GB, so all the sorting of a bam chunk is done in memory. The default value is 500000000 something.

Answer 1

[eonsim]

If you use the most recent samtools version from the github repository you'll be able to set unit of memory and use threads, makes a massive difference!

https://github.com/samtools/samtools

$ samtools sort
Usage:   samtools sort [options] <in.bam> <out.prefix>
Options: -n        sort by read name
     -f        use <out.prefix> as full file name instead of prefix
     -o        final output to stdout
     -l INT    compression level, from 0 to 9 [-1]
     -@ INT    number of sorting and compression threads [1]
     -m INT    max memory per thread; suffix K/M/G recognized [768M]

[Pierre Lindenbaum]

ah you're right! The version I tested at home was the 'old' 1.18

[William]

Nice! I'll certainly try this. What effect does the compression level have? Sam format is not compressed? Does is mean compression to keep the memory use down?

[Pierre Lindenbaum]

the records stored in memory are uncompressed.

[William]

What is the option -l for then?

[Pierre Lindenbaum]

It is used when samtools is writing the data (not in memory) https://github.com/samtools/samtools/blob/develop/bam_sort.c#L500

[William]

What is the advantage / disadvantage of using it? If the data is not compressed in memory and also the output data is not compressed?

[Pierre Lindenbaum]

non compressed in memory: the sorting algorithm goes faster because it doesn't need to compress/uncompress each item. output data compressed=less storage needed. output data uncompressed=faster if a downstream program reads the data in a pipeline.

Answer 2

[Pierre Lindenbaum]

Does the samtools sort maximum memory parameter (m- ) require / accept values given in bytes, kB, MB of GB?

sort answer: no

$grep  "max_mem" bam_sort.c 

size_t max_mem = 500000000;
    case 'm': max_mem = atol(optarg); break;

but you can always use a bash inline multiplication:

samtools sort -m $((10*1024)) ....

[William]

Ehm. So a number of bytes is expected? Or is it a number of records?

[Pierre Lindenbaum]

it's a number of bytes:

  max_mem  approxiate maximum memory (very inaccurate)
  (...)
  buf = (bam1_t**)calloc(max_mem / BAM_CORE_SIZE, sizeof(bam1_t*));