Question: Internal memory allocator is full!
gravatar for cmckay
4.9 years ago by
cmckay0 wrote:

Hello, thank you for making the Minia software. I hope you can help me get it working with my dataset, which is 24GB (compressed) of illumina reads. The sample is an environmental metagenome, so is very complex. I selected an estimated_genome_size of 243000000000 not for biological reasons, but to try to compel minia to use a goodly portion of the 240GB ram on this machine. As you can see at the end, Minia crashed. If you have any input or ideas, I'd appreciate the help.

# minia all_paired.below.khmer.fasta.gz 31 3 243000000000 all_paired
estimated values: nbits Bloom 1466351878144, nb FP 3526388992, max memory 174802 MB
taille cell 16 
Available disk space in /mnt: 548951 MB
Sequentially counting ~409503 MB of kmers with 2 partition(s) and 5 passes using 1 thread(s), ~174802 MB of memory and ~95175 MB of disk space
| First step: Converting input file into Binary format                                               |
| Counting kmers                                                                                     |
100    %     elapsed:    262 min 24    sec      estimated remaining:      0 min 0     sec 

Saved 4571564210 solid kmers
-------------------Counted kmers time Wallclock  16624.1 s

------------------ Counted kmers and kept those with abundance >=3,     
nelem 4571564210 nbits 6.03437 
freesize 27586508800 estimated_BL1_freesize  3288.6 MB of memory for the main Bloom structure (6.03437 bits/kmer)
binary pass 
Insert solid Kmers in Bloom 4571560000
Inserted 4571564210 solid kmers in the bloom structure.
 Writing positive Bloom Kmers 4571560000
10588954139 kmers written
-------------------Write all positive kmers time Wallclock  8596.82 s
max size for this hash is 2^32, resuming with max value 
0 partitions will be needed
Build Hash table 4290770000Internal memory allocator is full!
Segmentation fault (core dumped)
ADD COMMENTlink modified 2.4 years ago by _r_am32k • written 4.9 years ago by cmckay0


I think you are using an old version of minia. You should have a try to the new one here (much faster with improved result quality); there are chances that you won't face the same memory issues with this new version.

ADD REPLYlink modified 2.4 years ago by _r_am32k • written 4.9 years ago by edrezen720

Edited to add "minia" tag.

ADD REPLYlink modified 2.4 years ago by _r_am32k • written 4.9 years ago by Dan D7.1k
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1816 users visited in the last hour