Question: HISAT index incomplete even with HUGE RAM
0
gravatar for emcc
10 months ago by
emcc10
emcc10 wrote:

Hello,

I'm hoping someone can help. I'm attempting to create index files for a species not available on the pre-made list and continually getting errors associated with memory. I have allocated 480GB RAM (for a 1.3GB genome) on a HPC cluster and it produces 6 of the 8 ht2 files (file 5 and 6 missing) and all of the .rf files, which I realise are temporary and I should not see. I have successfully used this command with sample data. I can successfully create the splice sites and exons files (I do need this to be annotated). And the really frustrating thing is, a colleague can successfully finish this job using a lower memory allocation (and I have tried their exact script with my file names). I'm hoping there is something simple I am missing. Any help would be greatly appreciated.

The script I submit looks like this:

#!/bin/bash
#$ -N HS_Index
#$ -o /mnt/scratch/users/*******/file/jobs/create_HISAT2_index.$JOB_ID
#$ -M email@uni.ac.uk -m bea

#request 3 nodes with 160GB RAM each
#$ -pe mpinodes-verbose 3

#load hisat2 module environment
module load hisat2

# set working directory to scratch space project folder
#$ -wd /mnt/scratch/users/*******/file

hisat2-build -p 8 --ss genome.ss --exon genome.exon genome.fa genome_tran

The errors I get often look like this:

Settings:
  Output files: "genome_tran.*.ht2"
  Line rate: 7 (line is 128 bytes)
  Lines per side: 1 (side is 128 bytes)
  Offset rate: 4 (one in 16)
  FTable chars: 10
  Strings: unpacked
  Local offset rate: 3 (one in 8)
  Local fTable chars: 6
  Local sequence length: 57344
  Local sequence overlap between two consecutive indexes: 1024
  Endianness: little
  Actual local endianness: little
  Sanity checking: disabled
  Assertions: disabled
  Random seed: 0
  Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
  genome.fa
Reading reference sizes
  Time reading reference sizes: 00:00:10
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
  Time to join reference sequences: 00:00:05
  Time to read SNPs and splice sites: 00:00:07
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
  Ran out of memory; automatically trying more memory-economical parameters.
Could not find approrpiate bmax/dcv settings for building this index.
Switching to a packed string representation.
Total time for call to driver() for forward index: 01:00:28
==================================================
SGE job completed on 
==================================================
indexing index hisat • 920 views
ADD COMMENTlink modified 10 months ago by Devon Ryan86k • written 10 months ago by emcc10

I added code markup to your post for increased readability. You can do this by selecting the text and clicking the 101010 button. When you compose or edit a post that button is in your toolbar, see image below:

101010 Button

In addition, I changed this post to a "question" (rather than a "forum").

ADD REPLYlink written 10 months ago by WouterDeCoster35k

Requesting 3 nodes is a waste of 2 of them. Like most programs, hisat2 can only make use of a single node.

ADD REPLYlink written 10 months ago by Devon Ryan86k
0
gravatar for Devon Ryan
10 months ago by
Devon Ryan86k
Freiburg, Germany
Devon Ryan86k wrote:

You're requesting 3 cores, not 3 full nodes (not that hisat2 or any other common bioinformatics tool could use more than one node). Whatever your cluster defaults to for per-process memory is insufficient. Not being an SGE expert, I would presume the following would be more reasonable settings:

#$ -pe smp 8
#$ -l h_vmem=3G

That should result in 24GB of memory and 8 cores, if I remember SGE syntax correctly. You can, of course, increase the memory allocation up to what's available on a single node...though that'd be over-kill.

ADD COMMENTlink modified 10 months ago • written 10 months ago by Devon Ryan86k

On every SGE system I have worked on specifying h_vmem does not limit memory usage for the task, rather it tells the queue controller not to kill the job for using too much memory.

That is if you specify h_vmem=24GB using 25GB will not result in an out of memory error (as long as the memory exists on the node), rather your job will be terminated by the controller and the script would not return SGE job completed

My guess is that some other memory hungry job is running on the node requested, potentially using more memory that it requested (not all SGE installs actually enforce memory limits). As far as SGE is concerned the job is using less memory than the default max mem, but when the job actually comes to try and use it, it is not free.

ADD REPLYlink written 10 months ago by i.sudbery2.8k

Could be, I assumed the SGE implementation in use here was somehow actually limiting the memory availability, but your explanation seems more likely (we have our slurm controller set to kill jobs that use more memory than they request to prevent this sort of thing).

ADD REPLYlink written 10 months ago by Devon Ryan86k

Thank you Devon, I didn't realise about the nodes, I was kind of "throwing the kitchen sink at it" by requesting as much processing memory as possible as various other attempts have not worked.

I have used your exact instructions, and:

   #$ -pe smp-verbose 8
    #$ -l h_vmem=64G

And further variations of this to request more RAM, but I still get the same error message and an incomplete file set.

ADD REPLYlink modified 10 months ago • written 10 months ago by emcc10
1

Two things about your SGE spec there:

1) You are asking for 64GB per core and 8 cores. This means you are asking for 512GB. However, 2) you are also asking for this as virtual memory rather than physical memory, and HISAT can't use virtual memory.

Couple of things you might like to try: adding a free -g before you run HISAT - this should tell you how much physical memory is available on the node. Secondly add a ulimit -a, this will tell you if the OS is limiting the amount of RAM you can access.

If you need to increase the amount of physical RAM available, you need to ask your local SGE admins how to do this, I think it differs from system to system.

ADD REPLYlink written 10 months ago by i.sudbery2.8k

Thank you

I needed to request access to a 1TB node and the settings below then worked. Thanks again for your help :-)

>     > #$ -pe smp-verbose 8
>     > #$ -l h_vmem=125G
>     > #$ -l himem=true
ADD REPLYlink modified 9 months ago • written 9 months ago by emcc10
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1149 users visited in the last hour