Hello,
I'm hoping someone can help. I'm attempting to create index files for a species not available on the pre-made list and continually getting errors associated with memory. I have allocated 480GB RAM (for a 1.3GB genome) on a HPC cluster and it produces 6 of the 8 ht2 files (file 5 and 6 missing) and all of the .rf files, which I realise are temporary and I should not see. I have successfully used this command with sample data. I can successfully create the splice sites and exons files (I do need this to be annotated). And the really frustrating thing is, a colleague can successfully finish this job using a lower memory allocation (and I have tried their exact script with my file names). I'm hoping there is something simple I am missing. Any help would be greatly appreciated.
The script I submit looks like this:
#!/bin/bash
#$ -N HS_Index
#$ -o /mnt/scratch/users/*******/file/jobs/create_HISAT2_index.$JOB_ID
#$ -M email@uni.ac.uk -m bea
#request 3 nodes with 160GB RAM each
#$ -pe mpinodes-verbose 3
#load hisat2 module environment
module load hisat2
# set working directory to scratch space project folder
#$ -wd /mnt/scratch/users/*******/file
hisat2-build -p 8 --ss genome.ss --exon genome.exon genome.fa genome_tran
The errors I get often look like this:
Settings:
Output files: "genome_tran.*.ht2"
Line rate: 7 (line is 128 bytes)
Lines per side: 1 (side is 128 bytes)
Offset rate: 4 (one in 16)
FTable chars: 10
Strings: unpacked
Local offset rate: 3 (one in 8)
Local fTable chars: 6
Local sequence length: 57344
Local sequence overlap between two consecutive indexes: 1024
Endianness: little
Actual local endianness: little
Sanity checking: disabled
Assertions: disabled
Random seed: 0
Sizeofs: void*:8, int:4, long:8, size_t:8
Input files DNA, FASTA:
genome.fa
Reading reference sizes
Time reading reference sizes: 00:00:10
Calculating joined length
Writing header
Reserving space for joined string
Joining reference sequences
Time to join reference sequences: 00:00:05
Time to read SNPs and splice sites: 00:00:07
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Ran out of memory; automatically trying more memory-economical parameters.
Could not find approrpiate bmax/dcv settings for building this index.
Switching to a packed string representation.
Total time for call to driver() for forward index: 01:00:28
==================================================
SGE job completed on
==================================================
I added code markup to your post for increased readability. You can do this by selecting the text and clicking the 101010 button. When you compose or edit a post that button is in your toolbar, see image below:
In addition, I changed this post to a "question" (rather than a "forum").
Requesting 3 nodes is a waste of 2 of them. Like most programs, hisat2 can only make use of a single node.