Entering edit mode
3.2 years ago
priya.bmg
▴
60
Hello
I have started using snakemake 6.5.2 to align fastq files with reference file. I have pasted the error below in this question. How to allocate memory in the snakefile and read the header from samfile, '-'. This is the snakefile (wrapper for running alignment):
rule bwa_mem2_mem:
input:
reads=["/scicore/home/cichon/GROUP/test_workflow/samples/{sample}.1.fq", "/scicore/home/cichon/GROUP/test_workflow/samples/{sample}.2.fq"]
output:
"/scicore/home/cichon/GROUP/test_workflow/output/{sample}.bam"
log:
"/scicore/home/cichon/GROUP/test_workflow/output/{sample}.log"
params:
index="/scicore/home/cichon/GROUP/test_workflow/data/gch38.fa",
extra=r"-R '@RG\tID:{sample}\tSM:{sample}'",
sort="none", # Can be 'none', 'samtools' or 'picard'.
sort_order="coordinate", # Can be 'coordinate' (default) or 'queryname'.
sort_extra=""
threads: 8
wrapper:
"0.77.0/bio/bwa-mem2/mem"
I used this command to execute the above snakemake file:
snakemake --use-conda --snakefile Snakefile --cores -p /scicore/home/cichon/GROUP/test_workflow/output/DE98NGSUKBD117612_1.bam /scicore/home/cichon/GROUP/test_workflow/output/DE71NGSUKBD117613_3.bam
Error from log file:
Looking to launch executable "/scicore/home/cichon/GROUP/test_workflow/.snakemake/conda/0ecf6ba401c70d7501c73bb1c733a060/bin/bwa-mem2.avx", simd = .avx
Launching executable "/scicore/home/cichon/GROUP/test_workflow/.snakemake/conda/0ecf6ba401c70d7501c73bb1c733a060/bin/bwa-mem2.avx"
-----------------------------
Executing in AVX mode!!
-----------------------------
* SA compression enabled with xfactor: 8
* Ref file: /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa
* Entering FMI_search
* Index file found. Loading index from /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa.bwt.2bit.64
* Reference seq len for bi-index = 6544178411
* sentinel-index: 2782712944
* Count:
0, 1
1, 1914743810
2, 3272089206
3, 4629434602
4, 6544178411
* Reading other elements of the index from files /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa
* Index prefix: /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa
* Read 0 ALT contigs
[fread] Cannot allocate memory
[main_samview] fail to read the header from "-".
Thanks
How much memory is available /allocated on that machine you run it on?
1 GB of memory for each core. 20 cores will be used by default
But here you used 8 cores, is it? So 8GB, that is not enough. See the bwa-mem2 GitHub page, the human index on disk alone is 10GB.
I have specified number of threads as 8 and given the number of cores as default (which is 20)
You are out of memory apparently, so try to find out why. Run the command outside of the snakefile and see whether it works, e.g. on a small subset of reads. That'll tell you whether the machine itself can load the index into memory.
Earlier, I have used 8 cores with 10GB of memory allocated to each. In Snakemake, where do I specify the memory?
https://stackoverflow.com/questions/60403990/how-to-set-binding-memory-limits-in-snakemake
Not sure. When I include memory limit in rule file, it is shown as syntax error.
Can you post the error? How did you call snakemake after including the rule?
Here is the complete snakemake documentation of the above thread:
Skip to the topic "Resources"
Sorry, I included mem_gb in the rule and it showed syntax error. Now, I changed to mem_mb and I get attribute error. Is there any error in the wrapper script?
I think you should update your snakemake to the latest version. Trouble running snakemake while defining maximum RAM usage
EDIT: Or see if downgrading helps. Downgrade to 6.4.1