Snakemake-Aligment using BWA-MEM2
0
0
Entering edit mode
3.2 years ago
priya.bmg ▴ 60

Hello

I have started using snakemake 6.5.2 to align fastq files with reference file. I have pasted the error below in this question. How to allocate memory in the snakefile and read the header from samfile, '-'. This is the snakefile (wrapper for running alignment):

  rule bwa_mem2_mem:
        input:
            reads=["/scicore/home/cichon/GROUP/test_workflow/samples/{sample}.1.fq", "/scicore/home/cichon/GROUP/test_workflow/samples/{sample}.2.fq"]
        output:
            "/scicore/home/cichon/GROUP/test_workflow/output/{sample}.bam"
        log:
            "/scicore/home/cichon/GROUP/test_workflow/output/{sample}.log"
        params:
            index="/scicore/home/cichon/GROUP/test_workflow/data/gch38.fa",
            extra=r"-R '@RG\tID:{sample}\tSM:{sample}'",
            sort="none",             # Can be 'none', 'samtools' or 'picard'.
            sort_order="coordinate", # Can be 'coordinate' (default) or 'queryname'.
            sort_extra=""     

        threads: 8
        wrapper:
            "0.77.0/bio/bwa-mem2/mem"

I used this command to execute the above snakemake file:

snakemake --use-conda --snakefile Snakefile --cores -p /scicore/home/cichon/GROUP/test_workflow/output/DE98NGSUKBD117612_1.bam /scicore/home/cichon/GROUP/test_workflow/output/DE71NGSUKBD117613_3.bam

Error from log file:

Looking to launch executable "/scicore/home/cichon/GROUP/test_workflow/.snakemake/conda/0ecf6ba401c70d7501c73bb1c733a060/bin/bwa-mem2.avx", simd = .avx
Launching executable "/scicore/home/cichon/GROUP/test_workflow/.snakemake/conda/0ecf6ba401c70d7501c73bb1c733a060/bin/bwa-mem2.avx"
-----------------------------
Executing in AVX mode!!
-----------------------------
* SA compression enabled with xfactor: 8
* Ref file: /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa
* Entering FMI_search
* Index file found. Loading index from /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa.bwt.2bit.64
* Reference seq len for bi-index = 6544178411
* sentinel-index: 2782712944
* Count:
0,  1
1,  1914743810
2,  3272089206
3,  4629434602
4,  6544178411

* Reading other elements of the index from files /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa
* Index prefix: /scicore/home/cichon/GROUP/test_workflow/data/gch38.fa
* Read 0 ALT contigs
[fread] Cannot allocate memory
[main_samview] fail to read the header from "-".

Thanks

memory-issue Snakemake • 2.9k views
ADD COMMENT
0
Entering edit mode

How much memory is available /allocated on that machine you run it on?

ADD REPLY
0
Entering edit mode

1 GB of memory for each core. 20 cores will be used by default

ADD REPLY
0
Entering edit mode

But here you used 8 cores, is it? So 8GB, that is not enough. See the bwa-mem2 GitHub page, the human index on disk alone is 10GB.

ADD REPLY
0
Entering edit mode

I have specified number of threads as 8 and given the number of cores as default (which is 20)

ADD REPLY
0
Entering edit mode

You are out of memory apparently, so try to find out why. Run the command outside of the snakefile and see whether it works, e.g. on a small subset of reads. That'll tell you whether the machine itself can load the index into memory.

ADD REPLY
0
Entering edit mode

Earlier, I have used 8 cores with 10GB of memory allocated to each. In Snakemake, where do I specify the memory?

ADD REPLY
0
Entering edit mode

Not sure. When I include memory limit in rule file, it is shown as syntax error.

ADD REPLY
0
Entering edit mode

Can you post the error? How did you call snakemake after including the rule?

Here is the complete snakemake documentation of the above thread:

Skip to the topic "Resources"

ADD REPLY
0
Entering edit mode

Sorry, I included mem_gb in the rule and it showed syntax error. Now, I changed to mem_mb and I get attribute error. Is there any error in the wrapper script?

snakemake --resources mem_mb=100 #command used to run snake make file

 snakemake --resources mem_mb=100
Traceback (most recent call last):
  File "/scicore/home/cichon/thirun0000/miniconda3/envs/snakemake_6.5.2/bin/snakemake", line 10, in <module>
    sys.exit(main())
  File "/scicore/home/cichon/thirun0000/miniconda3/envs/snakemake_6.5.2/lib/python3.9/site-packages/snakemake/__init__.py", line 2400, in main
    resources = parse_resources(args.resources)
  File "/scicore/home/cichon/thirun0000/miniconda3/envs/snakemake_6.5.2/lib/python3.9/site-packages/snakemake/resources.py", line 85, in parse_resources
    for res, val in resources_args.items():
AttributeError: 'list' object has no attribute 'items'
ADD REPLY
1
Entering edit mode

I think you should update your snakemake to the latest version. Trouble running snakemake while defining maximum RAM usage

EDIT: Or see if downgrading helps. Downgrade to 6.4.1

ADD REPLY

Login before adding your answer.

Traffic: 1283 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6