Trouble running snakemake while defining maximum RAM usage
2
2
Entering edit mode
3.3 years ago
cfos4698 ★ 1.1k

Hi,

One rule in my snakemake workflow (in which I run kraken2) has a large RAM requirement of ~50 GB. When running the workflow on many samples in parallel (e.g., snakemake --cores 12 -s workflow.smk), I get a bunch of errors because the system's RAM is exhausted.

As a result, I've tried limiting the amount of RAM used by snakemake by setting a maximum memory usage within the kraken2 rule:

    resources:
        mem_mb=55000

Intuitively, I thought that this would mean that snakemake would run each job at a time since running >1 job would exceed 55000 MB (=55 GB) of RAM usage. However, the workflow still fails by running out of RAM despite my system having 64 GB of RAM.

Next, I tried also setting a maximum amount of RAM on the command line using the --resources flag:

snakemake -j12 -s workflow.smk --resources mem_mb=55000

However, I get a traceback error:

Traceback (most recent call last):
  File "/home/cfos/.local/bin/snakemake", line 8, in <module>
    sys.exit(main())
  File "/home/cfos/.local/lib/python3.8/site-packages/snakemake/__init__.py", line 2400, in main
    resources = parse_resources(args.resources)
  File "/home/cfos/.local/lib/python3.8/site-packages/snakemake/resources.py", line 85, in parse_resources
    for res, val in resources_args.items():
AttributeError: 'list' object has no attribute 'items'

What am I doing wrong here? Any advice would be appreciated.

snakemake • 3.0k views
ADD COMMENT
0
Entering edit mode

Which verision of snakemake are you using? Also, try to write a minimal, self-contained example that reproduces the issue and post it here if you cannot find a solution.

ADD REPLY
2
Entering edit mode

Good point. I'm using snakemake v6.5.1 installed via mamba. Minimal example:

test.smk:

rule all:
    input:
        "test.txt"
    message:
        "This is a test!"

rule touch:
    output:
        file = "test.txt"
    resources: mem_mb=55000
    shell:
        """
        touch {output.file}
        """

Command:

snakemake -j1 -s test.smk --resources mem_mb=50000

Output:

Traceback (most recent call last):
  File "/home/vrl/miniconda3/bin/snakemake", line 10, in <module>
    sys.exit(main())
  File "/home/vrl/miniconda3/lib/python3.8/site-packages/snakemake/__init__.py", line 2400, in main
    resources = parse_resources(args.resources)
  File "/home/vrl/miniconda3/lib/python3.8/site-packages/snakemake/resources.py", line 85, in parse_resources
    for res, val in resources_args.items():
AttributeError: 'list' object has no attribute 'items'
ADD REPLY
3
Entering edit mode
3.3 years ago

UPDATE: This has been fixed in version 6.5.3

ADD COMMENT
1
Entering edit mode
3.3 years ago

I think you found a bug in v6.5.1. I tested your example with 6.0.0 and it worked, upgraded to 6.5.1 and I can reproduce the error (don't know about versions in between).

EDIT

The issue has been submitted to github just few hours after you posted your question!

ADD COMMENT
1
Entering edit mode

Just as an update, I upgraded to v6.5.2 and the issue still persists. I downgraded to 6.4.1 and the issues does not occur.

ADD REPLY
0
Entering edit mode

Excellent, good to know that I'm not going crazy! Thanks for pointing that out.

ADD REPLY

Login before adding your answer.

Traffic: 889 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6