Snakemake issue with wrappers
0
0
Entering edit mode
3 months ago

I have issues when running a wrapper of BWA mem with Snakemake. The error message "No module named 'snakemake_wrapper_utils'" appear (see below).

However, when checking if the package is installed in Python, I found the following:

import snakemake_wrapper_utils
print(snakemake_wrapper_utils.__version__)
0.1.0

Did anyone have this problem? Would you know why there is a module error, while the module is installed?

snakemake --cores all

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Conda environments: ignored
Job stats:
job        count
-------  -------
all            1
bwa_mem        3
total          4

Select jobs to execute...

[Thu Nov  2 17:43:16 2023]
rule bwa_mem:
    input: reads/110627_0240_AC0254ABXX_2_SA-PE-001.1.fastq.gz, reads/110627_0240_AC0254ABXX_2_SA-PE-001.2.fastq.gz, genome.amb, genome.ann, genome.bwt, genome.pac, genome.sa
    output: mapped/110627_0240_AC0254ABXX_2_SA-PE-001.bam
    log: logs/bwa_mem/110627_0240_AC0254ABXX_2_SA-PE-001.log
    jobid: 1
    reason: Missing output files: mapped/110627_0240_AC0254ABXX_2_SA-PE-001.bam
    wildcards: sample=110627_0240_AC0254ABXX_2_SA-PE-001
    resources: tmpdir=/tmp/tmichel_8894483

Traceback (most recent call last):
  File "/mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/.snakemake/scripts/tmpya0pk1fa.wrapper.py", line 14, in <module>
    from snakemake_wrapper_utils.java import get_java_opts
ModuleNotFoundError: No module named 'snakemake_wrapper_utils'
[Thu Nov  2 17:43:21 2023]
Error in rule bwa_mem:
    jobid: 1
    input: reads/110627_0240_AC0254ABXX_2_SA-PE-001.1.fastq.gz, reads/110627_0240_AC0254ABXX_2_SA-PE-001.2.fastq.gz, genome.amb, genome.ann, genome.bwt, genome.pac, genome.sa
    output: mapped/110627_0240_AC0254ABXX_2_SA-PE-001.bam
    log: logs/bwa_mem/110627_0240_AC0254ABXX_2_SA-PE-001.log (check log file(s) for error details)
    conda-env: /mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/.snakemake/conda/8ec8a2e6cccaa67a3428f1bf7dcc48b8_

RuleException:
CalledProcessError in file /mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/workflow/Snakefile, line 76:
Command 'set -euo pipefail;  /mnt/shared/home/usr/mambaforge/envs/snakemake/bin/python3.12 /mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/.snakemake/scripts/tmpya0pk1fa.wrapper.py' returned non-zero exit status 1.
  File "/mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/workflow/Snakefile", line 76, in __rule_bwa_mem
  File "/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-11-02T174251.293416.snakemake.log
Python Snakemake • 983 views
ADD COMMENT
1
Entering edit mode

I noticed conda-env in the error message. Wrapper defines its own environment to run the wrapper, I'd remove the conda: attribute from your rule and try again with snakemake --use-conda. Hopefully that fixes the issue.

ADD REPLY
0
Entering edit mode

Thank you Eric Lim, I had a conda activate environement in the batch script I was using to run Snakemake. Unfortunately, removing it does not change the previous error.

import os
import snakemake.io
import glob
import pandas as pd

configfile: "config/config.yaml"

table=pd.read_csv("config/table_reads.tsv", delim_whitespace=True,  header=None, index_col=False)

SAMPLES=table.loc[:, 0].to_list()
READS=["1","2"]

rule all:
        input:
                expand("mapped/{sample}.bam", sample=SAMPLES)

rule fastqc:
        input:
                expand("reads/{sample}.{read}.fastq.gz", sample=SAMPLES, read=READS)


        output:
                html="qc/fastqc/{sample}.html",
                zip="qc/fastqc/{sample}_fastqc.zip" # the suffix _fastqc.zip is necessary for multiqc to find the file. If not using multiqc, you are free to choose an arbitrary filename
        params:
                extra = "--quiet"
        log:
                "logs/fastqc/{sample}.log"
        threads: 1
        resources:
                mem_mb = 1024
        wrapper:
                "v2.6.0/bio/fastqc"

rule bwa_index:
        input:
                config["reference"]
                #"{genome}.fasta", 
        output:
                idx=multiext("{genome}", ".amb", ".ann", ".bwt", ".pac", ".sa"),
        log:
                "logs/bwa_index/{genome}.log",
        params:
                algorithm="bwtsw",
        wrapper:
                "v2.6.0/bio/bwa/index"

rule bwa_mem:
        input:
                reads=["reads/{sample}.1.fastq.gz", "reads/{sample}.2.fastq.gz"],
                idx=multiext("genome", ".amb", ".ann", ".bwt", ".pac", ".sa"),
        output:
                "mapped/{sample}.bam",
        log:
                "logs/bwa_mem/{sample}.log",
        params:
                extra=r"-R '@RG\tID:{sample}\tSM:{sample}'",
                sorting="none",  # Can be 'none', 'samtools' or 'picard'.
                sort_order="queryname",  # Can be 'queryname' or 'coordinate'.
                sort_extra="",  # Extra args for samtools/picard.
        threads: 8
        wrapper:
                "v2.9.0-45-g0055391/bio/bwa/mem"
ADD REPLY
0
Entering edit mode

I have noticed that adding "import snakemake_wrapper_utils" in the Snakefile trigger the error:

No module named 'snakemake_wrapper_utils'

So there is definitely a problem with this module, which I can find in Python, but not running Snakemake.

ADD REPLY
1
Entering edit mode

Can you print sys.path both in python and within the snakemake script and see if the results are the same?

ADD REPLY
0
Entering edit mode

Thank you Ram, I am not sure if I should have the same result.

In Snakemake:

print(sys.path)
['/mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/workflow', '/mnt/shared/home/usr/mambaforge/envs/snakemake/bin', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python312.zip', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12/lib-dynload', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12/site-packages', '/opt/gurobi201/linux32/lib/python2.5']

In python:

>>> print(sys.path)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sys' is not defined
ADD REPLY
1
Entering edit mode

You should really have done this on your own, but import sys then print(sys.path) to address the error in the second case. The results should at least overlap.

ADD REPLY
0
Entering edit mode

Ah, I did not know sys was a Python library. It is worth mentioning it.

print(sys.path)
['/mnt/shared/scratch/usr/comp_baits_pseudo-chr/baits/workflow', '/mnt/shared/home/usr/mambaforge/envs/snakemake/bin', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python312.zip', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12/lib-dynload', '/mnt/shared/home/usr/mambaforge/envs/snakemake/lib/python3.12/site-packages', '/opt/gurobi201/linux32/lib/python2.5']

>>> import sys
>>> print(sys.path)
['', '/mnt/shared/home/tmichel/mambaforge/envs/snakemake/lib/python312.zip', '/mnt/shared/home/tmichel/mambaforge/envs/snakemake/lib/python3.12', '/mnt/shared/home/tmichel/mambaforge/envs/snakemake/lib/python3.12/lib-dynload', '/mnt/shared/home/tmichel/mambaforge/envs/snakemake/lib/python3.12/site-packages']
ADD REPLY
1
Entering edit mode

I'm going to make a wild guess here. The snakemake env you have under the tmichel home dir is different from the snakemake env you have with the /mnt/shared/ prefix. The latter is missing the required library.

ADD REPLY
0
Entering edit mode

I think the two different environment are at the source of the problem indeed.

I am not sure it is due to a missing snakemake_wrapper_utils module, as I have installed it in both environment. But it might be related to the Python version loaded.

There is a python2.5 in the snakemake environment, and I think it might cause the snakemake_wrapper_utils module not to be found. I am trying to find a way not to load python2.5.

ADD REPLY

Login before adding your answer.

Traffic: 997 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6