Entering edit mode
21 months ago
David
•
0
Hello,
I implemented a Snakemake pipeline for metagenomic analysis with Qiime2 in Ubuntu 20.04.4. It was working fine but recently it stopped working, I am not sure if it happened after updating conda. From what I can read in the error message, Qiime2 is trying to cache its current deployment but it can't due to some import error. But no clue how to solve it. Any help is greatly appreciated.
rule all:
input:
"processed_data/222112314/qiime2/paired_end_demux.qza"
rule qiime_import_seqs:
input:
config["manifest_file_path"],
output:
"processed_data/222112314/qiime2/paired_end_demux.qza"
log:
"logs/qiime2/import_q2.log"
conda:
"envs/qiime2-2021.8.yaml"
shell:
"qiime tools import \
--type 'SampleData[PairedEndSequencesWithQuality]' \
--input-path {input[0]} \
--output-path {output} \
--input-format PairedEndFastqManifestPhred33V2"
Then I get the following error:
(snakemake) David@David-HP-ZBook-15:~/repos/metagenomics-pipeline$ snakemake -c1 --use-conda
processed_data/222112314/qiime2/paired_end_demux.qza/
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job count min threads max threads
----------------- ------- ------------- -------------
all 1 1 1
qiime_import_seqs 1 1 1
total 2 1 1
Select jobs to execute...
[Tue Jul 12 15:57:12 2022]
rule qiime_import_seqs:
input: processed_data/222112314/manifest.file
output: processed_data/222112314/qiime2/paired_end_demux.qza
log: logs/qiime2/import_q2.log
jobid: 3
reason: Missing output files: processed_data/222112314/qiime2/paired_end_demux.qza
resources: tmpdir=/tmp
Activating conda environment: .snakemake/conda/3f3d93188afda05bca836a00cf1e1770
QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
Traceback (most recent call last):
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/bin/qiime", line 11, in <module>
sys.exit(qiime())
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/click/core.py", line 781, in main
with self.make_context(prog_name, args, **extra) as ctx:
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/click/core.py", line 700, in make_context
self.parse_args(ctx, args)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2cli/click/command.py", line 43, in parse_args
return super().parse_args(ctx, args)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/click/core.py", line 1209, in parse_args
echo(ctx.get_help(), color=ctx.color)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/click/core.py", line 570, in get_help
return self.command.get_help(self)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/click/core.py", line 975, in get_help
self.format_help(ctx, formatter)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/click/core.py", line 1003, in format_help
self.format_options(ctx, formatter)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2cli/click/command.py", line 157, in format_options
for subcommand in self.list_commands(ctx):
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2cli/commands.py", line 92, in list_commands
plugins = sorted(self._plugin_lookup)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2cli/commands.py", line 76, in _plugin_lookup
import q2cli.core.cache
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2cli/core/cache.py", line 406, in <module>
CACHE = DeploymentCache()
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2cli/core/cache.py", line 61, in __init__
self._state = self._get_cached_state(refresh=refresh)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2cli/core/cache.py", line 107, in _get_cached_state
self._cache_current_state(current_requirements)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2cli/core/cache.py", line 200, in _cache_current_state
state = self._get_current_state()
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2cli/core/cache.py", line 238, in _get_current_state
plugin_manager = qiime2.sdk.PluginManager()
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 57, in __new__
self._init(add_plugins=add_plugins)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 95, in _init
plugin = entry_point.load()
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2465, in load
return self.resolve()
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2471, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2_diversity_lib/plugin_setup.py", line 16, in <module>
from unifrac._meta import CONSOLIDATIONS
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/unifrac/__init__.py", line 30, in <module>
from unifrac._api import ssu, faith_pd, ssu_to_file
ImportError: /home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/unifrac/_api.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z19destroy_results_vecPP11results_vec
Traceback (most recent call last):
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2cli/builtin/tools.py", line 156, in import_data
artifact = qiime2.sdk.Artifact.import_data(type, input_path,
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/qiime2/sdk/result.py", line 243, in import_data
view_type = qiime2.sdk.parse_format(view_type)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/qiime2/sdk/util.py", line 88, in parse_format
pm = qiime2.sdk.PluginManager()
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 57, in __new__
self._init(add_plugins=add_plugins)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 95, in _init
plugin = entry_point.load()
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2465, in load
return self.resolve()
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2471, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/q2_diversity_lib/plugin_setup.py", line 16, in <module>
from unifrac._meta import CONSOLIDATIONS
File "/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/unifrac/__init__.py", line 30, in <module>
from unifrac._api import ssu, faith_pd, ssu_to_file
ImportError: /home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/unifrac/_api.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z19destroy_results_vecPP11results_vec
An unexpected error has occurred:
/home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770/lib/python3.8/site-packages/unifrac/_api.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z19destroy_results_vecPP11results_vec
See above for debug info.
[Tue Jul 12 15:57:23 2022]
Error in rule qiime_import_seqs:
jobid: 3
output: processed_data/222112314/qiime2/paired_end_demux.qza
log: logs/qiime2/import_q2.log (check log file(s) for error message)
conda-env: /home/David/repos/metagenomics-pipeline/.snakemake/conda/3f3d93188afda05bca836a00cf1e1770
shell:
qiime tools import --type 'SampleData[PairedEndSequencesWithQuality]' --input-path processed_data/222112314/manifest.file --output-path processed_data/222112314/qiime2/paired_end_demux.qza --input-format PairedEndFastqManifestPhred33V2
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-07-12T155709.168416.snakemake.log
I don't know the answer, but whatever it is has to do with the unifrac package's compiled cython code inside of the conda env built from the qiime2-2021.8.yaml file (look near the bottom of the stacktrace). One low-effort thing to try (assuming it contains nothing you care about!) is to just delete your .snakemake directory and let it rebuild everything on the next run, including the conda env it'll put there. (Or you can be a little more precise and remove just the one env directory.) If that doesn't do it, you could activate the conda environment in a shell and poke around, and you could build a standalone unifrac env straight from bioconda for comparison (See "Install (bioconda)" at https://pypi.org/project/unifrac)