Launch a snakefile involving multiple conda environment using an snakemake specific environment

Question

[regue.hadrien]

Hi everyone,

I'm working on a snakefile involving several conda environements. As it's more simple (and avoid package troubles) to split each tools in a specific conda environment, I've separated my tools and calling them in my rules just like this:

#trimming
rule trimming:
message:
    "Filtering fastq using NanoFilt."
input:
    merged_fastq = rules.merge.output.merged_fastq
output:
    trimmed_fastq = resultpath+"TRIMMED/{barcode}_trimmed.fastq"
conda:
    "env/nanofilt.yaml"
shell:
    "NanoFilt --quality 10 --length 100 --maxlength 1500 {input} > {output} "

On my computer, snakemake is already installed and I basicaly typing the command below:

 snakemake -s viralION.py --use-conda

The full pipeline is executing perfectly.

However, my trouble appear when I'm trying to launch the same command, but from an environment containing snakemake (and snakemake only) created with this command:

conda create -c conda-forge -c bioconda -n snakemake snakemake
conda activate snakemake
snakemake -s viralION.py --use-conda

snakemake return an error telling me that NanoFilt command was not found. As this rule is the first one involving an environment, i can guess this problem will extend for others tools (samtools,bcftools etc...)

Here is the full message of the failed job:

[Wed Jul  1 12:10:17 2020]
Job 5: Filtering fastq using NanoFilt.
/bin/bash: activate: No such file or folder
/bin/bash: NanoFilt : command not found
Activating conda environment: /home/chu-lyon.fr/regueex/git/MinION_HBV/.snakemake/conda/8c13cc8c
RuleException:
CalledProcessError in line 72 of /home/chu-lyon.fr/regueex/git/MinION_HBV/viralION.py:
Command 'source activate /home/chu-lyon.fr/regueex/git/MinION_HBV/.snakemake/conda/8c13cc8c; set -euo pipefail;  NanoFilt --quality 10 --length 100 --maxlength 1500 /srv/nfs/ngs-stockage/NGS_Virologie/HadrienR/CARO_PIPELINE/RESULT_HDV/MERGED/barcode03_merged.fastq > /srv/nfs/ngs-stockage/NGS_Virologie/HadrienR/CARO_PIPELINE/RESULT_HDV/TRIMMED/barcode03_trimmed.fastq ' returned non-zero exit status 127.
  File "/home/chu-lyon.fr/regueex/git/MinION_HBV/viralION.py", line 72, in __rule_trimming
  File "/home/chu-lyon.fr/regueex/anaconda3/envs/snakemake/lib/python3.6/concurrent/futures/thread.py", line 56, in run

I dont realy get what the problem is. Do you have some clues?

EDIT: I've also tried to execute my snakefile from a empty conda environment but it's not working too(NanoFilt cant be found):

conda create -n testLaunchSnakefile
conda activate testLaunchSnakefile
snakemake -s viralION.py --use-conda

Thanks, and have a nice day.

Hadrien

[Sej Modha]

AFAIK, you'd need to ensure that all tools included in the Snakemake pipeline are installed in the specific environment you're running the Snakemake pipeline from. You'd need to install NanoFilt and other tools either in the new conda environment.

[regue.hadrien]

as mentionned on the snakemake documentation and biostar tutorial its possible to call tools from separate environments using snakemake. For exemple, NanoFilt is not installed on this computed, but the snakefile is working fine. However I dont know why my pipeline dont work when executed from a conda environment.

[Sej Modha]

Sorry, you're right, I misunderstood the question.

Can you clarify what you mean by 'On my computer, snakemake is already installed' ? Do you mean you have snakemake installed in your base env?

[regue.hadrien]

To be more precise, I'm working on a server, where snakemake is already installed. I haven't root access, so I've installed anaconda personnaly to use other bioinformatic tools. So yeah , snakemake is present in my base environment, but was installed before conda.

Hope I was clear:)

[Sej Modha]

Thanks for clarifying that, just to check, could you try putting nanofilt.yaml at the same directory level as the snakemake file and modify the file path in the script? Re-run the script and see if still get the same error? I am just wondering if snakemake is able to find the env file specified in the script.

[Jeremy Leipzig]

can you show us what env/nanofilt.yaml looks like?

[regue.hadrien]

Sure, here is the full file:

name: nanofilt
channels:
  - bioconda
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - biopython=1.76=py37h7b6447c_0
  - blas=1.0=mkl
  - bzip2=1.0.8=h7b6447c_0
  - ca-certificates=2020.6.24=0
  - certifi=2020.6.20=py37_0
  - curl=7.71.0=hbc83047_0
  - intel-openmp=2020.1=217
  - krb5=1.18.2=h173b8e3_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libcurl=7.71.0=h20c2e04_0
  - libdeflate=1.0=h14c3975_1
  - libedit=3.1.20191231=h7b6447c_0
  - libffi=3.3=he6710b0_1
  - libgcc-ng=9.1.0=hdf63c60_0
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libssh2=1.9.0=h1ba5d50_1
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - mkl=2020.1=217
  - mkl-service=2.3.0=py37he904b0f_0
  - mkl_fft=1.1.0=py37h23d657b_0
  - mkl_random=1.1.1=py37h0573a6f_0
  - nanofilt=2.7.1=py_0
  - nanoget=1.13.2=py_0
  - nanomath=0.23.3=py_0
  - ncurses=6.2=he6710b0_1
  - numpy=1.18.5=py37ha1c710e_0
  - numpy-base=1.18.5=py37hde5b4d6_0
  - openssl=1.1.1g=h7b6447c_0
  - pandas=1.0.5=py37h0573a6f_0
  - pip=20.1.1=py37_1
  - pysam=0.15.3=py37hda2845c_1
  - python=3.7.7=hcff3b4d_5
  - python-dateutil=2.8.1=py_0
  - pytz=2020.1=py_0
  - readline=8.0=h7b6447c_0
  - setuptools=47.3.1=py37_0
  - six=1.15.0=py_0
  - sqlite=3.32.3=h62c20be_0
  - tk=8.6.10=hbc83047_0
  - wheel=0.34.2=py37_0
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7b6447c_3
prefix: /home/chu-lyon.fr/regueex/anaconda3/envs/nanofilt

Answer 1

[regue.hadrien]

Well, I think I have solved my problem.

I haven't created my .yaml files manualy, but individualy created individual environments for each tools and exported them such as:

conda create -n nanofilt -c bioconda nanofilt
conda activate nanofilt
conda env export > env/nanofilt.yaml

I tried to remove each single environments and deleted the prefix line from my yaml files, and now , it seems to work. I can now execute my snakefile from any environment, and each individual environemnts are correctly created in the .snakemake/ dir

However I dont know if this was the last line from my yaml files or the singles environments the problem, anyway, everything is working well now:)

Thank you for your anwsers.

Hadrien