Nextflow using existing Conda Environments
1
0
Entering edit mode
16 months ago
Barry Digby ▴ 780

Hi,

I am attempting to use an existing conda environment in a nextflow script (I do not want to point to a .yml file and compile environments). I have followed the instructions suggested by nextflow for using an existing environment:

process foo {
conda '/path/to/an/existing/env/directory'

'''
your_command --here
'''
}

Here is my attempt:

#!/usr/bin/env nextflow

process foo {
echo true
conda '/home/barry/anaconda3/envs/circexplorer2'

script:
"""
CIRCexplorer2 --help
"""
}

It gives the folowing error, pointing to the .command.run line 77:

.command.run: line 77: activate: No such file or directory

Looking at the file:

# conda environment
source activate /home/barry/anaconda3/envs/circexplorer2

It seems strange that it is using the depreciated source activate call, but maybe I'm missing a trick. Any thoughts?

p.s I have stripped the path and provided conda 'circexplorer2' to the nextflow script, but it tries to create the environment which is not the behaviour I was hoping for.

Regards,

Barry

nextflow conda anaconda • 2.4k views
ADD COMMENT
1
Entering edit mode

Maybe you saw this Github issue?

If I want to use conda as per your example, because the HPC I am on uses Modules, I need to: module load conda, then initialise: conda init bash and then reload the shell: exec bash (this gets a (base) bmoran@cluster:$ prompt). Then I can specify as you want.

I found it easier to create the conda env in a Singularity container from an env.yaml, these build so fast it is rarely an inconvenience, and I think having the container frozen, and potentially available on Singularity Hub/Library for posterity, is useful.

ADD REPLY
0
Entering edit mode

Hi Bruce,

Yep, I derived the local solution (answer below) from a suggestion on the github page linked. Cool, I did not know you could create a container from a .yml file, could you share the singularity call used to build the container? I am keen on making this pipeline reproducible , +1 for posterity.

ADD REPLY
0
Entering edit mode

Based on a Github repo with my_env.yaml therein; environment section is called every time the container executes, so that is activating the env; you can pull the centos7_conda container to see what's in there, just installs miniconda, used to install that in 'project' containers but more efficient to have a 'base' container IMO.

Bootstrap:library
From:bruce.moran/default/bases:centos7_conda

%environment

  source /opt/miniconda/etc/profile.d/conda.sh
  conda activate my_env

%post

  source /opt/miniconda/etc/profile.d/conda.sh
  git clone https://github.com/brucemoran/my_env
  conda env create -n my_env -f ./my_env/my_env.yaml
  conda clean -y -a
ADD REPLY
0
Entering edit mode

Hi Bruce,

I've had a spot of bother making the image. I have replaced the %post with my github repo but singularity build returns ERROR: 'Bootstrap' type not supported: library. I also went to the page where the container is hosted and tried to install via singularity pull, but it returned ERROR: pull is only supported for shub URIs.

ADD REPLY
1
Entering edit mode

Check singularity --version, then update your Singularity version=D

N.B. the code chunk in my example needs to be saved entirely into a recipe.container_name file, then you can use singularity build container_name.sif recipe.container_name to build it and push to https://cloud.sylabs.io/library, or build on the Singularity Hub. During building the env.yaml is cloned from Github to provide the container with the conda env.

ADD REPLY
0
Entering edit mode
16 months ago
Barry Digby ▴ 780

Local solution:

sudo ln -s /home/barry/anaconda3/bin/activate /usr/bin/activate

HPC solution :

export PATH="/home/bdigby/Anaconda3/4.4.0/bin/:$PATH"

add to ~/.bashrc

ADD COMMENT

Login before adding your answer.

Traffic: 2362 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6