Question: Nextflow using existing Conda Environments
0
gravatar for Barry Digby
21 days ago by
Barry Digby390
National University of Ireland, Galway
Barry Digby390 wrote:

Hi,

I am attempting to use an existing conda environment in a nextflow script (I do not want to point to a .yml file and compile environments). I have followed the instructions suggested by nextflow for using an existing environment:

process foo {
conda '/path/to/an/existing/env/directory'

'''
your_command --here
'''
}

Here is my attempt:

#!/usr/bin/env nextflow

process foo {
echo true
conda '/home/barry/anaconda3/envs/circexplorer2'

script:
"""
CIRCexplorer2 --help
"""
}

It gives the folowing error, pointing to the .command.run line 77:

.command.run: line 77: activate: No such file or directory

Looking at the file:

# conda environment
source activate /home/barry/anaconda3/envs/circexplorer2

It seems strange that it is using the depreciated source activate call, but maybe I'm missing a trick. Any thoughts?

p.s I have stripped the path and provided conda 'circexplorer2' to the nextflow script, but it tries to create the environment which is not the behaviour I was hoping for.

Regards,

Barry

nextflow conda anaconda • 147 views
ADD COMMENTlink modified 21 days ago by ATpoint36k • written 21 days ago by Barry Digby390
1

Maybe you saw this Github issue?

If I want to use conda as per your example, because the HPC I am on uses Modules, I need to: module load conda, then initialise: conda init bash and then reload the shell: exec bash (this gets a (base) bmoran@cluster:$ prompt). Then I can specify as you want.

I found it easier to create the conda env in a Singularity container from an env.yaml, these build so fast it is rarely an inconvenience, and I think having the container frozen, and potentially available on Singularity Hub/Library for posterity, is useful.

ADD REPLYlink modified 12 days ago • written 12 days ago by bruce.moran830

Hi Bruce,

Yep, I derived the local solution (answer below) from a suggestion on the github page linked. Cool, I did not know you could create a container from a .yml file, could you share the singularity call used to build the container? I am keen on making this pipeline reproducible , +1 for posterity.

ADD REPLYlink modified 12 days ago • written 12 days ago by Barry Digby390

Based on a Github repo with my_env.yaml therein; environment section is called every time the container executes, so that is activating the env; you can pull the centos7_conda container to see what's in there, just installs miniconda, used to install that in 'project' containers but more efficient to have a 'base' container IMO.

Bootstrap:library
From:bruce.moran/default/bases:centos7_conda

%environment

  source /opt/miniconda/etc/profile.d/conda.sh
  conda activate my_env

%post

  source /opt/miniconda/etc/profile.d/conda.sh
  git clone https://github.com/brucemoran/my_env
  conda env create -n my_env -f ./my_env/my_env.yaml
  conda clean -y -a
ADD REPLYlink modified 10 days ago • written 12 days ago by bruce.moran830

Hi Bruce,

I've had a spot of bother making the image. I have replaced the %post with my github repo but singularity build returns ERROR: 'Bootstrap' type not supported: library. I also went to the page where the container is hosted and tried to install via singularity pull, but it returned ERROR: pull is only supported for shub URIs.

ADD REPLYlink written 6 days ago by Barry Digby390
1

Check singularity --version, then update your Singularity version=D

N.B. the code chunk in my example needs to be saved entirely into a recipe.container_name file, then you can use singularity build container_name.sif recipe.container_name to build it and push to https://cloud.sylabs.io/library, or build on the Singularity Hub. During building the env.yaml is cloned from Github to provide the container with the conda env.

ADD REPLYlink written 1 day ago by bruce.moran830
0
gravatar for Barry Digby
21 days ago by
Barry Digby390
National University of Ireland, Galway
Barry Digby390 wrote:

Local solution:

sudo ln -s /home/barry/anaconda3/bin/activate /usr/bin/activate

HPC solution :

export PATH="/home/bdigby/Anaconda3/4.4.0/bin/:$PATH"

add to ~/.bashrc

ADD COMMENTlink modified 21 days ago • written 21 days ago by Barry Digby390
0
gravatar for ATpoint
21 days ago by
ATpoint36k
Germany
ATpoint36k wrote:

In general you activate conda environments with `conda activate <env-name>'. On our HPC using SLURM I use:

eval "$(conda shell.bash hook)"
conda activate <env-name>

Took this over from: https://github.com/conda/conda/issues/7980

Don't ask my why but you cannot use the activate command without the eval part in a script (no clue what this is actually doing but it works...). The PATH solution is of course possible but tedious, I prefer activating the environments as above.

ADD COMMENTlink modified 21 days ago • written 21 days ago by ATpoint36k
1

I see, I'm a fan of using

source /home/Anaconda3/4.4.0/etc/profile.d/conda.sh
conda activate env-name

Many ways to die in the west as my sys admin says!

The PATH method is for nextflow specifically :)

ADD REPLYlink written 21 days ago by Barry Digby390
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1065 users visited in the last hour