Source other conda environments in a nextflow pipeline when nextflow itself is in a conda environment?
2
0
Entering edit mode
11 weeks ago
chaco001 ▴ 40

Hello,

I work in an HPC environment which uses slurm.

I am writing nextflow pipelines. Right now, nextflow itself has been installed in a conda environment. However, some of the processes in my nextflow pipeline need to use a different conda environment, because of dependency conflicts.

I have two related questions:

  1. How can I have processes use a conda environment that differs from the environment in which nextflow is being run?

  2. Have I set this up wrong in the first place and is it better to have nextflow outside of the conda environment? If so, what's the best way to distribute the whole pipeline along with nextflow?

Thank you!

hpc conda nextflow • 569 views
ADD COMMENT
0
Entering edit mode

Use the standalone version of Nextflow and specify the process-specific conda environments paths.

ADD REPLY
2
Entering edit mode
11 weeks ago

I used to use nextflow in a conda env. That requires the env to be active when a pipeline is started. Also I have productive pipelines which require fully isolated nextflow, so that would mean an individual conda env per pipeline. That is slow to create and includes a lot of small files.

That is why I use the nextflow single jar distribution and a globally located java17 on an NFS storage. The jar is very quick to download.

I use singularity containers created from a Dockerfile for the pipeline dependencies, as they tend to be more static and reliable than conda envs. Micromamba can be easily used for conda packages within a Docker/Sing. container.

You can set containers or conda per process in the nextflow main.nf file, or -better - in the nextflow.config file in a process block (along with RAM, cpus and retry). Isolating the hardware config from the code into the config file is the way to go.

ADD COMMENT
2
Entering edit mode
11 weeks ago
ATpoint 83k

You can make each process use a dedicated environment, see https://www.nextflow.io/docs/latest/conda.html#use-existing-conda-environments

The nextflow binary via conda is fine. No preference here. Or just install Nextflow as described in the Nextflow docs. It doesn't matter.

ADD COMMENT

Login before adding your answer.

Traffic: 1259 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6