Singularity image is not working in nextflow
0
0
Entering edit mode
15 months ago

Hello everyone,

I'm having trouble getting Singularity to work with my custom bcftools pipeline in Nextflow.

I've tested my pipeline with Conda and it works great. However, when I install software locally (instead of using Conda), it runs approximately 10 times faster due to the large number of times the Conda environment needs to be invoked.

To test Singularity, I followed these steps:

  1. downloaded bcftools singularity image from: https://cloud.sylabs.io/library/weizhu365/mocca-sv/bcftools_1-9
  2. tested it with singularity run bcftools_1-9_1.0.0.sif
  3. added necessary lines to nextflow.config:
    process.container = '/path/to/bcftools_1-9_1.0.0.sif'
    singularity.enabled = true
    

However, it doesn't seem to recognize the image. I've successfully run other Nextflow pipelines with Singularity, such as nf-core's nextflow run nf-core/sarek -profile test,singularity -r 2.7.1.

Do I need to take any additional steps? Any help would be greatly appreciated.

nextflow singularity • 2.3k views
ADD COMMENT
2
Entering edit mode

It should work. At least this works:

$ cat main.nf 
nextflow.enable.dsl = 2

process Test {

  output:
  stdout

  script:
  """
  bcftools --version
  """

}

workflow { Test() }  

$ cat nextflow.config 
process.container = "/home/a/a_toen03/bcftools_1.17--haef29d1_0.sif"
singularity.enabled = true

...correctly running the bcftools command. Please elaborate on what exactly goes wrong. What is the error message (if any)? Can you try with the container I linked? Get via singularity pull docker://quay.io/biocontainers/bcftools:1.17--haef29d1_0 from quay.

ADD REPLY
0
Entering edit mode

The command bcftools --version is functioning properly. However, when attempting to use the command bcftools view -h file.vcf.gz to view a specific file in the current directory, an error message is generated indicating that the file does not exist, even though it's in the working directory. In fact, no file can be detected when singularity image is specified.

Can you please test if this issue is present in your system?

My singularity version: 3.4.1-4.2.ohpc.1.3.9

ADD REPLY
0
Entering edit mode

So the issue is that the container somehow does not have an access to file.vcf.gz? If you specified file.vcf.gz as nextflow's input you should have a softlink to that file in the nextflow work subdirectory. You may just get into the container singularity shell your_container.sif and check what is visible from there (ls, df)

ADD REPLY
0
Entering edit mode

I tested singularity shell your_container.sif inside the Work subdirectory and, even though file.vcf.gz is visible (ls -l), it can not be red when your_container.sif is activated

less: can't open, no such file or dir

On the other hand, process files (e.g. .command.sh) can be red

ADD REPLY
1
Entering edit mode

That's not a helpful diagnostic as you need to mount folders if running singularity manually -- something that nextflow does for you. Again, please provide a reproducible example that includes some minimal testing data and the Nextflow code. Singularity is used with Nextflow by thousands of users, so it must be something with how you read/stage/provide the input data.

ADD REPLY
0
Entering edit mode

Please provide a reproducible example for debugging.

ADD REPLY

Login before adding your answer.

Traffic: 3029 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6