Nextflow- cannot run bash script from Docker
4
0
Entering edit mode
6 months ago
davidmaimoun ▴ 50

Hi,

I would like to run a simple bash script I created and dockerised it. But when I want to run it via Nextflow, it throw me an error.

Could you help me please ?

Thank you

Dockerfile:

FROM ubuntu:kinetic-20220602

WORKDIR /scripts

COPY neisseria_typing.sh .  

Nextflow code:

#!/usr/bin/env nextflow

nextflow.enable.dsl=2

process fetchProfile {

    echo true

    input:
    path(sequence)

    """
    ./neisseria_typing.sh ${sequence}
    """
}

workflow {

    input_ch = Channel.fromPath(params.input)  

    fetchProfile(input_ch)
}

nextflow.config:

params.input = "sequence.fasta"

docker.enabled = true

docker.runOptions = '-u $(id -u):$(id -g)'


process {

    withName: fetchProfile {

        container = 'myDocker/genomics_scripts:v1.0'
    }

}

Error:

Command error:
  .command.sh: line 2: ./neisseria_typing.sh: No such file or directory
script bash nextflow from docker • 1.6k views
ADD COMMENT
2
Entering edit mode

Thank you guys. You are right, I don't see how Nextflow could get the scripts. I created images with, trimmomatic and skesa for instance, and it did work well so I assumed it will be the same with simple scripts that I wrote. Clearly I misunderstood how Nextflow works with Docker, I need to get back to docs :)

Thank you very much for your help

ADD REPLY
2
Entering edit mode

No worries. The beauty of Nextflow is that it exactly takes that burden from you. You just provide the code outside of any container and Nextflow then takes care of all the mounting and copying/staging. If you put the actual code on GitHub you have the full package, so code version control via GitHub, automatized processing by Nextflow and software-wise reproducibility via the container. Nextflow can directly pull repository code from GitHub, so essentially you could launch a pipeline without even copying the code manually once it is in a GitHub repo.

ADD REPLY
0
Entering edit mode

Very nice! Thanks for the explanations

ADD REPLY
3
Entering edit mode
6 months ago
Maxime Garcia ▴ 160

Why having the script in a Docker container? You can have it in the bin/ or lib/ folder of your script, and Nextflow will find it.

But I'm guessing you have your reasons to do so. If you do docker run myDocker/genomics_scripts:v1.0 neisseria_typing.sh ${sequence} is it working well? Can you check in your work dir, the .command.run, there should be a nxf_launch() function that show the actual docker command run by Nextflow for this particular process.

ADD COMMENT
0
Entering edit mode

Hi Maxime, I work with people that have PC with very limited access. I can't download programs, or run bash script. Soon we will start to work with cloud, and I want dockerize my scripts to use them on the cloud

When I run via docker run it is doesn't work (to make it more simple, the script echo "Hello World"):

   docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container 
   process: exec: "neisseria_typing.sh": executable file not found in $PATH: unknown.

But when I run docker via " run -it ", and "./neisseria_typing.sh" it works.

From nxf_launch():

docker run -i --cpus 1.0 -v /home/user1/Desktop/david/docker/genomics:/home/user1/Desktop/david/docker/genomics -v "$PWD":"$PWD" -w "$PWD" --entrypoint /bin/bash -u $(id -u):$(id -g) --name $NXF_BOXID myDocker/genomics_scripts:v1.0 -c "/bin/bash -ue /home/user1/Desktop/david/docker/genomics/work/f4/57fa67ab2900a7cd8b2c1c5ca11abe/.command.sh"

Thank you !

ADD REPLY
2
Entering edit mode
6 months ago
ATpoint 68k

This is a bit non-standard how you do it. The thing is that nextflow looks (afaik) for scripts and binaries relative to its launchDir, so the directory is was launched from, and that is outside of the Docker container. It would be probably easier to put that bash script into the launchDir and then launch like $launchDir/your_script.sh. It is not really necessary to copy it into the container. I am not even sure nextflow would find it there (others may correct me). Basically, at least the way I write my nextflow stuff, Docker really only holds the software, and everything else comes from the actual file/host system, so input data etc. So the actual execution will still be in Docker, but the script can sit on your host filesystem.

ADD COMMENT
1
Entering edit mode

This makes sense. Nextflow just can't find the script in it's current work directory.

You might just want to do this then in the nextflow process (which uses symbolic links of all files specified in input to a different work directory for each process to my knowledge)

cp $launchDir/your_script.sh .
bash your_script.sh
ADD REPLY
2
Entering edit mode

Nextlow automatically mount the bin/ and lib/ folder from the $launchDir, would make more sense to put the bach script there, and use a bland container.

ADD REPLY
2
Entering edit mode
6 months ago

Also mind that your Dockerfile lacks any ENTRYPOINT and CMD statements, so that will not work. Here is an example how to execute a custom script with parameters inside a Docker container. Once that is fixed and works, you can try to embed that into a Nextflow script.

ADD COMMENT
2
Entering edit mode
6 months ago
davidmaimoun ▴ 50

Thank you guys,

Based on your very good advises and explanations, I was able to understand better how it is working.

I managed to run my script by :

  • In the Dockerfile, putting the script in bin: COPY [your script] ‘./bin’

  • In Nextflow process, in script: bash /bin/[your script]

But like you said, it is easier to put that bash script into the launchDir, at least for what I need to do for my project.

Again Thanks!

ADD COMMENT

Login before adding your answer.

Traffic: 2077 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6