First nextflow script and I'm facing alot of errors, is my approach even correct?
1
0
Entering edit mode
4 months ago
Faith ▴ 50

Hello,

I'm trying to write a nextflow script that automates a workflow that will run on a high computing cloud and local computers. The script requires only Bash and Python to be used.

What I'm trying to do is:

  1. Load server modules, if it fails because we're on a local computer it just skips that process.

2.Create a conda environment from the yml file that is in the same directory as the nextflow script.

3.Python process in that activated conda environment.

My code: --> This runs in the command line: nextflow run demo_2.nf -with-conda

#!/usr/bin/env nextflow

//process getDirectory{

    """
    // Skip this step for now
    """
//}

process loadModules {

 script:
    """
    echo "Step 1/10 Loading modules if we're on a server - skip if fails"

    ml anaconda3/2023.03 qiime2/2023.5

    echo "Step 2/10 - Creating Conda environment"

    conda env create -f automation_env.yml 

    conda activate automation

    echo "Step 2/10 - Completed"

    """
}


process pythonManifestCreation {
    script:
    """
    #!/usr/bin/env python

    def main():
    # Importing important libraries
        try:
            import pandas as pd
            from pathlib import Path
            import shutil
            import glob
            import os
            print("All necessary modules imported successfully.")
        except ImportError as e:
            print(f"An error occurred while importing modules: {e}")

        main_directory = os.getwd()

output:
path 'r.csv'
}

workflow {

    // Defining the workflow

    loadModules()
    pythonManifestCreation()

}

Here are the errors that I get:

1. If I try to run the script as is, I get an error that conda command not found

Caused by:
  Process loadModules terminated with an error exit status (127)


Command executed:

  echo "Step 2/10 - Creating Conda environment"
  conda env create -f automation_env.yml 
  conda activate automation
  echo "Step 2/10 - Completed"

Command exit status:
  127

Command output:
  Step 2/10 - Creating Conda environment

Command error:
  Step 2/10 - Creating Conda environment
  .command.sh: line 3: conda: command not found

Work dir:
  /home/s/Desktop/nextflow/work/b0/6548e83f5d1fd52282a7828c54075c

2. If I try to run just python script without loading any modules (Because I already have anaconda on my system and the environment is already created! I get this error:

Command exit status:
  127

Command output:
  (empty)

Command error:
  /usr/bin/env: ‘python’: No such file or directory

I have also tried to write a process that captures the directory = \$pwd and pipes it as output: var directory

Everytime I try to pass it into other functions I also get an error for that.

I know this is a very long post, but could someone point me to what I'm doing wrong and how can I approach it better?

Conda Bash Python Groovy Nextflow • 480 views
ADD COMMENT
5
Entering edit mode
4 months ago

Loading modules in HPC with Nextflow is usually done through the beforeScript process directive (ref here). As for conda package/environment management, in Nextflow it is done through the conda process directive (ref here). The code you shared makes me believe you're not fully aware of how Nextflow works, and this can make your journey with Nextflow start on the wrong foot. I strongly recommend you go through the Nextflow fundamentals training (or at least the Hello Nextflow training) before trying anything else with Nextflow. I assure you it will save you a lot of time ;)

ADD COMMENT

Login before adding your answer.

Traffic: 1780 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6