Question: CWL in toil
0
gravatar for a.james
4 weeks ago by
a.james180
Germany
a.james180 wrote:

Dear All,

I am using CWL workflow in toil. And it is not recognising the path environment, which otherwise is fine. I mean, if I run the workflow as cwltool it is finishing successfully. When I try it in Toil it is not recognising the path env. The command line tool: sp.cwl

    cwlVersion: v1.0
    class: CommandLineTool
    doc: Spladder

    baseCommand: [python, -m, spla.spla]

    requirements:
     - class: InlineJavascriptRequirement
     - class: InitialWorkDirRequirement
       listing:
        - entry: "$({class: 'Directory', listing: []})"
          entryname: $(inputs.split_outDir)
          writable: true
     - class: EnvVarRequirement
       envDef:
        - envName: PYTHONPATH
          envValue: $(inputs.spl_pythonpath)

 inputs:
    ......
     split_outDir:
      type: string
      spl_pythonpath:
       type: string

output:
spla_out:
  type: File
  outputBinding:
   glob: $(inputs.split_outDir)/splr/*.pi

The YML file:

    spl_pythonpath: /cluster/home/user/spl/python 
    split_outDir: splice

The workflow which I used to call the above commandline tool,

    #!/usr/bin/env cwl-runner
    cwlVersion: v1.0
    class: Workflow

    requirements:
     - class: ScatterFeatureRequirement
     - class: SubworkflowFeatureRequirement
     - class: InlineJavascriptRequirement
     - class: MultipleInputFeatureRequirement

    inputs:
     .......
..........

    outputs:
     spla_dir:
       type: Directory
       outputSource: collect/spla_dir

    steps:
      spl:
        run: sp.cwl
        scatter: [spla_bam]
        scatterMethod: dotproduct
        in:
       ........
..................
         split_outDir: spladder_outDir
         spl_pythonpath: spl_pythonpath
        out: [spla_out]
      collect:
        in:
          spla_bam_files:
            source: [spl/spla_out]
            linkMerge: merge_flattened
        out: [spla_out]
        run:
          class: ExpressionTool
          id: "collect_step"
          inputs:
            spladder_bam_files: File[]
          outputs:
            spla_dir: Directory
          expression: |
           ${
            return {
              "spla_dir": {
                 "class": "Directory",
                 "basename": "Splicing/spladder",
                 "listing": [].concat.apply([], inputs.spladder_bam_files)
              }
            };
            }

Then I used Toil to run the above workflow as,

(toil-cwl-runner --stats --clusterStats --retryCount=0 --batchSystem=lsf --disableCaching --tmpdir-prefix ${TMP_DIR} --tmp-outdir-prefix ${TMP_OUT_DIR} --workDir ${WORK_DIR} --realTimeLogging --cleanWorkDir=never --clean=never --outdir ${OUT_DIR} --logDebug  --logFile ${OUT_DIR}/CWL.log --jobStore ${JOB_STORE} pipeline_main_parallel.cwl pipeline_main_example.yml)>& CWL.${DATE}.log &

Which is throwing error at,

330756-WARNING:toil.leader:E/b/jobUIrJ3T        y
330799-WARNING:toil.leader:E/b/jobUIrJ3T    /cluster/home/user/software/anaconda/envs/py2/bin/python2.7: can't open file '/cluster/home/user/spl/python/spla.py': [Errno 2] No such file or directory

The tool is using 10 parameter and the after the 10th parameter the python path is defined in inputs. Toil is not running it as it is defined in base command toil should run it as python -m spla.spla within the defined path.

cwl toil • 95 views
ADD COMMENTlink modified 4 weeks ago • written 4 weeks ago by a.james180

Hello a.james.

Does it work when you use the CWL reference runner, cwltool?

Also, neither cwltool nor toil support conda for installation. We both recommend the use of a fresh Python virtualenv. That may fix your issue.

ADD REPLYlink written 28 days ago by Michael R. Crusoe1.4k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 2259 users visited in the last hour