Question: CWL: Using the "EnvVarRequirement" to pass an environment variable to a script
1
gravatar for skanwal
13 months ago by
skanwal50
skanwal50 wrote:

Hello,

I am running a bash script inside a docker container that expects DEPLOY_ENV variable to be set to an appropriate value. However, when running cwltool ./runFolderCheck.cwl ./runFolderCheck-job.yml, I get:

INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430 INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl' ERROR Got workflow error Traceback (most recent call last): File "/Users/kanwals/miniconda3/envs/cwl/lib/python3.6/site-packages/cwltool/executors.py", line 169, in run_jobs for job in jobiter: File "/Users/kanwals/miniconda3/envs/cwl/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 506, in job self.updatePathmap(builder.outdir, builder.pathmapper, l) File "/Users/kanwals/miniconda3/envs/cwl/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 295, in updatePathmap for sf in fn.get("secondaryFiles", []): AttributeError: 'str' object has no attribute 'get' ERROR Workflow error, try again with --debug for more information: 'str' object has no attribute 'get'

And with the --debug flag:

INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430 INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl' usage: ./runFolderCheck.cwl [-h] --denv DENV [--input_folder INPUT_FOLDER] [job_order] ./runFolderCheck.cwl: error: the following arguments are required: --denv

The cwl-tool definition runFolderCheck.cwl, I am using is:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool

hints:
 DockerRequirement:
   dockerImageId: umccr/pipeline

requirements:
  EnvVarRequirement:
    envDef:
      DEPLOY_ENV: $(inputs.denv)
  InitialWorkDirRequirement:
    listing: $(inputs.input_folder)

inputs:    
  denv: string

  input_folder:
    type: string?
    inputBinding:
      position: 1

outputs:
  log_out:
    type: stdout

stdout: output/output_runFOlderCheck.txt

baseCommand: [bash, '-c', /scripts/runfolder-check.sh]

And the input yaml runFolderCheck-job.yml is:

denv: dev

input_folder: /Users/kanwals/Documents/UMCCR/data/bcl2fastq

Any help to solve the issue will be appreciated.


Sehrish K

ADD COMMENTlink modified 13 months ago • written 13 months ago by skanwal50
1
gravatar for Tom
13 months ago by
Tom530
Tom530 wrote:

Hi Sehrish!

There seems to be an issue with the InitialWorkDirRequirement. You cannot pass a string to the listings field. It has to be either files, directories or dirents. We can, however, build a dirent for an empty directory named after the input_folder string:

requirements:
  EnvVarRequirement:
    envDef:
      DEPLOY_ENV: $(inputs.denv)
  InlineJavascriptRequirement: {}
  InitialWorkDirRequirement:
    listing:
      - entry: "$({class: 'Directory', listing: []})"
        entryname: $(inputs.input_folder)
        writable: true

I can't do any testing because i do not have the docker container you are using, but this should fix the error you are describing.

Best regards,

Tom

ADD COMMENTlink modified 13 months ago • written 13 months ago by Tom530
0
gravatar for skanwal
13 months ago by
skanwal50
skanwal50 wrote:

Hi Tom,

Thanks for the response. Changing the tool definition according to your suggestion is raising the following issue:

INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430 INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl' ERROR Workflow error, try again with --debug for more information: No mandatory DockerRequirement, yet path is outside the designated output directory, also know as $(runtime.outdir): MapperEnt(resolved='_:23a2471b-ce8d-4d95-84c6-27b195f675fc', target='/Users/kanwals/Documents/UMCCR/data/bcl2fastq', type='WritableDirectory', staged=True)

However, trying:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool

hints:
 DockerRequirement:
   dockerImageId: umccr/pipeline

requirements:
  EnvVarRequirement:
    envDef:
      DEPLOY_ENV: $(inputs.denv)
  InitialWorkDirRequirement:
    listing: $(inputs.input_folder.listing)

inputs:    
  denv: string

  input_folder:
    type: Directory
    inputBinding:
      position: 1

outputs:
  log_out:
    type: stdout

stdout: scripts/runfolder-check.sh.dev.log

baseCommand: [bash, '-c', /scripts/runfolder-check.sh]

Seems to be progressing a little more and I get:

$ cwltool ./runFolderCheck.cwl ./runFolderCheck-job.yml
INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430
INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl'
INFO [job runFolderCheck.cwl] /private/tmp/docker_tmp3ifase1d$ docker \
    run \
    -i \
    --volume=/private/tmp/docker_tmp3ifase1d:/XCMkDM:rw \
    --volume=/private/tmp/docker_tmpo87g08oy:/tmp:rw \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq:/var/lib/cwl/stgf816efbf-5870-4786-bc21-856ea92e2ddd/bcl2fastq:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/RTAComplete.txt:/XCMkDM/RTAComplete.txt:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/190704_A00130_0109_BHLHLWDSXX:/XCMkDM/190704_A00130_0109_BHLHLWDSXX:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SequenceComplete.txt:/XCMkDM/SequenceComplete.txt:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SampleSheet.csv.custom.1.truseq:/XCMkDM/SampleSheet.csv.custom.1.truseq:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/output:/XCMkDM/output:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/CopyComplete.txt:/XCMkDM/CopyComplete.txt:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SampleSheet.csv.custom.2.10X:/XCMkDM/SampleSheet.csv.custom.2.10X:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SampleSheet.csv:/XCMkDM/SampleSheet.csv:ro \
    --workdir=/XCMkDM \
    --read-only=true \
    --log-driver=none \
    --user=1457398319:2094513965 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/XCMkDM \
    --cidfile=/private/tmp/docker_tmp5erahzgb/20191007223553-774195.cid \
    --env=DEPLOY_ENV=dev \
    umccr/pipeline \
    bash \
    -c \
    /scripts/runfolder-check.sh \
    /var/lib/cwl/stgf816efbf-5870-4786-bc21-856ea92e2ddd/bcl2fastq > /private/tmp/docker_tmp3ifase1d/scripts/runfolder-check.sh.dev.log
/scripts/runfolder-check.sh: line 31: /scripts/runfolder-check.sh.dev.log: Read-only file system
INFO [job runFolderCheck.cwl] Max memory used: 0MiB
WARNING [job runFolderCheck.cwl] completed permanentFail
{
    "log_out": {
        "location": "file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runfolder-check.sh.dev.log",
        "basename": "runfolder-check.sh.dev.log",
        "class": "File",
        "checksum": "sha1$dce702ab890b6b1974157b1437d75482a2fea8cd",
        "size": 85,
        "path": "/Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runfolder-check.sh.dev.log"
    }
}
WARNING Final process status is permanentFail

Any idea why it could be failing now?

ADD COMMENTlink modified 13 months ago • written 13 months ago by skanwal50
2

/scripts/runfolder-check.sh: line 31: /scripts/runfolder-check.sh.dev.log: Read-only file system

The script is trying to write a log file in its directory. It should either use $TMPDIR, or write the log to a user specified path.

ADD REPLYlink written 13 months ago by Michael R. Crusoe1.8k
2

Hi Michael,

Thanks a lot for the response. I have edited the bash script to write to the standard output and using stdout in the cwl definition to write it to a file. This works as expected.


Sehrish K

ADD REPLYlink written 13 months ago by skanwal50
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 944 users visited in the last hour