CWL: Using the "EnvVarRequirement" to pass an environment variable to a script
2
1
Entering edit mode
5.1 years ago
skanwal ▴ 50

Hello,

I am running a bash script inside a docker container that expects DEPLOY_ENV variable to be set to an appropriate value. However, when running cwltool ./runFolderCheck.cwl ./runFolderCheck-job.yml, I get:

INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430
INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl'
ERROR Got workflow error Traceback (most recent call last):
   File "/Users/kanwals/miniconda3/envs/cwl/lib/python3.6/site-packages/cwltool/executors.py", line 169, in run_jobs
    for job in jobiter:   File "/Users/kanwals/miniconda3/envs/cwl/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 506, in job
    self.updatePathmap(builder.outdir, builder.pathmapper, l)
   File "/Users/kanwals/miniconda3/envs/cwl/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 295, in updatePathmap
    for sf in fn.get("secondaryFiles", []): AttributeError: 'str' object has no attribute 'get'
ERROR Workflow error, try again with --debug for more information: 'str' object has no attribute 'get'

And with the --debug flag:

INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430
INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl'
usage: ./runFolderCheck.cwl [-h] --denv DENV [--input_folder INPUT_FOLDER]
                            [job_order] ./runFolderCheck.cwl: error: the following arguments are required: --denv

The cwl-tool definition runFolderCheck.cwl, I am using is:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool

hints:
 DockerRequirement:
   dockerImageId: umccr/pipeline

requirements:
  EnvVarRequirement:
    envDef:
      DEPLOY_ENV: $(inputs.denv)
  InitialWorkDirRequirement:
    listing: $(inputs.input_folder)

inputs:    
  denv: string

  input_folder:
    type: string?
    inputBinding:
      position: 1

outputs:
  log_out:
    type: stdout

stdout: output/output_runFOlderCheck.txt

baseCommand: [bash, '-c', /scripts/runfolder-check.sh]

And the input yaml runFolderCheck-job.yml is:

denv: dev

input_folder: /Users/kanwals/Documents/UMCCR/data/bcl2fastq

Any help to solve the issue will be appreciated.

Sehrish K

Common-Workflow-Language CWL • 2.4k views
ADD COMMENT
1
Entering edit mode
5.1 years ago
Tom ▴ 540

Hi Sehrish!

There seems to be an issue with the InitialWorkDirRequirement. You cannot pass a string to the listings field. It has to be either files, directories or dirents. We can, however, build a dirent for an empty directory named after the input_folder string:

requirements:
  EnvVarRequirement:
    envDef:
      DEPLOY_ENV: $(inputs.denv)
  InlineJavascriptRequirement: {}
  InitialWorkDirRequirement:
    listing:
      - entry: "$({class: 'Directory', listing: []})"
        entryname: $(inputs.input_folder)
        writable: true

I can't do any testing because i do not have the docker container you are using, but this should fix the error you are describing.

Best regards,

Tom

ADD COMMENT
0
Entering edit mode
5.1 years ago
skanwal ▴ 50

Hi Tom,

Thanks for the response. Changing the tool definition according to your suggestion is raising the following issue:

INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430 INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl' ERROR Workflow error, try again with --debug for more information: No mandatory DockerRequirement, yet path is outside the designated output directory, also know as $(runtime.outdir): MapperEnt(resolved='_:23a2471b-ce8d-4d95-84c6-27b195f675fc', target='/Users/kanwals/Documents/UMCCR/data/bcl2fastq', type='WritableDirectory', staged=True)

However, trying:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool

hints:
 DockerRequirement:
   dockerImageId: umccr/pipeline

requirements:
  EnvVarRequirement:
    envDef:
      DEPLOY_ENV: $(inputs.denv)
  InitialWorkDirRequirement:
    listing: $(inputs.input_folder.listing)

inputs:    
  denv: string

  input_folder:
    type: Directory
    inputBinding:
      position: 1

outputs:
  log_out:
    type: stdout

stdout: scripts/runfolder-check.sh.dev.log

baseCommand: [bash, '-c', /scripts/runfolder-check.sh]

Seems to be progressing a little more and I get:

$ cwltool ./runFolderCheck.cwl ./runFolderCheck-job.yml
INFO /Users/kanwals/miniconda3/envs/cwl/bin/cwltool 1.0.20190915164430
INFO Resolved './runFolderCheck.cwl' to 'file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runFolderCheck.cwl'
INFO [job runFolderCheck.cwl] /private/tmp/docker_tmp3ifase1d$ docker \
    run \
    -i \
    --volume=/private/tmp/docker_tmp3ifase1d:/XCMkDM:rw \
    --volume=/private/tmp/docker_tmpo87g08oy:/tmp:rw \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq:/var/lib/cwl/stgf816efbf-5870-4786-bc21-856ea92e2ddd/bcl2fastq:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/RTAComplete.txt:/XCMkDM/RTAComplete.txt:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/190704_A00130_0109_BHLHLWDSXX:/XCMkDM/190704_A00130_0109_BHLHLWDSXX:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SequenceComplete.txt:/XCMkDM/SequenceComplete.txt:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SampleSheet.csv.custom.1.truseq:/XCMkDM/SampleSheet.csv.custom.1.truseq:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/output:/XCMkDM/output:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/CopyComplete.txt:/XCMkDM/CopyComplete.txt:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SampleSheet.csv.custom.2.10X:/XCMkDM/SampleSheet.csv.custom.2.10X:ro \
    --volume=/Users/kanwals/Documents/UMCCR/data/bcl2fastq/SampleSheet.csv:/XCMkDM/SampleSheet.csv:ro \
    --workdir=/XCMkDM \
    --read-only=true \
    --log-driver=none \
    --user=1457398319:2094513965 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/XCMkDM \
    --cidfile=/private/tmp/docker_tmp5erahzgb/20191007223553-774195.cid \
    --env=DEPLOY_ENV=dev \
    umccr/pipeline \
    bash \
    -c \
    /scripts/runfolder-check.sh \
    /var/lib/cwl/stgf816efbf-5870-4786-bc21-856ea92e2ddd/bcl2fastq > /private/tmp/docker_tmp3ifase1d/scripts/runfolder-check.sh.dev.log
/scripts/runfolder-check.sh: line 31: /scripts/runfolder-check.sh.dev.log: Read-only file system
INFO [job runFolderCheck.cwl] Max memory used: 0MiB
WARNING [job runFolderCheck.cwl] completed permanentFail
{
    "log_out": {
        "location": "file:///Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runfolder-check.sh.dev.log",
        "basename": "runfolder-check.sh.dev.log",
        "class": "File",
        "checksum": "sha1$dce702ab890b6b1974157b1437d75482a2fea8cd",
        "size": 85,
        "path": "/Users/kanwals/Documents/UMCCR/git/bcl2fastq/tools/runfolder-check.sh.dev.log"
    }
}
WARNING Final process status is permanentFail

Any idea why it could be failing now?

ADD COMMENT
2
Entering edit mode

/scripts/runfolder-check.sh: line 31: /scripts/runfolder-check.sh.dev.log: Read-only file system

The script is trying to write a log file in its directory. It should either use $TMPDIR, or write the log to a user specified path.

ADD REPLY
2
Entering edit mode

Hi Michael,

Thanks a lot for the response. I have edited the bash script to write to the standard output and using stdout in the cwl definition to write it to a file. This works as expected.


Sehrish K

ADD REPLY

Login before adding your answer.

Traffic: 936 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6