Question: Staged files not visible with DockerRequirement
0
gravatar for psaffrey
2.8 years ago by
psaffrey20
psaffrey20 wrote:

I have been creating files on-the-fly using the InitialWorkDirRequirement as described here:

http://www.commonwl.org/v1.0/UserGuide.html#Staging_input_files_in_the_output_directory

This works but I find that when I combine this with a DockerRequirement it stops working and I can no longer find the created file. Example:

class: CommandLineTool
cwlVersion: v1.0
baseCommand: ["cat"]

requirements:
  - class: DockerRequirement
    dockerImageId: alpine
    dockerPull: alpine
  - class: InlineJavascriptRequirement
  - class: InitialWorkDirRequirement
    listing:
      - entryname: example.conf
        entry: |
          ${
            return inputs.test
          }


inputs:
  - id: test
    type: string
    inputBinding:
      valueFrom: "example.conf"
outputs: []

Input:

test: "test string"

If I do the same thing without the DockerRequirement I get:

/usr/local/bin/cwltool 1.0.20161216212910
Resolved 'javascript_file2.cwl' to 'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file2.cwl'
[job javascript_file2.cwl] /private/var/folders/dh/t5qjl_gx5tq3664vlgbgqplh0000gq/T/tmpYfWsl4$ cat \
    example.conf
test stringFinal process status is success
{}

With the DockerRequirement I get this:

/usr/local/bin/cwltool 1.0.20161216212910
Resolved 'javascript_file2.cwl' to 'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file2.cwl'
[job javascript_file2.cwl] /var/folders/dh/t5qjl_gx5tq3664vlgbgqplh0000gq/T/tmpgx_NP_$ docker \
    run \
    -i \
    --volume=/private/var/folders/dh/t5qjl_gx5tq3664vlgbgqplh0000gq/T/tmpgx_NP_:/private/var/spool/cwl:rw \
    --volume=/private/var/folders/dh/t5qjl_gx5tq3664vlgbgqplh0000gq/T/tmpRpeEVS:/tmp:rw \
    --workdir=/private/var/spool/cwl \
    --read-only=true \
    --user=1000 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/private/var/spool/cwl \
    alpine \
    cat \
    example.conf
cat: can't open 'example.conf': No such file or directory
[job javascript_file2.cwl] completed permanentFail
Final process status is permanentFail
Workflow error, try again with --debug for more information:
Process status is ['permanentFail']

This looks like it might be related to this post:

Create writable directory inside container before executing baseCommand. How to properly use InitialWorkDirRequirement

which has a few (unclosed) issues on github. Any thoughts?

Peter

cwl dockerrequirement • 1.0k views
ADD COMMENTlink modified 2.8 years ago by Michael R. Crusoe1.7k • written 2.8 years ago by psaffrey20

Alas, I can't get this to replicate:

(env2) michael@dan:~/cwltool$ cwltool test-2.cwl test-2.yaml 
/home/michael/cwltool/env2/bin/cwltool 1.0.20161221171240
Resolved 'test-2.cwl' to 'file:///home/michael/cwltool/test-2.cwl'
[job test-2.cwl] /tmp/tmpAOyyHm$ docker \
    run \
    -i \
    --volume=/tmp/tmpAOyyHm:/var/spool/cwl:rw \
    --volume=/tmp/tmpDURmMT:/tmp:rw \
    --workdir=/var/spool/cwl \
    --read-only=true \
    --user=1000 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/var/spool/cwl \
    alpine \
    cat \
    example.conf
test stringFinal process status is success
{}

What OS & docker version are you using?

ADD REPLYlink modified 2.8 years ago • written 2.8 years ago by Michael R. Crusoe1.7k
$ docker --version
Docker version 1.10.1, build 9e83765
$ echo $OSTYPE 
darwin15
$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G1212
ADD REPLYlink modified 2.8 years ago • written 2.8 years ago by psaffrey20

Mine seems to be going into /private/var/spool/cwl instead of /var/spool/cwl - is that an OS-specific thing?

ADD REPLYlink modified 2.8 years ago • written 2.8 years ago by psaffrey20
0
gravatar for Michael R. Crusoe
2.8 years ago by
Common Workflow Language project
Michael R. Crusoe1.7k wrote:

This appears to be an OS X specific bug, thank you for finding it! Can you file an issue at https://github.com/common-workflow-language/cwltool/issues/new?

ADD COMMENTlink written 2.8 years ago by Michael R. Crusoe1.7k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 912 users visited in the last hour