Staged files not visible with DockerRequirement

Question

[psaffrey]

I have been creating files on-the-fly using the InitialWorkDirRequirement as described here:

http://www.commonwl.org/v1.0/UserGuide.html#Staging_input_files_in_the_output_directory

This works but I find that when I combine this with a DockerRequirement it stops working and I can no longer find the created file. Example:

class: CommandLineTool
cwlVersion: v1.0
baseCommand: ["cat"]

requirements:
  - class: DockerRequirement
    dockerImageId: alpine
    dockerPull: alpine
  - class: InlineJavascriptRequirement
  - class: InitialWorkDirRequirement
    listing:
      - entryname: example.conf
        entry: |
          ${
            return inputs.test
          }


inputs:
  - id: test
    type: string
    inputBinding:
      valueFrom: "example.conf"
outputs: []

Input:

test: "test string"

If I do the same thing without the DockerRequirement I get:

/usr/local/bin/cwltool 1.0.20161216212910
Resolved 'javascript_file2.cwl' to 'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file2.cwl'
[job javascript_file2.cwl] /private/var/folders/dh/t5qjl_gx5tq3664vlgbgqplh0000gq/T/tmpYfWsl4$ cat \
    example.conf
test stringFinal process status is success
{}

With the DockerRequirement I get this:

/usr/local/bin/cwltool 1.0.20161216212910
Resolved 'javascript_file2.cwl' to 'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file2.cwl'
[job javascript_file2.cwl] /var/folders/dh/t5qjl_gx5tq3664vlgbgqplh0000gq/T/tmpgx_NP_$ docker \
    run \
    -i \
    --volume=/private/var/folders/dh/t5qjl_gx5tq3664vlgbgqplh0000gq/T/tmpgx_NP_:/private/var/spool/cwl:rw \
    --volume=/private/var/folders/dh/t5qjl_gx5tq3664vlgbgqplh0000gq/T/tmpRpeEVS:/tmp:rw \
    --workdir=/private/var/spool/cwl \
    --read-only=true \
    --user=1000 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/private/var/spool/cwl \
    alpine \
    cat \
    example.conf
cat: can't open 'example.conf': No such file or directory
[job javascript_file2.cwl] completed permanentFail
Final process status is permanentFail
Workflow error, try again with --debug for more information:
Process status is ['permanentFail']

This looks like it might be related to this post:

Create writable directory inside container before executing baseCommand. How to properly use InitialWorkDirRequirement

which has a few (unclosed) issues on github. Any thoughts?

Peter

[Michael R. Crusoe]

Alas, I can't get this to replicate:

(env2) michael@dan:~/cwltool$ cwltool test-2.cwl test-2.yaml 
/home/michael/cwltool/env2/bin/cwltool 1.0.20161221171240
Resolved 'test-2.cwl' to 'file:///home/michael/cwltool/test-2.cwl'
[job test-2.cwl] /tmp/tmpAOyyHm$ docker \
    run \
    -i \
    --volume=/tmp/tmpAOyyHm:/var/spool/cwl:rw \
    --volume=/tmp/tmpDURmMT:/tmp:rw \
    --workdir=/var/spool/cwl \
    --read-only=true \
    --user=1000 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/var/spool/cwl \
    alpine \
    cat \
    example.conf
test stringFinal process status is success
{}

What OS & docker version are you using?

[psaffrey]

$ docker --version
Docker version 1.10.1, build 9e83765
$ echo $OSTYPE 
darwin15
$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G1212

[psaffrey]

Mine seems to be going into /private/var/spool/cwl instead of /var/spool/cwl - is that an OS-specific thing?

Answer 1

[Michael R. Crusoe]

This appears to be an OS X specific bug, thank you for finding it! Can you file an issue at https://github.com/common-workflow-language/cwltool/issues/new?