CWL :File created on fly by one step of a workflow as input to another step

Question

[akashmdr90]

I am new into CWL coding, I am trying to implement a workflow and I ran across some issues . I am not able to mention the output file name outputted by one step as input for another step in my YML script . It says that such file is not present in the folder already. That file is being generated on "output7". I tried with staging the input file also , but that also throws error.I am attaching both CWL and YML file along with this post.

CWL:

    cwlVersion: cwl:draft-3
    class: Workflow
    inputs:
      - id: output7
        type: string
      - id: reference7
        type: File
      - id: output8
        type: string
      - id: reference9
        type: File
      - id: numThreads9
        type: int
      - id: input9
        type: File
      - id: output9
        type: string

    outputs:
      - id: workflowOutput
        type: File
        source: "#picard-ReorderSam/result"
      - id: workflowOutput1
        type: File
        source: "#samtoolsIndex/result"
      - id: workflowOutput2
        type: File
        source: "#gatk-RealignerTargetCreator/result"

    steps:
  - id: picard-ReorderSam
   run: picard-ReorderSam.cwl
   inputs:
      - id: input
        source: "#picard-AddorRep/result"
      - id: output
        source: "#output7"
      - id: reference
        source: "#reference7"
   outputs:
      - id: result

 - id: samtoolsIndex
   run: samtools-index.cwl
   inputs:
      - id: input
        source: "#picard-ReorderSam/result"
      - id: output
        source: "#output8"
   outputs:
      - id: result
 - id: gatk-RealignerTargetCreator
   run: gatk-RealignerTargetCreator.cwl
   inputs:
      - id: reference
        source: "#reference9"
      - id: numThreads
        source: "#numThreads9"
      - id: input
        source: "#input9"
      - id: output
        source: "#output9"
   outputs:
      - id: result

YAML script :

    output7: excapgene1normoutstp8.bam
    reference7:
      class: File
      path: /home/cc/ReferenceData/genome_sorted.fa
    output8: excapgene1normoutstp8.bai
    reference9:
      class: File
      path: /home/cc/ReferenceData/genome_sorted.fa
    numThreads9: 10
    input9:
      class: File
      path: excapgene1normoutstp8.bam
    output9: realignment_targets.list

**Error message generated **:

    ValidationException: exomecap-part1-job.yml:43:3: [Errno 2] No such file or directory: '/home/cc/Test_CWL/excapgene1normoutstp8.bam'
    [step gatk-RealignerTargetCreator] Cannot make job: exomecap-part1-job.yml:43:3: [Errno 2] No such file or directory: '/home/cc/Test_CWL/excapgene1normoutstp8.bam'
    [workflow exomecap-part1.cwl] outdir is /tmp/tmpE4Tn6Q
    {}
    Final process status is permanentFail

I tried to stage the file without outputting the file into the file and inputting for each step , am not able to stage the output from output7 to input9.

CWL :

cwlVersion: cwl:draft-3
class: Workflow
inputs:
  - id: output7
    type: string
  - id: reference7
    type: File
  - id: output8
    type: string
  - id: reference9
    type: File
  - id: numThreads9
    type: int
  - id: output9
    type: string

outputs:
  - id: workflowOutput
    type: File
    source: "#gatk-RealignerTargetCreator/result"

steps:

 - id: picard-ReorderSam
   run: picard-ReorderSam.cwl
   inputs:
      - id: input
        source: "#picard-AddorRep/result"
      - id: output
        source: "#output7"
      - id: reference
        source: "#reference7"
   outputs:
      - id: result


 - id: samtoolsIndex
   run: samtools-index.cwl
   inputs:
      - id: input
        source: "#picard-ReorderSam/result"
      - id: output
        source: "#output8"
   outputs:
      - id: result

 - id: gatk-RealignerTargetCreator
   run: gatk-RealignerTargetCreator.cwl
   inputs:
      - id: reference
        source: "#reference9"
      - id: numThreads
        source: "#numThreads9"
      - id: input
        source: "#picard-ReorderSam/result"
      - id: output
        source: "#output9"
   outputs:
      - id: result

YML script :

output7: excapgene1normoutstp8.bam
reference7:
  class: File
  path: /home/cc/ReferenceData/genome_sorted.fa
output8: excapgene1normoutstp8.bai
reference9:
  class: File
  path: /home/cc/ReferenceData/genome_sorted.fa
numThreads9: 10
output9: realignment_targets.list

Error generated:

ValidationException: [Errno 2] No such file or directory: '/tmp/tmp06gkDT/excapgene1normoutstp8.bai'
[step gatk-RealignerTargetCreator] Cannot make job: [Errno 2] No such file or directory: '/tmp/tmp06gkDT/excapgene1normoutstp8.bai'
[workflow exomecap-part1.cwl] outdir is /tmp/tmpuJgV1v
{}
Final process status is permanentFail

I tried to stage the input files from previous step output but for processing the "gatk-RealignerTargetCreator" it requires ".Bai" file outputted by SamtoolIndex as secondary file along with the input file "excapgene1normoutstp8.bam " which is outputed by "picard-ReorderSam".

Please kindly help me to solve this issue.

[Michael R. Crusoe]

Hello akashmdr90,

The first workflow listed is missing the picard-AddorRep step. Also, can you link to a GitHub repo with these files? It is hard to debug without having access to the underlying too definitions.

[Michael R. Crusoe]

Also, may I ask why you are using the draft-3 CWL syntax? I don't know of any system that supports draft-3 version of CWL and doesn't also support the v1.0 version that came after it. You can use the cwl-upgrader to upgrade your draft-3 files quickly to v1.0: https://pypi.python.org/pypi/cwl-upgrader