Question: CWL :File created on fly by one step of a workflow as input to another step
0
gravatar for akashmdr90
3.3 years ago by
akashmdr900
akashmdr900 wrote:

I am new into CWL coding, I am trying to implement a workflow and i ran across some issues . I am not able to mention the output file name outputted by one step as input for another step in my YML script . It says that such file is not present in the folder already. That file is being generated on "output7". I tried with staging the input file also , but that also throws error.I am attaching both CWL and YML file along with this post.

CWL :

    cwlVersion: cwl:draft-3
    class: Workflow
    inputs:
      - id: output7
        type: string
      - id: reference7
        type: File
      - id: output8
        type: string
      - id: reference9
        type: File
      - id: numThreads9
        type: int
      - id: input9
        type: File
      - id: output9
        type: string




    outputs:
      - id: workflowOutput
        type: File
        source: "#picard-ReorderSam/result"
      - id: workflowOutput1
        type: File
        source: "#samtoolsIndex/result"
      - id: workflowOutput2
        type: File
        source: "#gatk-RealignerTargetCreator/result"


    steps:
  - id: picard-ReorderSam
   run: picard-ReorderSam.cwl
   inputs:
      - id: input
        source: "#picard-AddorRep/result"
      - id: output
        source: "#output7"
      - id: reference
        source: "#reference7"
   outputs:
      - id: result


 - id: samtoolsIndex
   run: samtools-index.cwl
   inputs:
      - id: input
        source: "#picard-ReorderSam/result"
      - id: output
        source: "#output8"
   outputs:
      - id: result
 - id: gatk-RealignerTargetCreator
   run: gatk-RealignerTargetCreator.cwl
   inputs:
      - id: reference
        source: "#reference9"
      - id: numThreads
        source: "#numThreads9"
      - id: input
        source: "#input9"
      - id: output
        source: "#output9"
   outputs:
      - id: result

YAML script :

output7: excapgene1normoutstp8.bam
reference7:
  class: File
  path: /home/cc/ReferenceData/genome_sorted.fa
output8: excapgene1normoutstp8.bai
reference9:
  class: File
  path: /home/cc/ReferenceData/genome_sorted.fa
numThreads9: 10
input9:
  class: File
  path: excapgene1normoutstp8.bam
output9: realignment_targets.list

Error message generated :

ValidationException: exomecap-part1-job.yml:43:3: [Errno 2] No such file or directory: '/home/cc/Test_CWL/excapgene1normoutstp8.bam'
[step gatk-RealignerTargetCreator] Cannot make job: exomecap-part1-job.yml:43:3: [Errno 2] No such file or directory: '/home/cc/Test_CWL/excapgene1normoutstp8.bam'
[workflow exomecap-part1.cwl] outdir is /tmp/tmpE4Tn6Q
{}
Final process status is permanentFail

I tried to stage the file without outputting the file into the file and inputting for each step , am not able to stage the output from output7 to input9.

CWL :

cwlVersion: cwl:draft-3
class: Workflow
inputs:
  - id: output7
    type: string
  - id: reference7
    type: File
  - id: output8
    type: string
  - id: reference9
    type: File
  - id: numThreads9
    type: int
  - id: output9
    type: string


outputs:
  - id: workflowOutput
    type: File
    source: "#gatk-RealignerTargetCreator/result"


steps:

 - id: picard-ReorderSam
   run: picard-ReorderSam.cwl
   inputs:
      - id: input
        source: "#picard-AddorRep/result"
      - id: output
        source: "#output7"
      - id: reference
        source: "#reference7"
   outputs:
      - id: result


 - id: samtoolsIndex
   run: samtools-index.cwl
   inputs:
      - id: input
        source: "#picard-ReorderSam/result"
      - id: output
        source: "#output8"
   outputs:
      - id: result

 - id: gatk-RealignerTargetCreator
   run: gatk-RealignerTargetCreator.cwl
   inputs:
      - id: reference
        source: "#reference9"
      - id: numThreads
        source: "#numThreads9"
      - id: input
        source: "#picard-ReorderSam/result"
      - id: output
        source: "#output9"
   outputs:
      - id: result

YML script :

output7: excapgene1normoutstp8.bam
reference7:
  class: File
  path: /home/cc/ReferenceData/genome_sorted.fa
output8: excapgene1normoutstp8.bai
reference9:
  class: File
  path: /home/cc/ReferenceData/genome_sorted.fa
numThreads9: 10
output9: realignment_targets.list

Error generated:

ValidationException: [Errno 2] No such file or directory: '/tmp/tmp06gkDT/excapgene1normoutstp8.bai'
[step gatk-RealignerTargetCreator] Cannot make job: [Errno 2] No such file or directory: '/tmp/tmp06gkDT/excapgene1normoutstp8.bai'
[workflow exomecap-part1.cwl] outdir is /tmp/tmpuJgV1v
{}
Final process status is permanentFail

I tried to stage the input files from previous step output but for processing the "gatk-RealignerTargetCreator" it requires ".Bai" file outputted by SamtoolIndex as secondary file along with the input file "excapgene1normoutstp8.bam " which is outputed by "picard-ReorderSam".

Please kindly help me to solve this issue.

cwl common workflow language • 1.2k views
ADD COMMENTlink modified 3.3 years ago by Biostar ♦♦ 20 • written 3.3 years ago by akashmdr900

Hello akashmdr90,

The first workflow listed is missing the picard-AddorRep step. Also, can you link to a GitHub repo with these files? It is hard to debug without having access to the underlying too definitions.

ADD REPLYlink written 3.3 years ago by Michael R. Crusoe1.8k

Also, may I ask why you are using the draft-3 CWL syntax? I don't know of any system that supports draft-3 version of CWL and doesn't also support the v1.0 version that came after it. You can use the cwl-upgrader to upgrade your draft-3 files quickly to v1.0: https://pypi.python.org/pypi/cwl-upgrader

ADD REPLYlink written 3.3 years ago by Michael R. Crusoe1.8k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1761 users visited in the last hour