Question: Add an item from input to an output array in CWL
0
gravatar for drkennetz
3 months ago by
drkennetz440
drkennetz440 wrote:

Hi all, I have a cwl workflow where step1 outputs an array of items file. In step 2 I would like to add a file or an array of files to the array that is output from step1. Here is my example workflow:

cwlVersion: v1.0    
class: Workflow    
requirements:
 - class: ScatterFeatureRequirement
 - class: StepInputExpressionRequirement
 - class: InlineJavascriptRequirement

inputs:
  file1: File[]
  file2: File[]
  input_added_in_step2: File
  string_input: string_input
steps:
  step1:
    run: step1.cwl
    scatter: [file1, file2]
    scatterMethod: dotproduct
    in:
      file1: file1
      file2: file2
    out:
      [output_array]
  step2:
    run: step2.cwl
    in:
      input_array:
        source: step1/output_array
        valueFrom: $(self.concat(inputs.input_added_in_step2))
      string_input: string_input
    out: [output_array_and_file_added_and_rename]
outputs:
  output_array_plus_file:
    type: File[]
    outputSource: step2/output_array_and_file_added_and_rename

The biggest issue here is that the workflow actually runs to completion, but it doesn't ever add input_added_in_step2 to the array created by step1. It just returns the array created by step1 with the string addition.

Thanks

cwl • 186 views
ADD COMMENTlink modified 11 weeks ago by Michael R. Crusoe1.7k • written 3 months ago by drkennetz440
2
gravatar for Tom
12 weeks ago by
Tom520
Bielefeld University, CeBiTec, Germany
Tom520 wrote:

In the valueFrom: $(self.concat(inputs.input_added_in_step2)) expression, inputs will reference the inputs of the workflow step, not the input section at the beginning of the workflow document. Looking at the valueFrom section in this segment of the specifications leads me to believe there is no way to reference input_added_in_step2 in the StepInputExpression. You will need to define the file as input for step2 or create a separate step (presumably an expressionTool) with the necessary inputs to prepare the array for step2.

A similar problem was discussed here, maybe the examples can help: Combine subject with sample type in a scatter?

ADD COMMENTlink modified 12 weeks ago • written 12 weeks ago by Tom520
3
gravatar for Michael R. Crusoe
11 weeks ago by
Common Workflow Language project
Michael R. Crusoe1.7k wrote:

You can combine multiple inbound inputs using https://www.commonwl.org/v1.0/Workflow.html#MultipleInputFeatureRequirement

Try this:

cwlVersion: v1.0    
class: Workflow    
requirements:
 - class: ScatterFeatureRequirement
 - class: StepInputExpressionRequirement
 - class: MultipleInputFeatureRequirement

inputs:
  file1: File[]
  file2: File[]
  input_added_in_step2: File
  string_input: string_input
steps:
  step1:
    run: step1.cwl
    scatter: [file1, file2]
    scatterMethod: dotproduct
    in:
      file1: file1
      file2: file2
    out:
      [output_array]
  step2:
    run: step2.cwl
    in:
      input_array:
        source: [ step1/output_array,  input_added_in_step2 ]
        linkMerge: merge_flattened
      string_input: string_input
    out: [output_array_and_file_added_and_rename]
outputs:
  output_array_plus_file:
    type: File[]
    outputSource: step2/output_array_and_file_added_and_rename
ADD COMMENTlink written 11 weeks ago by Michael R. Crusoe1.7k
1

This is great! I didn't know about linkMerge, wish i would have learned this sooner.

ADD REPLYlink written 11 weeks ago by Tom520

Thanks Tom, maybe you could help us write a section in the CWL user guide about this?

https://github.com/common-workflow-language/user_guide/issues/103

You don't have to make a perfect lesson, even just writing what you can is helpful; we'll clean up and format as necessary :-)

ADD REPLYlink written 11 weeks ago by Michael R. Crusoe1.7k
2

I would be very happy to do so. Since i have to submit my master's thesis this month i'm a little stressed out and won't be able to take care of it right now. I have made a calendar entry to make sure i won't forget.

I spent a lot of time reading and writing CWL documents in the past months. A more in-depth user guide would have saved me some some long hours of being confused. It's great that you are expanding it!

*edit: Learning about linkMerge has just allowed me to purge an abhorrent ExpressionTool from one of my workflows, which makes me very happy :)

ADD REPLYlink modified 11 weeks ago • written 11 weeks ago by Tom520

Congratulations on your pending thesis! Yes, no rush. Any contribution is appreciated!

ADD REPLYlink written 11 weeks ago by Michael R. Crusoe1.7k
1

The merge_flattened is actually something I found on another one of your posts that ended up solving my problems:

CWL: combine two arrays as workflow step input

Super helpful for me!

ADD REPLYlink written 11 weeks ago by drkennetz440
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1830 users visited in the last hour