Question: CWL join two dimensional file array output
gravatar for silverspanch
12 days ago by
silverspanch0 wrote:

I have a CWL workflow that scatters a list of variables over a nested workflow which returns a file array. The output is then an array of file arrays, but it needs to be flattened for the next step. How do I gather the inputs from one step, and join the multidimensional file array into a single flat array?

scatter cwl workflow • 70 views
ADD COMMENTlink modified 10 days ago by Tom240 • written 12 days ago by silverspanch0

Did you consider using StepInputExpressionRequirement ( You can set a valueFrom using JS expression and transform a step input with $(self[0]) from, e.g. [ [ 1, 2 ] ] to [ 1, 2 ].

ADD REPLYlink modified 7 days ago • written 7 days ago by bogdan.gavrilovic210

Thanks, I didnt know about this functionality.

ADD REPLYlink written 5 days ago by silverspanch0
gravatar for Tom
10 days ago by
Bielefeld University, CeBiTec, Germany
Tom240 wrote:

I believe there is no built-in functionality for what you are asking.

I would suggest adding an ExpressionTool between the two workflow steps. I just tested the following and it seems to work:

 [in the steps section of your workflow]
      class: ExpressionTool
            type: array
              type: array
              items: File
            loadContents: true
          type: File[]
      expression: >
          var newArray= [];
          for (var i = 0; i < inputs.arrayTwoDim.length; i++) {
            for (var k = 0; k < inputs.arrayTwoDim[i].length; k++) {
          return { 'array1d' : newArray }
      arrayTwoDim: make2dArray/array2d
    out: [array1d]

It took me a while to figure out how cwl would like it's nested arrays described. Make sure to set the type of the output of the previous step as:

  type: array
    type: array
    items: File

Otherwise cwltool will throw a fit because the output is not compatible with the input of the ExpressionTool above. If you try passing type: File[] between the workflows steps cwl will, during runtime, realize that its actually passing along a nested array to a step where the input is supposed to be an array of files and abort.

ADD COMMENTlink modified 10 days ago • written 10 days ago by Tom240
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1090 users visited in the last hour