Split up record input when using scatter/gather

Question

[psaffrey]

I am building a CWL workflow that takes an array of record types as an input, in my case a list of pairs where each pair is a file and a string (term) that goes with it:

inputs:
- id: pairs
  type:
    items:
      fields:
      - {name: term, type: string}
      - {name: infile, type: File}
      type: record
    type: array

and then wants to pass only one part of the record type into a processing step. In the example above, let's say I want to filter all the files with a scattered grep step, but then I want to reassemble the filtered files with their paired term variable for later steps.

How do I pull out just the infile entry from the pairs to pass into grep? I've tried to use an expression when specifying inputs, like this:

steps:
  filter:
    run: grep.cwl
    in:
      invert: 
        default: true
      pattern:
        default: "^#"
      infile: 
        valueFrom: |
          ${
            return pairs.map(function(i) { return i.infile; });
          }
    scatter: infile
    out: [outfile]

but I get an error like this:

Unhandled exception
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/cwltool/workflow.py", line 304, in try_make_job
    Callable[[Any], Any],callback), **kwargs)
  File "/Library/Python/2.7/site-packages/cwltool/workflow.py", line 613, in dotproduct_scatter
    l = len(joborder[s])
TypeError: object of type 'NoneType' has no len()
Got workflow error
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/cwltool/main.py", line 226, in single_job_executor
    if r.outdir:
AttributeError: 'NoneType' object has no attribute 'outdir'
Workflow error, try again with --debug for more information:
'NoneType' object has no attribute 'outdir'

I'm not sure whether my approach even makes sense - can somebody help?

Answer 1

[Michael R. Crusoe]

Hello psaffrey,

You'll need to specify the input source and use the self in the Javascript Expression

steps:
  filter:
    run: grep.cwl
    in:
      invert: 
        default: true
      pattern:
        default: "^#"
      infile:
        source: someotherstep/pairs
        valueFrom: |
          ${
            return self.map(function(i) { return i.infile; });
          }
    scatter: infile
    out: [outfile]