Question: Split up record input when using scatter/gather
0
gravatar for psaffrey
3.2 years ago by
psaffrey20
psaffrey20 wrote:

I am building a CWL workflow that takes an array of record types as an input, in my case a list of pairs where each pair is a file and a string (term) that goes with it:

inputs:
- id: pairs
  type:
    items:
      fields:
      - {name: term, type: string}
      - {name: infile, type: File}
      type: record
    type: array

and then wants to pass only one part of the record type into a processing step. In the example above, let's say I want to filter all the files with a scattered grep step, but then I want to reassemble the filtered files with their paired term variable for later steps.

How do I pull out just the infile entry from the pairs to pass into grep? I've tried to use an expression when specifying inputs, like this:

steps:
  filter:
    run: grep.cwl
    in:
      invert: 
        default: true
      pattern:
        default: "^#"
      infile: 
        valueFrom: |
          ${
            return pairs.map(function(i) { return i.infile; });
          }
    scatter: infile
    out: [outfile]

but I get an error like this:

Unhandled exception
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/cwltool/workflow.py", line 304, in try_make_job
    Callable[[Any], Any],callback), **kwargs)
  File "/Library/Python/2.7/site-packages/cwltool/workflow.py", line 613, in dotproduct_scatter
    l = len(joborder[s])
TypeError: object of type 'NoneType' has no len()
Got workflow error
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/cwltool/main.py", line 226, in single_job_executor
    if r.outdir:
AttributeError: 'NoneType' object has no attribute 'outdir'
Workflow error, try again with --debug for more information:
'NoneType' object has no attribute 'outdir'

I'm not sure whether my approach even makes sense - can somebody help?

cwl scatter/gather • 1.6k views
ADD COMMENTlink modified 3.0 years ago by Michael R. Crusoe1.8k • written 3.2 years ago by psaffrey20
0
gravatar for Michael R. Crusoe
3.0 years ago by
Common Workflow Language project
Michael R. Crusoe1.8k wrote:

Hello psaffrey,

You'll need to specify the input source and use the self in the Javascript Expression

steps:
  filter:
    run: grep.cwl
    in:
      invert: 
        default: true
      pattern:
        default: "^#"
      infile:
        source: someotherstep/pairs
        valueFrom: |
          ${
            return self.map(function(i) { return i.infile; });
          }
    scatter: infile
    out: [outfile]
ADD COMMENTlink written 3.0 years ago by Michael R. Crusoe1.8k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1234 users visited in the last hour