Advice on rendering a config file

Question

[psaffrey]

I have a tool which consumes a CSV input where each line is an ID and a label, eg.

6479933119,first

6479959170,second

My tool then uses the ID to pull some data from somewhere else, analyse it and label the output with the corresponding string. I want the user to be able to specify a variable number of rows where I would expect the maximum to be around 10. Basically I'm looking for advice about how to specify this in CWL.

I was planning to use dependent record types for each row:

http://www.commonwl.org/v1.0/UserGuide.html#Record_inputs,_dependent_and_mutually_exclusive_parameters

so I would need a record that has an int and a string, something like this:

  dependent_parameters:
    type:
      type: record
      name: sample_label_pairs
      fields:
        id: 
          type: int
        label: 
          type: string

except I need this to be an array of these records, so the user can specify N of them.

And then I thought I would turn it into a file as suggested here:

http://www.commonwl.org/v1.0/UserGuide.html#Creating_files_at_runtime

My tool also expects this file to be consumed at the command line with an @ prefix, in the same way that curl consumes data files.

So I guess I have a few questions:

• Does this sound like a sensible approach?
• How do I specify an array of records as an input? Is it as simple as record[]?
• How do I add an @ prefix to a command line argument? I guess if I use inputBinding: prefix: I'll get a space between the @ and the file?

Thanks,

Peter

[Michael]

Hi, I don't see any relation to bioinformatics, it looks like a very technical question on cwl syntax which renders this question pure programming and seems to be of very limited interest to users of this site. Please explain the connection to bioinformatics in your question, as Biostars is not a cwl support forum (afaik;).

You might get better help on Stackoverflow

Hint: this might be partly due to the fact that your question is obscured, what is "my tool" this might be essential for any forum to get a good answer.

[psaffrey]

My apologies - I followed the link from here:

http://www.commonwl.org/

where it says:

The recommended place to ask a question about all things CWL is on Biostars.

There do seem to be a lot of technical questions on this site regarding the low-level syntax of CWL, but if you tell me my question is not of interest I will try to take it elsewhere.

Can any CWL expert here recommend where I should post?

[Michael]

Hi, my apologies, I didn't know Biostars was explicitly mentioned as support forum. I recognized at least one or two names on the list of maintainers. If CWL is mainly intended as a bioinformatics tool, that should be fine. Please ignore my concern and continue posting. As a side note, it is very hard for moderators to know whether this site is used as an official support forum for a specific tool.

[psaffrey]

I've made a bit more progress on this and I'm now stuck mostly on making an array of records. This is what I have so far:

inputs:
  rows: 
    type:
      type: array
      items:
        type: 
          type: record
          fields:
            entity_id: 
              type: string
            label: 
              type: string

[psaffrey]

and it gives me:

Tool definition failed validation:
Validation error in object file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl
  Could not validate `CommandLineTool` because
    CommandLineTool record could not validate field `inputs` because
      At position 0
        could not validate field `type` because
          the value `{'items': {'type': {'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                                          'type[...]`
           is not a valid type in the union, expected one of:
          - null, but
             the value `{'items': {'type': {'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                                            'type[...]` is not null
          - CWLType, but
             the value `{'items': {'type': {'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                                            'type[...]`
             is not a valid symbol in enum CWLType, expected one of 'File', 'Directory'
          - CommandInputRecordSchema, but
             could not validate field `type` because
              the value `'array'`
               is not a valid symbol in enum Record_symbol, expected one of 'record'

            could not validate field `items` because it is not recognized and strict is True, valid fields are: fields, type, label
          - CommandInputEnumSchema, but
             missing required field `symbols`
            could not validate field `type` because
              the value `'array'`
               is not a valid symbol in enum Enum_symbol, expected one of 'enum'

            could not validate field `items` because it is not recognized and strict is True, valid fields are: symbols, type, label, inputBinding
          - CommandInputArraySchema, but
             could not validate field `items` because
              the value `{'type': {'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                                    'type': 'string'},
                    [...]`
               is not a valid type in the union, expected one of:
              - CWLType, but
                 the value `{'type': {'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                                      'type': 'string'},
                      [...]`
                 is not a valid symbol in enum CWLType, expected one of 'File', 'Directory'
              - CommandInputRecordSchema, but
                 could not validate field `type` because
                  the value `{'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                               'type': 'string'},
                              {'name': u'f[...]`
                   is not a valid symbol in enum Record_symbol, expected one of 'record'

              - CommandInputEnumSchema, but
                 missing required field `symbols`
                could not validate field `type` because
                  the value `{'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                               'type': 'string'},
                              {'name': u'f[...]`
                   is not a valid symbol in enum Enum_symbol, expected one of 'enum'

[psaffrey]

              - CommandInputArraySchema, but
                 missing required field `items`
                could not validate field `type` because
                  the value `{'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                               'type': 'string'},
                              {'name': u'f[...]`
                   is not a valid symbol in enum Array_symbol, expected one of 'array'

              - string, but
                 the value `{'type': {'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                                      'type': 'string'},
                      [...]` is not string
              - array of <CWLType or CommandInputRecordSchema or CommandInputEnumSchema or CommandInputArraySchema or string>, but
                 the value `{'type': {'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                                      'type': 'string'},
                      [...]` is not a list, expected list of CWLType or CommandInputRecordSchema or CommandInputEnumSchema or CommandInputArraySchema or string

          - string, but
             the value `{'items': {'type': {'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                                            'type[...]` is not string
          - array of <CWLType or CommandInputRecordSchema or CommandInputEnumSchema or CommandInputArraySchema or string>, but
             the value `{'items': {'type': {'fields': [{'name': u'file:///Users/psaffrey/cwl_dev/cwl_examples/javascript_file.cwl#rows/entity_id',
                                            'type[...]` is not a list, expected list of CWLType or CommandInputRecordSchema or CommandInputEnumSchema or CommandInputArraySchema or string

Answer 1

[Michael R. Crusoe]

For the @ prefix, use separate: false in your inputBinding

http://www.commonwl.org/v1.0/CommandLineTool.html#CommandLineBinding

[Istvan Albert]

May I suggest to let the CWL folks do their own thing for a while - I understand that everyone just wants to help - but they have been bombarded with directions of doing this or that, or why they post here and not somewhere else etc - and that too can feel a little overwhelming. Let's just give them more space.

[GenoMax]

@Michael: You may want to include your Biostar user tag (https: //www.biostars.org/u/15908/ ) on CWL website and ask people to tag you (just include that link in their post without the space in https: //) in the CWL related posts. As long as you have "follow by email" selected in your Biostar profile you should get an email every time you are tagged in a question.

[Michael R. Crusoe]

Hello genomax2,

I subscribe to the cwl tag feed, and the CWL community is bigger than just myself. A big reason to use BioStars instead of other mediums is to grow peer support and avoid gatekeepers.

If you or anyone else has an issue with CWL posts on BioStars please email me directly [e-mail] and CC [e-mail]

Users are feeling harassed by the push back and it feels like BioStars is not the welcoming , inclusive community I thought it was :-/

[GenoMax]

I was just trying to be helpful if you wanted to make CWL users aware of the "tagging" feature so you would not miss a question. I am not sure where you are getting the "issues" part. @Michael even apologized to the user once it was pointed out to him that Biostars was official support forum for CWL.

[Michael]

As I said, I am very sorry for not knowing or googling that before. I should have remembered that Biostars can be used for support of specific software. It is clear that some of the cwl related questions will stick out though, mostly because they are both very well written and specific and also very technical. Therefore, it might be good to have an updated list of software that is officially supported on Biostars as part of policy documentations.

[WouterDeCoster]

Yes, I made a similar mistake a few weeks ago.

Answer 2

[Michael R. Crusoe]

Here's an array of records:

inputs:
  rows: 
    type:
      type: array
      items:
        type: record
        fields:
          - name: entity_id
            type: string
          - name: label 
            type: string

[psaffrey]

This is great and now validates, but I can't seem to make the input work. My input looks like this:

rows: 
  - entity_id: "12345"
    label: "first"

And I get this error:

Traceback (most recent call last):
  File "build/bdist.macosx-10.11-intel/egg/cwltool/main.py", line 227, in single_job_executor
    for r in jobiter:
  File "build/bdist.macosx-10.11-intel/egg/cwltool/draft2tool.py", line 326, in job
    et = {u"entry": builder.do_eval(t["entry"])}
  File "build/bdist.macosx-10.11-intel/egg/cwltool/builder.py", line 206, in do_eval
    timeout=self.timeout)
  File "build/bdist.macosx-10.11-intel/egg/cwltool/expression.py", line 186, in do_eval
    jslib=jslib)
  File "build/bdist.macosx-10.11-intel/egg/cwltool/expression.py", line 144, in interpolate
    timeout=timeout)
  File "build/bdist.macosx-10.11-intel/egg/cwltool/expression.py", line 127, in evaluator
    return next_seg(m.group(0)[m.end(1) - m.start(0):-1], obj[m.group(1)])
  File "build/bdist.macosx-10.11-intel/egg/cwltool/expression.py", line 113, in next_seg
    return next_seg(remain[m.end(0):], obj[m.group(0)[1:]])
  File "build/bdist.macosx-10.11-intel/egg/cwltool/expression.py", line 113, in next_seg
    return next_seg(remain[m.end(0):], obj[m.group(0)[1:]])
TypeError: list indices must be integers, not str
Workflow error, try again with --debug for more information:
  list indices must be integers, not str

[psaffrey]

My bad, I had something wrong in a different part of my spec. Thanks to Peter Amstutz for folding better error messages into cwltool master - that really helped :)