Question: secondaryFiles where nameroot differs
gravatar for Peter vH
16 months ago by
Peter vH130
Peter vH130 wrote:

I am trying to write a wrapper for kraken2 and struggling to express the database as a File with secondaryFiles. The structure of the DB is a directory with 3 files: hash.k2d, opts.k2d and taxo.k2d. Because of this structure the typical secondaryFiles format that assumes at least a shared nameroot between the files does not work. However, the specification states that secondaryFiles can be an expression that "must return a filename string relative to the path to the primary File, a File or Directory object with either path or location and basename fields set, or an array consisting of strings or File or Directory objects.".

I thus tried:

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
id: kraken2
  - kraken2
      - Directory
      - File
    label: "Kraken 2 DB"
      position: 0
      prefix: --db
      valueFrom: $(self.dirname)
    secondaryFiles: |
        let dirname = self.location.split('/').slice(0,-1).join('/');
        return [
          { class: "File", location: dirname + '/opts.k2d' },
          { class: "File", location: dirname + '/taxo.k2d' }

but this yields the following cwltool error (cwltool version 1.0.20190831161204):

 cwltool kraken2.cwl  input1.yml 
INFO /home/pvh/.virtualenvs/cwltool/bin/cwltool 1.0.20190831161204
INFO Resolved 'kraken2.cwl' to 'file:///home/pvh/Documents/code/SANBI/pvh-forks/bio-cwl-tools/kraken2/kraken2.cwl'
ERROR Got workflow error
Traceback (most recent call last):
  File "/home/pvh/.virtualenvs/cwltool/lib/python3.7/site-packages/cwltool/", line 168, in run_jobs
    for job in jobiter:
  File "/home/pvh/.virtualenvs/cwltool/lib/python3.7/site-packages/cwltool/", line 430, in job
    builder = self._init_job(job_order, runtimeContext)
  File "/home/pvh/.virtualenvs/cwltool/lib/python3.7/site-packages/cwltool/", line 718, in _init_job
    discover_secondaryFiles=getdefault(runtime_context.toplevel, False)))
  File "/home/pvh/.virtualenvs/cwltool/lib/python3.7/site-packages/cwltool/", line 276, in bind_input
    bindings.extend(self.bind_input(f, datum[f["name"]], lead_pos=lead_pos, tail_pos=f["name"], discover_secondaryFiles=discover_secondaryFiles))
  File "/home/pvh/.virtualenvs/cwltool/lib/python3.7/site-packages/cwltool/", line 251, in bind_input
    self.bind_input(schema, datum, lead_pos=lead_pos, tail_pos=tail_pos, discover_secondaryFiles=discover_secondaryFiles)
  File "/home/pvh/.virtualenvs/cwltool/lib/python3.7/site-packages/cwltool/", line 332, in bind_input
    sf_location = datum["location"][0:datum["location"].rindex("/")+1]+sfname
TypeError: can only concatenate str (not "dict") to str
ERROR Workflow error, try again with --debug for more information:
can only concatenate str (not "dict") to str

So I presume the CWL I have is incorrect. Is there a way to specify this structure of files?

And even if this can be done with some form of expression, the specification warns:

"To work on non-filename-preserving storage systems, portable tool descriptions should avoid constructing new values from location, but should construct relative references using basename or nameroot instead."

Yet since dirname is seemingly not available to the expression, I am forced to rely on location - I do not know what a non-filename-preserving storage system (e.g. S3?) would look like from this perspective.

cwl • 460 views
ADD COMMENTlink modified 16 months ago by Michael R. Crusoe1.8k • written 16 months ago by Peter vH130
gravatar for Michael R. Crusoe
16 months ago by
Common Workflow Language project
Michael R. Crusoe1.8k wrote:

Hello Peter vH,

I think there are two separate challenges here:

1) describing a File with secondaryFiles that don't share anything in common name-wise


2) staging a File and its secondaries inside a named directory

For (1), on can provide the plain name for each of the desired secondaryFiles using a CWL Expression. (If we put the plain name without using a CWL expression then it is interpreted as extensions to the original filename).

For (2), one can use an CWL Expression to provide the entire listing.

Here's my solution for both:

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
baseCommand: kraken2

  InlineJavascriptRequirement: {}
      - |
        $( {"class": "Directory",
            "basename": "kraken_db",
            "listing": [ inputs.database ]

    type: File
    label: "Kraken 2 DB"
      - $("opts.k2d")
      - $("taxo.k2d")

  - prefix: --db
    valueFrom: "kraken_db"

outputs: []
ADD COMMENTlink written 16 months ago by Michael R. Crusoe1.8k

Thanks for the clarification about the use of expressions in contrast to bare strings with seoncaryFiles. I ended up with:

      - Directory
      - File
    label: "Kraken 2 DB"
    doc: "(either a File refer to the hash.k2d file in the DB or a Directory to reference the entire directory)"
      position: 0
      prefix: --db
      valueFrom: |
        ${ return (self.class == "File") ? self.dirname : self.path }
      - $("opts.k2d")
      - $("taxo.k2d")

which allows the database parameter to be either a File or a Directory. In the case of the File input, it looks like:

    class: File
    path: db/hash.k2d

and in the case of the Directory it is:

    class: Directory
    path: db
ADD REPLYlink modified 16 months ago • written 16 months ago by Peter vH130
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 2614 users visited in the last hour