Question: Converting stdout (File) to string
0
gravatar for eduardo
2.7 years ago by
eduardo10
eduardo10 wrote:

I am constructing a workflow that contains two steps:

  1. the output of the first step is stdout (compatible, I believe, with File type)
  2. the input of the second is a string

I am not able to modify these steps. How can I connect these two steps? Do I have to make an adaptor step?

cwl • 1.6k views
ADD COMMENTlink modified 2.7 years ago by Michael R. Crusoe1.8k • written 2.7 years ago by eduardo10

Please post your example code and expected output

ADD REPLYlink written 2.7 years ago by cpad011213k

Output from first step:

outputs:
    accessions:
        type: stdout

Input from second step:

inputs:
    sequences:
        type: string
ADD REPLYlink modified 2.6 years ago • written 2.7 years ago by eduardo10

Besides the tag it's unclear that this is about cwl - would be good to mention that. The question doesn't make sense if you don't read the tags.

ADD REPLYlink written 2.7 years ago by WouterDeCoster44k
2
gravatar for Michael R. Crusoe
2.7 years ago by
Common Workflow Language project
Michael R. Crusoe1.8k wrote:

Here's the ExpressionTool version:

$ cat u.cwl 

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: ExpressionTool

requirements: { InlineJavascriptRequirement: {} }

inputs:
  a_File:
    type: File
    inputBinding:
      loadContents: true

expression: |
  ${ return { "a_string": inputs.a_File.contents }; }

outputs:
  a_string: string

And it's output

$ echo "Hello, World!" > message
$ cwltool u.cwl --a_File message 
/home/michael/cwltool/env/bin/cwltool 1.0.20171107133715
Resolved 'u.cwl' to 'file:///home/michael/u.cwl'
{
    "a_string": "Hello, World!\n"
}
Final process status is success
ADD COMMENTlink written 2.7 years ago by Michael R. Crusoe1.8k

This solution also has the 64kb limitation, correct?

ADD REPLYlink written 2.7 years ago by eduardo10

Yes, thank you for noticing that. The only way to get around the 64kb limit is with a dedicated CommandLineTool.

ADD REPLYlink written 2.7 years ago by Michael R. Crusoe1.8k

Which I've added an example at A: Converting stdout (File) to string

ADD REPLYlink written 2.7 years ago by Michael R. Crusoe1.8k
1
gravatar for Michael R. Crusoe
2.7 years ago by
Common Workflow Language project
Michael R. Crusoe1.8k wrote:

And a third way, that converts any size of File that will fit into system memory into a CWL string. This is achieved using the JSON feature from http://www.commonwl.org/v1.0/CommandLineTool.html#Output_binding

$ echo 'Hello,
World!' > message

Our input file

$ cat v.cwl 
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool

inputs:
  a_File: File

baseCommand: python
arguments:
 - prefix: -c
   valueFrom: |
     import json
     a_string = ""
     with open("$(inputs.a_File.path)", "r") as a_File:
         a_string = a_File.read()
     with open("cwl.output.json", "w") as output:
         json.dump({"a_string": a_string}, output)

outputs:
  a_string: string

The CWL CommandLineTool itself

$ cwltool ../v.cwl --a_File message
/home/michael/common-workflow-language/env/bin/cwltool 1.0.20171107133715
Resolved 'v.cwl' to 'file:///home/michael/v.cwl'
[job v.cwl] /tmp/tmp4YLbgd$ python \
    -c \
    'import json
import os
a_string = ""
with open("/tmp/tmp3xHM90/stgdc883f3a-b4e7-4eab-aa42-3cddb08a822b/message", "r") as a_File:
    a_string = a_File.read()
with open("cwl.output.json", "w") as output:
    json.dump({"a_string": a_string}, output)'
[job v.cwl] completed success
{
    "a_string": "Hello,\nWorld!\n"
}
Final process status is success

And the CWL File to CWL string converter in action. As you can see, this reads the entire file into memory.

ADD COMMENTlink written 2.7 years ago by Michael R. Crusoe1.8k
1
gravatar for Michael R. Crusoe
2.7 years ago by
Common Workflow Language project
Michael R. Crusoe1.8k wrote:

Hello eduardo,

If the stdout is less than 64kb then you can express it as a string either during the gathering of the outputs.

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool

inputs:
  message: string

baseCommand: echo

arguments: [ $(inputs.message) ]

stdout: message

outputs:
  message_File:  # Just to illustrate that you can process a file multiple times, not necessary for message_string to work
    type: File
    outputBinding:
      glob: message

  message_string:
    type: string
    outputBinding:
      glob: message
      loadContents: true
      outputEval: $(self[0].contents)

And the output:

$ cwltool ../t.cwl --message "Hello, world!"
/home/michael/cwltool/env/bin/cwltool 1.0.20171107133715
Resolved '../t.cwl' to 'file:///home/michael/t.cwl'
[job t.cwl] /tmp/tmpAljfTQ$ echo \
    'Hello, world!' > /tmp/tmpAljfTQ/message
[job t.cwl] completed success
{
    "message_string": "Hello, world!\n", 
    "message_File": {
        "checksum": "sha1$09fac8dbfd27bd9b4d23a00eb648aa751789536d", 
        "basename": "message", 
        "location": "file:///home/michael/u/message", 
        "path": "/home/michael/u/message", 
        "class": "File", 
        "size": 14
    }
}
Final process status is success

If you don't want to change your tool definition, then you can use an ExpressionTool to convert the File to a string.

ADD COMMENTlink modified 2.7 years ago • written 2.7 years ago by Michael R. Crusoe1.8k

Thanks! That's what I was looking for.

ADD REPLYlink written 2.7 years ago by eduardo10
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1803 users visited in the last hour