Question: CWL to write into for writing lines one from one file to new file
0
gravatar for a.james
8 months ago by
a.james210
Germany
a.james210 wrote:

Dear All,

I have small Shell script to write the lines in a file to another file

writer.sh
#!/usr/bin/env bash
infile=$1
outfile=$2
while read line
do
        echo $line >> "$outfile"
done < "$infile"

I converted it for CWL as following, writer.cwl

cwlVersion: v1.0
class: CommandLineTool
baseCommand: [sh,writer.sh]


inputs:

 infile:
  type: File
  inputBinding:
   position: 1

 outfile:
  type: string
  inputBinding:
   position: 2

outputs:
 out_message:
  type: File
  outputBinding:
   glob: $(inputs.outfile)

And the YML writer.yml file looks as following,

infile:
 class: File
 path:  /user/home/hello.txt
outfile: monday.txt
   class: File
   path:  /user/home/tuesday.txt

When I run as following, cwltool writer.cwl writer.yml. it is throwing permanent fail error as,

[job writer.cwl] /tmp/tmpmht4svls$ sh \
    search.sh \
    /tmp/tmpv3evb_ic/stg1a74b4ca-15ba-4d28-9598-62b7604e8185/hello.txt \
    monday.txt
sh: writer.sh: No such file or directory
[job writer.cwl] Job error:
Error collecting output for parameter 'out_message':
writer.cw.cwl:18:2: Traceback (most recent call last):
writer.cw.cwl:18:2: 
writer.cw:18:2:   File "/cluster/home/aalva/.local/venvs/cwltool/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 724, in collect_output
writer.cwl: 18:2:     raise WorkflowException("Did not find output file with glob pattern: '{}'".format(globpatterns))
writer.cwl:18:2: 
writer.cwl:18:2: cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['monday.txt']'
writer.cwl:18:2: 



  writer.cwl:21:4: cwltool.errors.WorkflowException: Did not find output file with glob pattern: '['monday.txt']'
    [job writer.cwl] completed permanentFail
    [job writer.cwll] {}
    [job writer.cwll] Removing input staging directory /tmp/tmpv3evb_ic
    [job writer.cwl] Removing temporary directory /tmp/tmp8v9yxvvo
    {}
    Final process status is permanentFail

I am not understanding what is wrong here.Any suggestions would be appreciated

cwl • 353 views
ADD COMMENTlink modified 8 months ago • written 8 months ago by a.james210
2
gravatar for Tom
8 months ago by
Tom510
Bielefeld University, CeBiTec, Germany
Tom510 wrote:

Hi fellow pipeline enthusiast,

Is the script named "writer.sh" or "search.sh"? In the error you posted, the shell seems to throw the error search.sh: No such file or directory. I guess it can't find the script, does nothing except throw the error message and CWL complains because no output file is being generated.

Is the script stored in a location that is referenced in your $PATH? A dirty but quick fix would be to just pass your script as an argument in inputs.

Regards, Tom

ADD COMMENTlink written 8 months ago by Tom510

Hey, Sorry, it is an error , It should be writer.sh. Is the script stored in a location that is referenced in your $PATH? Yes, it is store the place where it is being invoked. A dirty but quick fix would be to just pass your script as an argument in inputs. >> I did not quite understand what you mean?

You mean without outputs ? or by passing something like

   arguments:
         valueFrom: $(runtime.outdir)/$(inputs.outfile) ?
ADD REPLYlink modified 8 months ago • written 8 months ago by a.james210

The place were you invoke is most likely not referenced in the $PATH environment variable. This means if you call sh writer.sh then writer.sh has to be in the directory you are calling sh from. However, CWL creates a temporary directory (in your example it was /tmp/tmpmht4svls ) where it copies all your input data (or symlinks to the data) and then tries to run what you have specified in baseCommand. In this temporary directory, writer.sh is not present, so sh can't find it. If you pass the script as one of your input files, then CWL will know to copy it into the temporary directory during runtime and sh will find it.

.cwl-description:

cwlVersion: v1.0
class: CommandLineTool
baseCommand: sh

inputs:
  script:
    type: File
    inputBinding:
      position: 0    
  infile:
    type: File
    inputBinding:
     position: 1
  outfile:
    type: string
    inputBinding:
     position: 2
outputs:
  out_message:
    type: File
    outputBinding:
      glob: $(inputs.outfile)

.yml-file:

script:
  class: File
  path: /WHEREVER/THE/SCRIPT/LIVES/writer.sh
infile:
  class: File
  path:  /user/home/hello.txt
outfile: monday.txt
  class: File
  path:  /user/home/tuesday.txt
ADD REPLYlink written 8 months ago by Tom510
1

Thanks Tom, this is working. I tried it to on scatter workflow with array of file and string as input. It got finished successfully

ADD REPLYlink written 8 months ago by a.james210
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1599 users visited in the last hour