Question: Copying to multiple files using CWL v1.0
1
0
Entering edit mode
9.0 years ago

I'm working on a CWL v1.0 workflow that copies a header file to a bunch of .sam files.

In bash, it would look like this:

cat chromosome_list.txt | parallel cp wg19header.txt sample_name.{}.sam

In CWL, I'm a little confused on how to copy files, and where inputs and outputs go. For example, my code is:

scatter-wf1.cwl

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
  chr_chrom: string[] #try array of files, see scatter-clt-copies-to-foo.txt.cwl
  wg_19_header: File
outputs: []
requirements:
  - class: ScatterFeatureRequirement

steps:
  step1:
    in:
      chroms: chr_chrom
      header: wg_19_header
    out: []
    scatter: chroms
    run:
      class: CommandLineTool
      inputs:
        chroms:
          type: string
          inputBinding:
            position: 1
        header:
          type: File
          inputBinding:
            position: 0
      outputs: []
      baseCommand: "cp"

wf.json

    "chr_chrom": ["chr1","chr10","chr11","chr12","chr13",
                      "chr14","chr15","chr16","chr17","chr18",
                      "chr19","chr2","chr20","chr21","chr22",
                      "chr3","chr4","chr5","chr6","chr7",
                      "chr8","chr9","chrM","chrX","chrY"],
    "wg_19_header": {
        "class":"File",
        "path":"WG_19_HEADER.txt"
    }

out:

cwl-runner scatter-wf1.cwl wf.json 
/usr/local/bin/cwl-runner 1.0.20160712154127
[job step1] /tmp/tmpYeX0Jh$ cp \
    /tmp/tmpxfocja/stgda17bec7-cc42-47e4-9b14-43eb5b0d172c/WG_19_HEADER.txt \
    chr1
[job step1_2] /tmp/tmpTbkbkg$ cp \
    /tmp/tmpZGzuut/stg8eef238b-e8de-4b82-b4c8-f55a24a108a8/WG_19_HEADER.txt \
    chr10
[job step1_3] /tmp/tmpqh0zae$ cp \
    /tmp/tmp4_ezcO/stgb18ca7d5-a4ce-4aac-9777-ca3f03543d20/WG_19_HEADER.txt \
    chr11
[job step1_4] /tmp/tmp8SuQry$ cp \
    /tmp/tmpcA4uid/stgdbf26c9c-4e8b-4bff-a0e4-fc0e78cd50a1/WG_19_HEADER.txt \
    chr12
[job step1_5] /tmp/tmpe_oKVD$ cp \
    /tmp/tmpdig2y9/stg34997f9c-d295-462b-a040-f2f1bcb8427f/WG_19_HEADER.txt \
    chr13
[job step1_6] /tmp/tmpxK8Rk4$ cp \
    /tmp/tmpHxwzD9/stg3fbef6a9-62c2-43fc-be00-4dd56b0670d9/WG_19_HEADER.txt \
    chr14
[job step1_7] /tmp/tmpSYoYvs$ cp \
    /tmp/tmpwhopwS/stgfaff69bb-c155-46e2-b26f-cb026b87a0d1/WG_19_HEADER.txt \
    chr15
[job step1_8] /tmp/tmprJgGoC$ cp \
    /tmp/tmpDB_H1r/stg943d25cd-03b4-45ec-855c-3982d2668718/WG_19_HEADER.txt \
    chr16
[job step1_9] /tmp/tmpSDyPBG$ cp \
    /tmp/tmpvQOzIv/stgb8add355-b76b-4e5d-8881-5abd3221df07/WG_19_HEADER.txt \
    chr17
[job step1_10] /tmp/tmpxVg9Hz$ cp \
    /tmp/tmpBdpL6G/stga17012be-d7e2-4598-a9fa-f2def5235588/WG_19_HEADER.txt \
    chr18
[job step1_11] /tmp/tmprs0Qvz$ cp \
    /tmp/tmpS7qC7L/stgf5c7df94-7c78-4f74-90a8-8c6dedc2996c/WG_19_HEADER.txt \
    chr19
[job step1_12] /tmp/tmpqXGm06$ cp \
    /tmp/tmpi0ZoMS/stg13a60c8e-728c-426b-9c98-46a2fbf1fef6/WG_19_HEADER.txt \
    chr2
[job step1_13] /tmp/tmpVTK4z4$ cp \
    /tmp/tmpJ_uOXH/stg6ebd6145-f6a1-4b5f-aa55-8106f10cf04a/WG_19_HEADER.txt \
    chr20
[job step1_14] /tmp/tmpDtwARH$ cp \
    /tmp/tmpAJTkhx/stg4ca575ae-d993-42f5-a47c-fe7e9db75254/WG_19_HEADER.txt \
    chr21
[job step1_15] /tmp/tmpAgE7mT$ cp \
    /tmp/tmpvc8lnE/stg55aa1a27-6ee0-416c-ba7d-58a48aa15f5b/WG_19_HEADER.txt \
    chr22
[job step1_16] /tmp/tmptGk1kY$ cp \
    /tmp/tmpBjH7fu/stgfa446779-b6f3-459c-9079-1bdaf2b2f653/WG_19_HEADER.txt \
    chr3
[job step1_17] /tmp/tmpZLm39N$ cp \
    /tmp/tmppMNd12/stgeebade44-ce62-4bd5-bd6a-61ea1540ac61/WG_19_HEADER.txt \
    chr4
[job step1_18] /tmp/tmpnGGHbj$ cp \
    /tmp/tmpnsP5sY/stg74fc6dd1-3ee7-4147-86f5-2e7a5dc9f9da/WG_19_HEADER.txt \
    chr5
[job step1_19] /tmp/tmpTIAxLg$ cp \
    /tmp/tmpu5qs1Z/stg16277549-13fe-496c-b2c4-e44dcbf198b0/WG_19_HEADER.txt \
    chr6
[job step1_20] /tmp/tmpzy1XqG$ cp \
    /tmp/tmpOG9F75/stg5e120e4b-0a17-4fe1-83a3-e408f4fbdd10/WG_19_HEADER.txt \
    chr7
[job step1_21] /tmp/tmpl6CqyI$ cp \
    /tmp/tmpeayL_J/stg085d50d8-c54d-4750-b852-634644fa1252/WG_19_HEADER.txt \
    chr8
[job step1_22] /tmp/tmpCSMeuS$ cp \
    /tmp/tmpNZnPR2/stg43d46de0-6cc6-4036-9035-6c9b62f677ca/WG_19_HEADER.txt \
    chr9
[job step1_23] /tmp/tmpNIxScl$ cp \
    /tmp/tmpWXDD2S/stg7b9fec54-8237-4af2-acd2-3d8dd564cf8b/WG_19_HEADER.txt \
    chrM
[job step1_24] /tmp/tmp3sGi5v$ cp \
    /tmp/tmpsZ_IVh/stg6dc8363f-1882-46b6-b949-51258321c600/WG_19_HEADER.txt \
    chrX
[job step1_25] /tmp/tmpwzZrGn$ cp \
    /tmp/tmpHwPv1C/stge54fac4d-d9c9-4bd7-9809-38fc6ad0e235/WG_19_HEADER.txt \
    chrY
[step step1] completion status is success
[workflow scatter-wf1.cwl] outdir is /tmp/tmp7eIMDc
Final process status is success
{}

What's the best way to accomplish this? Do I need to change the tmp directory to output the sam file? Do I have to make the json an array of files?

CWL common-workflow-language • 2.5k views
ADD COMMENT
0
Entering edit mode
9.0 years ago

Hello Bryan,

You need to specify the outputs of each step (only one in your case) and their Processes (the embedded CommandLineTool) and the overall workflow.

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
  chr_chrom: string[]
  wg_19_header: File
outputs:
  named_files:
    type: File[]
    outputSource: step1/copied-file

requirements:
  - class: ScatterFeatureRequirement

steps:
  step1:
    in:
      chroms: chr_chrom
      header: wg_19_header
    out:
     - copied-file
    scatter: chroms
    run:
      class: CommandLineTool
      inputs:
        chroms:
          type: string
          inputBinding:
            position: 1
        header:
          type: File
          inputBinding:
            position: 0
      outputs:
        copied-file:
          type: File
          outputBinding:
            glob: $(inputs.chroms)
      baseCommand: "cp"

Your input document was missing the outer curly brackets; here it is in YAML form:

chr_chrom: 
    - chr1
    - chr10
    - chr11
    - chr12
    - chr13
    - chr14
    - chr15
    - chr16
    - chr17
    - chr18
    - chr19
    - chr2
    - chr20
    - chr21
    - chr22
    - chr3
    - chr4
    - chr5
    - chr6
    - chr7
    - chr8
    - chr9
    - chrM
    - chrX
    - chrY
wg_19_header: 
    class: File
    path: WG_19_HEADER.txt
ADD COMMENT

Login before adding your answer.

Traffic: 2709 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6