Question: Copying to multiple files using CWL v1.0

Question: Question: Copying to multiple files using CWL v1.0

3.6 years ago by

I'm working on a CWL v1.0 workflow that copies a header file to a bunch of .sam files.

In bash, it would look like this:

cat chromosome_list.txt | parallel cp wg19header.txt sample_name.{}.sam

In CWL, I'm a little confused on how to copy files, and where inputs and outputs go. For example, my code is:

scatter-wf1.cwl

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
  chr_chrom: string[] #try array of files, see scatter-clt-copies-to-foo.txt.cwl
  wg_19_header: File
outputs: []
requirements:
  - class: ScatterFeatureRequirement

steps:
  step1:
    in:
      chroms: chr_chrom
      header: wg_19_header
    out: []
    scatter: chroms
    run:
      class: CommandLineTool
      inputs:
        chroms:
          type: string
          inputBinding:
            position: 1
        header:
          type: File
          inputBinding:
            position: 0
      outputs: []
      baseCommand: "cp"

wf.json

    "chr_chrom": ["chr1","chr10","chr11","chr12","chr13",
                      "chr14","chr15","chr16","chr17","chr18",
                      "chr19","chr2","chr20","chr21","chr22",
                      "chr3","chr4","chr5","chr6","chr7",
                      "chr8","chr9","chrM","chrX","chrY"],
    "wg_19_header": {
        "class":"File",
        "path":"WG_19_HEADER.txt"
    }

out:

cwl-runner scatter-wf1.cwl wf.json 
/usr/local/bin/cwl-runner 1.0.20160712154127
[job step1] /tmp/tmpYeX0Jh$ cp \
    /tmp/tmpxfocja/stgda17bec7-cc42-47e4-9b14-43eb5b0d172c/WG_19_HEADER.txt \
    chr1
[job step1_2] /tmp/tmpTbkbkg$ cp \
    /tmp/tmpZGzuut/stg8eef238b-e8de-4b82-b4c8-f55a24a108a8/WG_19_HEADER.txt \
    chr10
[job step1_3] /tmp/tmpqh0zae$ cp \
    /tmp/tmp4_ezcO/stgb18ca7d5-a4ce-4aac-9777-ca3f03543d20/WG_19_HEADER.txt \
    chr11
[job step1_4] /tmp/tmp8SuQry$ cp \
    /tmp/tmpcA4uid/stgdbf26c9c-4e8b-4bff-a0e4-fc0e78cd50a1/WG_19_HEADER.txt \
    chr12
[job step1_5] /tmp/tmpe_oKVD$ cp \
    /tmp/tmpdig2y9/stg34997f9c-d295-462b-a040-f2f1bcb8427f/WG_19_HEADER.txt \
    chr13
[job step1_6] /tmp/tmpxK8Rk4$ cp \
    /tmp/tmpHxwzD9/stg3fbef6a9-62c2-43fc-be00-4dd56b0670d9/WG_19_HEADER.txt \
    chr14
[job step1_7] /tmp/tmpSYoYvs$ cp \
    /tmp/tmpwhopwS/stgfaff69bb-c155-46e2-b26f-cb026b87a0d1/WG_19_HEADER.txt \
    chr15
[job step1_8] /tmp/tmprJgGoC$ cp \
    /tmp/tmpDB_H1r/stg943d25cd-03b4-45ec-855c-3982d2668718/WG_19_HEADER.txt \
    chr16
[job step1_9] /tmp/tmpSDyPBG$ cp \
    /tmp/tmpvQOzIv/stgb8add355-b76b-4e5d-8881-5abd3221df07/WG_19_HEADER.txt \
    chr17
[job step1_10] /tmp/tmpxVg9Hz$ cp \
    /tmp/tmpBdpL6G/stga17012be-d7e2-4598-a9fa-f2def5235588/WG_19_HEADER.txt \
    chr18
[job step1_11] /tmp/tmprs0Qvz$ cp \
    /tmp/tmpS7qC7L/stgf5c7df94-7c78-4f74-90a8-8c6dedc2996c/WG_19_HEADER.txt \
    chr19
[job step1_12] /tmp/tmpqXGm06$ cp \
    /tmp/tmpi0ZoMS/stg13a60c8e-728c-426b-9c98-46a2fbf1fef6/WG_19_HEADER.txt \
    chr2
[job step1_13] /tmp/tmpVTK4z4$ cp \
    /tmp/tmpJ_uOXH/stg6ebd6145-f6a1-4b5f-aa55-8106f10cf04a/WG_19_HEADER.txt \
    chr20
[job step1_14] /tmp/tmpDtwARH$ cp \
    /tmp/tmpAJTkhx/stg4ca575ae-d993-42f5-a47c-fe7e9db75254/WG_19_HEADER.txt \
    chr21
[job step1_15] /tmp/tmpAgE7mT$ cp \
    /tmp/tmpvc8lnE/stg55aa1a27-6ee0-416c-ba7d-58a48aa15f5b/WG_19_HEADER.txt \
    chr22
[job step1_16] /tmp/tmptGk1kY$ cp \
    /tmp/tmpBjH7fu/stgfa446779-b6f3-459c-9079-1bdaf2b2f653/WG_19_HEADER.txt \
    chr3
[job step1_17] /tmp/tmpZLm39N$ cp \
    /tmp/tmppMNd12/stgeebade44-ce62-4bd5-bd6a-61ea1540ac61/WG_19_HEADER.txt \
    chr4
[job step1_18] /tmp/tmpnGGHbj$ cp \
    /tmp/tmpnsP5sY/stg74fc6dd1-3ee7-4147-86f5-2e7a5dc9f9da/WG_19_HEADER.txt \
    chr5
[job step1_19] /tmp/tmpTIAxLg$ cp \
    /tmp/tmpu5qs1Z/stg16277549-13fe-496c-b2c4-e44dcbf198b0/WG_19_HEADER.txt \
    chr6
[job step1_20] /tmp/tmpzy1XqG$ cp \
    /tmp/tmpOG9F75/stg5e120e4b-0a17-4fe1-83a3-e408f4fbdd10/WG_19_HEADER.txt \
    chr7
[job step1_21] /tmp/tmpl6CqyI$ cp \
    /tmp/tmpeayL_J/stg085d50d8-c54d-4750-b852-634644fa1252/WG_19_HEADER.txt \
    chr8
[job step1_22] /tmp/tmpCSMeuS$ cp \
    /tmp/tmpNZnPR2/stg43d46de0-6cc6-4036-9035-6c9b62f677ca/WG_19_HEADER.txt \
    chr9
[job step1_23] /tmp/tmpNIxScl$ cp \
    /tmp/tmpWXDD2S/stg7b9fec54-8237-4af2-acd2-3d8dd564cf8b/WG_19_HEADER.txt \
    chrM
[job step1_24] /tmp/tmp3sGi5v$ cp \
    /tmp/tmpsZ_IVh/stg6dc8363f-1882-46b6-b949-51258321c600/WG_19_HEADER.txt \
    chrX
[job step1_25] /tmp/tmpwzZrGn$ cp \
    /tmp/tmpHwPv1C/stge54fac4d-d9c9-4bd7-9809-38fc6ad0e235/WG_19_HEADER.txt \
    chrY
[step step1] completion status is success
[workflow scatter-wf1.cwl] outdir is /tmp/tmp7eIMDc
Final process status is success
{}

What's the best way to accomplish this? Do I need to change the tmp directory to output the sam file? Do I have to make the json an array of files?

cwl common-workflow-language • 1.4k views

ADD COMMENT • link •

modified 3.6 years ago by Michael R. Crusoe • 1.8k • written 3.6 years ago by bryan.cosca • 0

3.6 years ago by

Michael R. Crusoe • 1.8k

Common Workflow Language project

Michael R. Crusoe • 1.8k wrote:

Hello Bryan,

You need to specify the outputs of each step (only one in your case) and their Processes (the embedded CommandLineTool) and the overall workflow.

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow
inputs:
  chr_chrom: string[]
  wg_19_header: File
outputs:
  named_files:
    type: File[]
    outputSource: step1/copied-file

requirements:
  - class: ScatterFeatureRequirement

steps:
  step1:
    in:
      chroms: chr_chrom
      header: wg_19_header
    out:
     - copied-file
    scatter: chroms
    run:
      class: CommandLineTool
      inputs:
        chroms:
          type: string
          inputBinding:
            position: 1
        header:
          type: File
          inputBinding:
            position: 0
      outputs:
        copied-file:
          type: File
          outputBinding:
            glob: $(inputs.chroms)
      baseCommand: "cp"

Your input document was missing the outer curly brackets; here it is in YAML form:

chr_chrom: 
    - chr1
    - chr10
    - chr11
    - chr12
    - chr13
    - chr14
    - chr15
    - chr16
    - chr17
    - chr18
    - chr19
    - chr2
    - chr20
    - chr21
    - chr22
    - chr3
    - chr4
    - chr5
    - chr6
    - chr7
    - chr8
    - chr9
    - chrM
    - chrX
    - chrY
wg_19_header: 
    class: File
    path: WG_19_HEADER.txt

ADD COMMENT • link written 3.6 years ago by Michael R. Crusoe • 1.8k

Similar posts • Search »