Question: Why is ShellCommandRequirement necessary for Pipes
gravatar for ionox0
5 weeks ago by
ionox0330 wrote:

This is a general question regarding the ShellCommandRequirement feature.

Would anyone happen to be able to explain why ShellCommandRequirement is required to use Pipes in the CWL arguments section?

The pipes will work if I use ShellCommandRequirement and set ShellQuote to false for the pipes.

However, the following CWL syntax looks like it should run fine based on the command line that is generated in the cwltool logging, but for some reason it seems the pipe is not working.

Arguments section:

- /opt/common/CentOS_6-dev/vardict/v1.5.1/bin/VarDict
- -E
- $(inputs.column_for_region_end)
- -G
- $(inputs.reference_fasta)
- -N
- $(inputs.tumor_sample_name)
- -c
- $(inputs.column_for_chromosome)
- -b
- $(inputs.tumor_bam.path + '|' + inputs.normal_bam.path)
- -g
- $(inputs.column_for_gene_name)
- -f
- $(inputs.allele_freq_thres)
- -S
- $(inputs.column_for_region_start)
- -r
- $(inputs.min_num_variant_reads)
- $(inputs.bed_file)
- '|'
- /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/testsomatic.R
- '|'
- /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/
- -N
- $(inputs.tumor_sample_name + '|' + inputs.normal_sample_name)
- -f
- $(inputs.allele_freq_thres)

Resulting command from cwltool logging:

[job vardict] /scratch/tmpbKz3yH$ /opt/common/CentOS_6-dev/vardict/v1.5.1/bin/VarDict \
    -E \
    3 \
    -G \
    /scratch/tmpcR4isu/stg9c4f932e-84fb-4108-99a2-627fd7aceec5/Homo_sapiens_assembly19.fasta \
    -N \
    C-4PU567-L002-d \
    -c \
    1 \
    -b \
    '/scratch/tmpcR4isu/stgeb5ea926-d425-468c-871a-1d593ae22d49/C-4PU567-L002-d_cl_aln_srt_MD_IR_FX_BR__aln_srt_IR_FX-duplex.bam|/scratch/tmpcR4isu/stgb8386cf2-387e-41b4-b999-3d9c9d60c9c1/C-4PU567-N002-d_cl_aln_srt_MD_IR_FX_BR__aln_srt_IR_FX-duplex.bam' \
    -g \
    4 \
    -f \
    0.0002 \
    -S \
    2 \
    -r \
    1 \
    /scratch/tmpcR4isu/stg1e7697e8-7bd3-4e8e-a5fc-d5dea2b240f7/MSK-ACCESS-v1_0-probe-A.sorted.no_fp_snp.bed \
    | \
    /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/testsomatic.R \
    | \
    /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/ \
    -N \
    'C-4PU567-L002-d|C-4PU567-N002-d' \
    -f \
    0.0002 > /scratch/tmpbKz3yH/C-4PU567-L002-d.C-4PU567-N002-d.vardict.vcf
cwl • 161 views
ADD COMMENTlink modified 5 weeks ago • written 5 weeks ago by ionox0330
gravatar for ionox0
5 weeks ago by
ionox0330 wrote:

Answered my own question after a bit of research:

ShellCommandRequirement in CWL allows us to use /bin/sh -c in order to pass our command as a string to a shell intrepreter. ShellQuote:false leaves shell metacharacters unquoted, and thus they can still be interpreted and used by a shell.

From CommandLineTool.job():

    shellcmd, _ = self.get_requirement("ShellCommandRequirement")
    if shellcmd is not None:
        cmd = []  # type: List[Text]
        for b in builder.bindings:
            arg = builder.generate_arg(b)
            if b.get("shellQuote", True):
                arg = [shellescape.quote(a) for a in aslist(arg)]
        j.command_line = ["/bin/sh", "-c", " ".join(cmd)]

Without the ShellCommandRequirement the job will run with subprocess.Popen(shell=False) and thus no shell metacharacters can be used.

My misunderstanding was I thought that all CommandLineTools were run using a shell and could use metacharacters.

ADD COMMENTlink modified 5 weeks ago • written 5 weeks ago by ionox0330

I just wanted to drop by to say i love that you always take the time to come back and explain, even after solving the issue for yourself.

ADD REPLYlink written 5 weeks ago by Tom200

Thanks Tom! I still don't understand whether additiona arguments need to be quoted after the pipe when using /bin/sh -c, or whether that doesn't matter.

ADD REPLYlink written 5 weeks ago by ionox0330

I have no idea either! I'm not great at cwl yet. Just facing my own cwl-reated issues almost daily atm, and your threads often come up when i'm looking for solutions. Thank you for providing so many answers!

ADD REPLYlink written 5 weeks ago by Tom200

Glad it's helpful! It is also helpful for myself to be able to explain it. Another thing I realized was that even though the command is printed like that by cwltool when it runs the command, cwltool is a python process, and it's not the same as running the command from a shell / terminal outside CWL

ADD REPLYlink written 4 weeks ago by ionox0330
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 2234 users visited in the last hour