Question: Why is ShellCommandRequirement necessary for Pipes
0
gravatar for ionox0
15 months ago by
ionox0360
ionox0360 wrote:

This is a general question regarding the ShellCommandRequirement feature.

Would anyone happen to be able to explain why ShellCommandRequirement is required to use Pipes in the CWL arguments section?

The pipes will work if I use ShellCommandRequirement and set ShellQuote to false for the pipes.

However, the following CWL syntax looks like it should run fine based on the command line that is generated in the cwltool logging, but for some reason it seems the pipe is not working.

Arguments section:

arguments:
- /opt/common/CentOS_6-dev/vardict/v1.5.1/bin/VarDict
- -E
- $(inputs.column_for_region_end)
- -G
- $(inputs.reference_fasta)
- -N
- $(inputs.tumor_sample_name)
- -c
- $(inputs.column_for_chromosome)
- -b
- $(inputs.tumor_bam.path + '|' + inputs.normal_bam.path)
- -g
- $(inputs.column_for_gene_name)
- -f
- $(inputs.allele_freq_thres)
- -S
- $(inputs.column_for_region_start)
- -r
- $(inputs.min_num_variant_reads)
- $(inputs.bed_file)
- '|'
- /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/testsomatic.R
- '|'
- /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/var2vcf_paired.pl
- -N
- $(inputs.tumor_sample_name + '|' + inputs.normal_sample_name)
- -f
- $(inputs.allele_freq_thres)

Resulting command from cwltool logging:

[job vardict] /scratch/tmpbKz3yH$ /opt/common/CentOS_6-dev/vardict/v1.5.1/bin/VarDict \
    -E \
    3 \
    -G \
    /scratch/tmpcR4isu/stg9c4f932e-84fb-4108-99a2-627fd7aceec5/Homo_sapiens_assembly19.fasta \
    -N \
    C-4PU567-L002-d \
    -c \
    1 \
    -b \
    '/scratch/tmpcR4isu/stgeb5ea926-d425-468c-871a-1d593ae22d49/C-4PU567-L002-d_cl_aln_srt_MD_IR_FX_BR__aln_srt_IR_FX-duplex.bam|/scratch/tmpcR4isu/stgb8386cf2-387e-41b4-b999-3d9c9d60c9c1/C-4PU567-N002-d_cl_aln_srt_MD_IR_FX_BR__aln_srt_IR_FX-duplex.bam' \
    -g \
    4 \
    -f \
    0.0002 \
    -S \
    2 \
    -r \
    1 \
    /scratch/tmpcR4isu/stg1e7697e8-7bd3-4e8e-a5fc-d5dea2b240f7/MSK-ACCESS-v1_0-probe-A.sorted.no_fp_snp.bed \
    | \
    /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/testsomatic.R \
    | \
    /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/var2vcf_paired.pl \
    -N \
    'C-4PU567-L002-d|C-4PU567-N002-d' \
    -f \
    0.0002 > /scratch/tmpbKz3yH/C-4PU567-L002-d.C-4PU567-N002-d.vardict.vcf
cwl • 527 views
ADD COMMENTlink modified 15 months ago • written 15 months ago by ionox0360
2
gravatar for ionox0
15 months ago by
ionox0360
ionox0360 wrote:

Answered my own question after a bit of research:

ShellCommandRequirement in CWL allows us to use /bin/sh -c in order to pass our command as a string to a shell intrepreter. ShellQuote:false leaves shell metacharacters unquoted, and thus they can still be interpreted and used by a shell.

From CommandLineTool.job():

    shellcmd, _ = self.get_requirement("ShellCommandRequirement")
    if shellcmd is not None:
        cmd = []  # type: List[Text]
        for b in builder.bindings:
            arg = builder.generate_arg(b)
            if b.get("shellQuote", True):
                arg = [shellescape.quote(a) for a in aslist(arg)]
            cmd.extend(aslist(arg))
        j.command_line = ["/bin/sh", "-c", " ".join(cmd)]

Without the ShellCommandRequirement the job will run with subprocess.Popen(shell=False) and thus no shell metacharacters can be used.

My misunderstanding was I thought that all CommandLineTools were run using a shell and could use metacharacters.

ADD COMMENTlink modified 15 months ago • written 15 months ago by ionox0360
1

I just wanted to drop by to say i love that you always take the time to come back and explain, even after solving the issue for yourself.

ADD REPLYlink written 15 months ago by Tom520

Thanks Tom! I still don't understand whether additiona arguments need to be quoted after the pipe when using /bin/sh -c, or whether that doesn't matter.

ADD REPLYlink written 15 months ago by ionox0360

I have no idea either! I'm not great at cwl yet. Just facing my own cwl-reated issues almost daily atm, and your threads often come up when i'm looking for solutions. Thank you for providing so many answers!

ADD REPLYlink written 15 months ago by Tom520
1

Glad it's helpful! It is also helpful for myself to be able to explain it. Another thing I realized was that even though the command is printed like that by cwltool when it runs the command, cwltool is a python process, and it's not the same as running the command from a shell / terminal outside CWL

ADD REPLYlink written 15 months ago by ionox0360
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 803 users visited in the last hour