Question: Why is ShellCommandRequirement necessary for Pipes
0
gravatar for ionox0
5 months ago by
ionox0370
ionox0370 wrote:

This is a general question regarding the ShellCommandRequirement feature.

Would anyone happen to be able to explain why ShellCommandRequirement is required to use Pipes in the CWL arguments section?

The pipes will work if I use ShellCommandRequirement and set ShellQuote to false for the pipes.

However, the following CWL syntax looks like it should run fine based on the command line that is generated in the cwltool logging, but for some reason it seems the pipe is not working.

Arguments section:

arguments:
- /opt/common/CentOS_6-dev/vardict/v1.5.1/bin/VarDict
- -E
- $(inputs.column_for_region_end)
- -G
- $(inputs.reference_fasta)
- -N
- $(inputs.tumor_sample_name)
- -c
- $(inputs.column_for_chromosome)
- -b
- $(inputs.tumor_bam.path + '|' + inputs.normal_bam.path)
- -g
- $(inputs.column_for_gene_name)
- -f
- $(inputs.allele_freq_thres)
- -S
- $(inputs.column_for_region_start)
- -r
- $(inputs.min_num_variant_reads)
- $(inputs.bed_file)
- '|'
- /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/testsomatic.R
- '|'
- /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/var2vcf_paired.pl
- -N
- $(inputs.tumor_sample_name + '|' + inputs.normal_sample_name)
- -f
- $(inputs.allele_freq_thres)

Resulting command from cwltool logging:

[job vardict] /scratch/tmpbKz3yH$ /opt/common/CentOS_6-dev/vardict/v1.5.1/bin/VarDict \
    -E \
    3 \
    -G \
    /scratch/tmpcR4isu/stg9c4f932e-84fb-4108-99a2-627fd7aceec5/Homo_sapiens_assembly19.fasta \
    -N \
    C-4PU567-L002-d \
    -c \
    1 \
    -b \
    '/scratch/tmpcR4isu/stgeb5ea926-d425-468c-871a-1d593ae22d49/C-4PU567-L002-d_cl_aln_srt_MD_IR_FX_BR__aln_srt_IR_FX-duplex.bam|/scratch/tmpcR4isu/stgb8386cf2-387e-41b4-b999-3d9c9d60c9c1/C-4PU567-N002-d_cl_aln_srt_MD_IR_FX_BR__aln_srt_IR_FX-duplex.bam' \
    -g \
    4 \
    -f \
    0.0002 \
    -S \
    2 \
    -r \
    1 \
    /scratch/tmpcR4isu/stg1e7697e8-7bd3-4e8e-a5fc-d5dea2b240f7/MSK-ACCESS-v1_0-probe-A.sorted.no_fp_snp.bed \
    | \
    /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/testsomatic.R \
    | \
    /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/var2vcf_paired.pl \
    -N \
    'C-4PU567-L002-d|C-4PU567-N002-d' \
    -f \
    0.0002 > /scratch/tmpbKz3yH/C-4PU567-L002-d.C-4PU567-N002-d.vardict.vcf
cwl • 270 views
ADD COMMENTlink modified 5 months ago • written 5 months ago by ionox0370
2
gravatar for ionox0
5 months ago by
ionox0370
ionox0370 wrote:

Answered my own question after a bit of research:

ShellCommandRequirement in CWL allows us to use /bin/sh -c in order to pass our command as a string to a shell intrepreter. ShellQuote:false leaves shell metacharacters unquoted, and thus they can still be interpreted and used by a shell.

From CommandLineTool.job():

    shellcmd, _ = self.get_requirement("ShellCommandRequirement")
    if shellcmd is not None:
        cmd = []  # type: List[Text]
        for b in builder.bindings:
            arg = builder.generate_arg(b)
            if b.get("shellQuote", True):
                arg = [shellescape.quote(a) for a in aslist(arg)]
            cmd.extend(aslist(arg))
        j.command_line = ["/bin/sh", "-c", " ".join(cmd)]

Without the ShellCommandRequirement the job will run with subprocess.Popen(shell=False) and thus no shell metacharacters can be used.

My misunderstanding was I thought that all CommandLineTools were run using a shell and could use metacharacters.

ADD COMMENTlink modified 5 months ago • written 5 months ago by ionox0370
1

I just wanted to drop by to say i love that you always take the time to come back and explain, even after solving the issue for yourself.

ADD REPLYlink written 5 months ago by Tom340

Thanks Tom! I still don't understand whether additiona arguments need to be quoted after the pipe when using /bin/sh -c, or whether that doesn't matter.

ADD REPLYlink written 5 months ago by ionox0370

I have no idea either! I'm not great at cwl yet. Just facing my own cwl-reated issues almost daily atm, and your threads often come up when i'm looking for solutions. Thank you for providing so many answers!

ADD REPLYlink written 5 months ago by Tom340
2

Glad it's helpful! It is also helpful for myself to be able to explain it. Another thing I realized was that even though the command is printed like that by cwltool when it runs the command, cwltool is a python process, and it's not the same as running the command from a shell / terminal outside CWL

ADD REPLYlink written 5 months ago by ionox0370
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 557 users visited in the last hour