Nextflow runs Successfully but output files are empty. How to debug?

Question

[rohitsatyam102]

Hi Everyone!!

I am trying to alter the reference fasta files with the vcfs that I have. For that I wrote the following nextflow script. It runs successfully but the output fasta files are empty

nextflow.enable.dsl=2
params.raw = "results/06_gatkresults/vcfs/*.vcf.gz"
params.ref = "$baseDir/results/00_indexes/bwaidx/sequence.fasta"
params.outdir="results/07_assembly"
params.bed="$baseDir/results/05_lowcoverageBed"


process COV{
    publishDir "$params.outdir"
    input:
    tuple val(sid), path(vcf)
    output:
    file "*.fa"
        file "*.fasta"
    script:
    """ 
    echo ${sid}
    bcftools consensus -f ${params.ref} ${vcf} | sed 's/NC_045512.2/${sid}/' > ${sid}.fa 
    bedtools maskfasta -fi ${sid}.fa -bed ${params.bed}/${sid}.bed -fo ${sid}.fasta
    """
}

reads_ch = Channel.fromPath(params.raw).map { file -> tuple(file.simpleName, file) }

workflow {
  COV(reads_ch)
}

All the files are of zero bytes. How to find out what's wrong?

N E X T F L O W  ~  version 21.10.6
Launching `07_assembly.nf` [curious_swartz] - revision: 684fccf78d
executor >  local (12)
[8f/141386] process > COV (4) [100%] 12 of 12 ✔

[Pierre Lindenbaum]

and what's the content of

cat work/8f/141386*/.command.sh
cat work/8f/141386*/.command.out
cat work/8f/141386*/.command.err

also, retry with

set -x
set -o pipefail
echo ${sid}
bcftools version
bcftools consensus -f "${params.ref}" "${vcf}" | sed 's/NC_045512.2/${sid}/' > "${sid}.fa"
test -s "${sid}.fa"
bedtools maskfasta -fi ${sid}.fa -bed "${params.bed}/${sid}.bed" -fo "${sid}.fasta"
test -s "${sid}.fasta"

[Christopher]

.command.sh can be found in the process folder in the work directory. You will need to navigate to the right folder. In the initial post this would be work/8f/141386.... It's a hidden file so you will need something like ls -la to see it.

Answer 1

[colindaven]

1. Shouldn't your output files actualy be a path if you want to do a glob like "*.fasta" ? Can't you define as ${sid}.fasta like in the script section ?
2. Check your work directories and the logs (errors ?)
3. from the work directories use the various hidden files prepended by a dot eg .command.sh or whatever. One of these can be run with the exact input you have linked in. You can test, modify and improve this command directly, applying it in this work directory, before you know what's going wrong and repair your full script

[rohitsatyam102]

1. In the previous six scripts based on same template file worked just fine without any problem. But yes path I tried with output: path "*.fa" but the files are still empty.
2. Here is the log file: https://gist.github.com/Rohit.
3. I am not sure where to find .command.sh. Can you help?

[rohitsatyam102]

Tried your suggestion 1 and still the files are empty

nextflow.enable.dsl=2
params.raw = "results/06_gatkresults/vcfs/*.vcf.gz"
params.ref = "$baseDir/results/00_indexes/bwaidx/sequence.fasta"
params.outdir="results/07_assembly"
params.bed="$baseDir/results/05_lowcoverageBed"


process COV{
        publishDir "$params.outdir"
        input:
        tuple val(sid), path(vcf)
        output:
        path "*.fa"
        script:
        file1= sid + '.fa'
        file2= sid + '.fasta'
        """
        echo ${sid}
        bcftools consensus -f ${params.ref} ${vcf} | sed 's/NC_045512.2/${sid}/' > $file1
        bedtools maskfasta -fi ${sid}.fa -bed ${params.bed}/${sid}.bed -fo $file2
        """
}

reads_ch = Channel.fromPath(params.raw).map { file -> tuple(file.simpleName, file) }

workflow {
  COV(reads_ch)
}

Answer 2

[rohitsatyam102]

Okay!! So I tried just running the bcftools and I realized that it wasn't able to locate the index file .csi unlike other tools like bwa and gatk. So I have to use ${vcf.toRealPath()} instead of just ${vcf} and that solved the issue. Thanks anyway for helping me colindaven