ls: cannot access *.filter.trim.fastq.gz: No such file or directory (problems when combining individual steps into a full script)

Question

[Kai_Qi]

Hi:

I am learning to combine my individual steps into a whole script so that I can use it for any other file in future. My strategy is to combine them step by step. In the middle I got an error I couldn't understand, the content of my newly generated script (got an error at the very last step) is here:

#!/bin/bash
mkdir filtering
cd filtering/
for f in HePG2.HNRNPK.rep1.R2 HePG2.HNRNPK.rep2.R2; do
~/anaconda2/bin/perl ~/CTK_toolkit/ctk/fastq_filter.pl -v -if sanger -f mean:0-24:20 -of fastq ../fastq/$f.fastq.gz - | gzip -c > $f.filter.fastq.gz
done

# cut adapt
cutadapt -f fastq --match-read-wildcards --times 1 -e 0.1 -O 1 --quality-cutoff 5 -m 20 -a ATTGCTTAGATCGGAAGAGCGTCGTGT -a ACAAGCCAGATCGGAAGAGCGTCGTGT -a AACTTGTAGATCGGAAGAGCGTCGTGT -a AGGACCAAGATCGGAAGAGCGTCGTGT -a ANNNNGGTCATAGATCGGAAGAGCGTCGTGT -a ANNNNACAGGAAGATCGGAAGAGCGTCGTGT -a ANNNNAAGCTGAGATCGGAAGAGCGTCGTGT -a ANNNNGTATCCAGATCGGAAGAGCGTCGTGT -o HePG2.HNRNPK.rep1.R2.filter.trim.fastq.gz HePG2.HNRNPK.rep1.R2.filter.fastq.gz > HePG2.HNRNPK.rep1.R2.cutadapt.log &
cutadapt -f fastq --match-read-wildcards --times 1 -e 0.1 -O 1 --quality-cutoff 5 -m 20 -a ATTGCTTAGATCGGAAGAGCGTCGTGT -a ACAAGCCAGATCGGAAGAGCGTCGTGT -a AACTTGTAGATCGGAAGAGCGTCGTGT -a AGGACCAAGATCGGAAGAGCGTCGTGT -a ANNNNGGTCATAGATCGGAAGAGCGTCGTGT -a ANNNNACAGGAAGATCGGAAGAGCGTCGTGT -a ANNNNAAGCTGAGATCGGAAGAGCGTCGTGT -a ANNNNGTATCCAGATCGGAAGAGCGTCGTGT -o HePG2.HNRNPK.rep2.R2.filter.trim.fastq.gz HePG2.HNRNPK.rep2.R2.filter.fastq.gz > HePG2.HNRNPK.rep2.R2.cutadapt.log & 
#checkread
for f in `ls *.filter.trim.fastq.gz`; do
c=`zcat $f | wc -l`
c=$((c/4))
echo $f $c
done

The error happens when I used the checkread for loop. I copied it from a ".sh" file and put it here. The error I received is:

ls: cannot access *.filter.trim.fastq.gz: No such file or directory

So I did

$ cd filtering/
$ for f in `ls *.filter.trim.fastq.gz`; do
> c=`zcat $f | wc -l`
> c=$((c/4))
> echo $f $c
> done

to see what could be the problem, and it went through without a problem. I also changed the ls *.filter.trim.fastq.gz; part using absolute path in the combined script , but still the same error ls: cannot access /scratch/midway2/caiqi/HNRNPK_eCLIP_HePG/test/filtering/*.filter.trim.fastq.gz: No such file or directory

what is wrong with my strategy and what should i do to correct it?

Thanks,

[antonioggsousa]

Add to your script before the checkread loop the following line echopwd to see where the script is running. Like this:

#checkread
echo `pwd`
for f in `ls *.filter.trim.fastq.gz`; do
c=`zcat $f | wc -l`
c=$((c/4))
echo $f $c
done

I hope this helps,

António

[Kai_Qi]

Thanks, I checked the log and find that I was in the filtering directory when running the script

Answer 1

[Ram]

You're running both cutadapt commands as background jobs and then using their output in a foreground task without waiting for the bg jobs to run to completion. The cutadapt tasks probably don't get a chance to create the output before your checkread searches for the file, which is why you see the error.

Remove the &s and let the cutadapt tasks run in the foreground and your script will work fine.

[Kai_Qi]

Thank you so much. It works out and helps my understand better