samtools sort error

Question

[evelyn]

Hello All,

I am running bwa-mem with compressed paired-end input files using an array. I am trying to make sorted bam files as well. I am using this array:

INPUT=$(ls -1 input/*_1.fq.gz | sed -n ${RUN}p)
SAMPLE=$(basename "$INPUT" | sed 's/_1.fq.gz//')

bwa mem GENOME ${SAMPLE}_1.fq.gz ${SAMPLE}_2.fq.gz | samtools sort -o ${SAMPLE}_sorted.bam

However, at some point it stops running with an error:

[E::hts_open_format] fail to open file 'sorted.bam.tmp.0000.bam'
[bam_sort_core] failed to create temporary file "sorted.bam.tmp.0000.bam": File exists

I used the same array for uncompressed files before and there was no error. I am trying to save some space by using compressed files.

I will appreciate any input. Thanks!

[Ram]

basename takes a parameter to trim, you don't need the sed. Even without that, you could use ${INPUT/%_1.fq.gz/}

INPUT=ABCD_1.fq.gz
basename $INPUT _1.fq.gz
ABCD #Output
echo ${INPUT/%_1.fq.gz/}
ABCD #Output

[WouterDeCoster]

You ran this command, or similar, before in this directory. These temporary files need to be removed first.

[evelyn]

Thank you! I tried this. I removed the temporary files from previous runs. I also tried running it after making a new directory. But it always gives the same error.

Answer 1

[Ram]

You're overwriting your output file. Use -o ${SAMPLE}_sorted.bam instead of just -o sorted.bam

[evelyn]

This is my mistake to not mention it in the question correctly. I am running as you suggested but I didn't write it. I have edited my question. Thank you!

[ATpoint]

If the error persists you are still overwriting things. Check if ${SAMPLE} indeed contains information or if it is constantly empty because you did not correctly feed in information.

[evelyn]

echo $SAMPLE gives the sample names in the directory. It does not show as it is empty.

[Ram]

I have a feeling that your $SAMPLE has all the samples in a single string, breaking your bwa mem command. Can you try:

echo "bwa mem GENOME ${SAMPLE}_1.fq.gz ${SAMPLE}_2.fq.gz"

instead of the bwa mem | samtools sort line?

[evelyn]

I ran it and shows one sample out of the four in that directory.

bwa mem GENOME sample/USD1_1.fq.gz sample/USD1_2.fq.gz

[Ram]

The problem is with how you pick your $INPUT. Try a loop:

for f in input/*_1.fq.gz
do
INPUT=$(sed -n ${RUN}p $f)
SAMPLE=${INPUT/%_1.fq.gz/}
echo "bwa mem GENOME ${SAMPLE}_1.fq.gz ${SAMPLE}_2.fq.gz | samtools sort -o ${SAMPLE}_sorted.bam"
done

If the commands echoed look fine, remove the echo and the double quotes and you'll be all set.

[evelyn]

I am not sure if I would switch to a loop especially when this array worked for my similar jobs earlier. I am still trying to find out why I am getting an error regarding .tmp files.

[Ram]

If the loop works, you'll know the problem lies with the array structure and not the code being executed. It helps you in the process of elimination.