Question: bwa samtool giving empty bam files while piping
0
gravatar for Goku
10 months ago by
Goku0
Goku0 wrote:

Hi

I am getting empty bam files while using the following command. Any help is very much appreciated. Part of the output is also shown.

$ bwa mem -t 8 hg19.fa xxx_Q30_trim.fastq.gz | samtools view -bS - | samtools view -b -q 30 - | samtools sort - > xxx.bam

Thank you

[bam_header_read] EOF marker is absent. The input is probably truncated.

Usage: samtools sort [options] <in.bam> <out.prefix>

Options: -n sort by read name -f use <out.prefix> as full file name instead of prefix -o final output to stdout -l INT compression level, from 0 to 9 [-1] -@ INT number of sorting and compression threads [1] -m INT max memory per thread; suffix K/M/G recognized [768M]

[M::bwa_idx_load_from_disk] read 0 ALT contigs

rna-seq bwa samtools bam pipeing • 598 views
ADD COMMENTlink modified 10 months ago by h.mon21k • written 10 months ago by Goku0

Are you using a reasonably recent samtools version? I think you can reduce your samtools view commands to just one.

Is there more output than what you showed? Please add everything.

ADD REPLYlink written 10 months ago by WouterDeCoster35k

Program: samtools (Tools for alignments in the SAM format) Version: 0.1.19-44428cd

I will try with the new version too. I had submitted 10 files, so the output is pretty huge.

But one thing i noticed it contained this line [main] Version: 0.7.17-r1188 [main] CMD: bwa mem -t 8 /home/rcf-proj2/hg1/software/hg19_index/hg19.fa WTDAC5_Q30_trim.fastq.gz [main] Real time: 7239.654 sec; CPU: 5991.808 sec Does this mean piping is not working. I will try other things that are suggested here.

ADD REPLYlink modified 10 months ago • written 10 months ago by Goku0

Maybe this means you are using nohup and forgot to tell us?

ADD REPLYlink written 10 months ago by h.mon21k

i don't know whats nohup is. But the - - thing worked for me that you suggested.

But I didn't get any output log file. generally i should have got one. I am using pbs job submission. I think this may be a different problem. Now I am not sure whether I should trust these bam files, they look fine. but i still should find out about those output log files....may be from system administrator.

Thanks,

ADD REPLYlink written 10 months ago by Goku0
1
gravatar for h.mon
10 months ago by
h.mon21k
Brazil
h.mon21k wrote:

I don't know your SAMtools version, but as you use -S, I guess it is old. The following works for me with SAMtools 1.6:

bwa mem -t 8 hg19.fa xxx_Q30_trim.fastq.gz | \
    samtools view -b -q 30 -o - - | samtools sort -o xxx.bam -

I think the problem is you need two dashes for the samtools commands, if you want to redirect both standard input and standard output.

ADD COMMENTlink written 10 months ago by h.mon21k

OK. So i got the new samtools Program: samtools (Tools for alignments in the SAM format) Version: 1.7 (using htslib 1.7) It looks very different from the old one.

also the - - thing worked. (I thought the stdout is suppose to be auto redirected without a dash. I just started using pipes and &)

Thanks a lot

ADD REPLYlink written 10 months ago by Goku0

The & is nohup and likely the source of your problem

ADD REPLYlink written 10 months ago by WouterDeCoster35k

How are you using &? The & may work differently, depending on how you use it:

  • stdout 2 stderr (1>&2)
  • stderr 2 stdout (2>&1)
  • stderr and stdout 2 file (&>file)

You probably piped stderr into stdout.

ADD REPLYlink written 10 months ago by h.mon21k

Or just use the & at the end of the command to run it in the background (googling tells me it's not exactly the same as nohup though).

ADD REPLYlink written 10 months ago by WouterDeCoster35k

I use it at the end of line like this.

bwa mem -t 8 hg19.fa xxx1_Q30_trim.fastq.gz | samtools view -b -q 30 -o - - | samtools sort - -o xxx0.bam &
bwa mem -t 8 hg19.fa xxx5_Q30_trim.fastq.gz | samtools view -b -q 30 -o - - | samtools sort - -o xxx5.bam
ADD REPLYlink modified 10 months ago • written 10 months ago by Goku0
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1117 users visited in the last hour