Question: bwa samtool giving empty bam files while piping
1
gravatar for Goku
16 months ago by
Goku10
Goku10 wrote:

Hi

I am getting empty bam files while using the following command. Any help is very much appreciated. Part of the output is also shown.

$ bwa mem -t 8 hg19.fa xxx_Q30_trim.fastq.gz | samtools view -bS - | samtools view -b -q 30 - | samtools sort - > xxx.bam

Thank you

[bam_header_read] EOF marker is absent. The input is probably truncated.

Usage: samtools sort [options] <in.bam> <out.prefix>

Options: -n sort by read name -f use <out.prefix> as full file name instead of prefix -o final output to stdout -l INT compression level, from 0 to 9 [-1] -@ INT number of sorting and compression threads [1] -m INT max memory per thread; suffix K/M/G recognized [768M]

[M::bwa_idx_load_from_disk] read 0 ALT contigs

rna-seq bwa samtools bam pipeing • 936 views
ADD COMMENTlink modified 16 months ago by h.mon25k • written 16 months ago by Goku10

Are you using a reasonably recent samtools version? I think you can reduce your samtools view commands to just one.

Is there more output than what you showed? Please add everything.

ADD REPLYlink written 16 months ago by WouterDeCoster39k

Program: samtools (Tools for alignments in the SAM format) Version: 0.1.19-44428cd

I will try with the new version too. I had submitted 10 files, so the output is pretty huge.

But one thing i noticed it contained this line [main] Version: 0.7.17-r1188 [main] CMD: bwa mem -t 8 /home/rcf-proj2/hg1/software/hg19_index/hg19.fa WTDAC5_Q30_trim.fastq.gz [main] Real time: 7239.654 sec; CPU: 5991.808 sec Does this mean piping is not working. I will try other things that are suggested here.

ADD REPLYlink modified 16 months ago • written 16 months ago by Goku10

Maybe this means you are using nohup and forgot to tell us?

ADD REPLYlink written 16 months ago by h.mon25k

i don't know whats nohup is. But the - - thing worked for me that you suggested.

But I didn't get any output log file. generally i should have got one. I am using pbs job submission. I think this may be a different problem. Now I am not sure whether I should trust these bam files, they look fine. but i still should find out about those output log files....may be from system administrator.

Thanks,

ADD REPLYlink written 16 months ago by Goku10
1
gravatar for h.mon
16 months ago by
h.mon25k
Brazil
h.mon25k wrote:

I don't know your SAMtools version, but as you use -S, I guess it is old. The following works for me with SAMtools 1.6:

bwa mem -t 8 hg19.fa xxx_Q30_trim.fastq.gz | \
    samtools view -b -q 30 -o - - | samtools sort -o xxx.bam -

I think the problem is you need two dashes for the samtools commands, if you want to redirect both standard input and standard output.

ADD COMMENTlink written 16 months ago by h.mon25k

OK. So i got the new samtools Program: samtools (Tools for alignments in the SAM format) Version: 1.7 (using htslib 1.7) It looks very different from the old one.

also the - - thing worked. (I thought the stdout is suppose to be auto redirected without a dash. I just started using pipes and &)

Thanks a lot

ADD REPLYlink written 16 months ago by Goku10

The & is nohup and likely the source of your problem

ADD REPLYlink written 16 months ago by WouterDeCoster39k

How are you using &? The & may work differently, depending on how you use it:

  • stdout 2 stderr (1>&2)
  • stderr 2 stdout (2>&1)
  • stderr and stdout 2 file (&>file)

You probably piped stderr into stdout.

ADD REPLYlink written 16 months ago by h.mon25k

Or just use the & at the end of the command to run it in the background (googling tells me it's not exactly the same as nohup though).

ADD REPLYlink written 16 months ago by WouterDeCoster39k

I use it at the end of line like this.

bwa mem -t 8 hg19.fa xxx1_Q30_trim.fastq.gz | samtools view -b -q 30 -o - - | samtools sort - -o xxx0.bam &
bwa mem -t 8 hg19.fa xxx5_Q30_trim.fastq.gz | samtools view -b -q 30 -o - - | samtools sort - -o xxx5.bam
ADD REPLYlink modified 16 months ago • written 16 months ago by Goku10
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 653 users visited in the last hour