Samtools fixmate changes bitwise flag 67 to 65 or 131 to 129

Question

[terra1666]

Before executing 'samtools rmdup' to remove PCR duplicates from sam/bam file, I had executed 'samtools fixmate' as recommended somewhere. The lines in original sam/bam file of a pair of pair-end reads(ex. 6qyfyza) were shown like:

6qyfyza  67      1       182426  0       26M     =       182628   203    
6qyfyza  131     1       182628  0       22M     =       182426  -203

but after doing 'samtools fixmate', were changed like:

6qyfyza  65      1       182426  0       26M     =       182628   202    
6qyfyza  129     1       182628  0       22M     =       182426  -202

where not only the insert size(9th column) was modified from 203 to 202, but also bitwise flag(2nd column)was changed from 67 to 65 or 131 to 129. This happened to all the read pairs with 67 and 131 in the sam/bam file.

Is this really what was expected to happen? According to this site, it is explained that 65 and 129 means "mapped uniquely but wrong insert size, and could possibly reside in different contigs", so why am I getting such a bad flag by trying to fix insert size with samtools fixmate?

[terra1666]

Thank you for clear answer. Because I needed to work on old bam files whose original fastq files are missing, I don't know why mapping is in such orientation.

I usually use picard MarkDupilicates which only didn't work for the old bam files with the error:

htsjdk.samtools.SAMException: Value was put into PairInfoMap more than once

so I was thinking to use samtools rmdup for dedup. Now I will try samtools markdup. Thank you.

Answer 1

[Devon Ryan]

samtools fixmate is expecting standard illumina orientation for paired-end reads. Your example reads lack that, so it's unsetting the "properly paired" flag.

I strongly suggest that you not use samtools rmdup, it is antequated and only still exists for backward compatibility. If you want to use samtools, then you're strongly encouraged to use samtools markdup, assuming you have version 1.6. Alternatively, use picard MarkDuplicates.

[owen.chapman1]

I believe picard MarkDuplicates requires the 0x002 flag. What is the "standard illumina orientation" that fixmate expects?

[Devon Ryan]

Mates should point toward each other, as in standard paired-end data.