Samtools mapped and unmapped flags, what is the difference between the flags?
1
0
Entering edit mode
17 months ago
DNAngel ▴ 250

I want to obtain all mapped and unmapped reads from my bamfiles but I don't understand the difference between these sets of flags:

For mapped reads:

samtools view -b -F 4 bamfile > mapped
# versus
samtools view -b -F 4 -F 260 bamfile > mapped

I've seen a ton of tutorials on this and about a small % use this -F 260 flag, which I understand means primary alignment. So does this mean just using -F 4 would extract any mapped read (even with its mate unmapped) and potential secondary alignments? And is this considered an issue?

Also I don't understand why there are sometimes three commands for unmapped reads, or just one command:

For unmapped reads

samtools view -b -f 4 bamfile > -f 4 is unmapped # the most common command, okay simple enough
# versus
samtools view -b -f 4 -F 264 bamfile > unmapped1 # -f 4 is unmapped, but what is -F 264 saying??

The -F 264 is confusing to understand. Isn't that just the combination of flags 4 and 256? So saying -F 264 really reads as NOT unmapped but IS a primary alignment? This is what's confusing me because I do want to keep only reads in the primary alignment but I want them as unmapped. Why isn't this commonly just -f 4 and -F 256? I am basing this on Novocraft and some other tutorials I've seen online.

The other commands to extract remaining unmapped reads:

samtools view -b -f 8 -F 260 bamfile > unmapped2  # -f 8 is a mapped read but mate is unmapped; but -F 260 is the same thing as before, a combo of -f 4 and -f 256, but now the number is different?

Once again, according to samtools flags on the Picard website, 260 translates to mate unmapped (0x8) and not primary alignment (0x100), but using -F 260, wouldn't this mean the opposite? So in fact, it is a mapped mate and it is primary alignment?

The final command:

samtools view -b -f 12 -F 256 bamfile > unmapped3 # -f 12 is read unmapped and mate unmapped; -F 256 is primary alignment; this makes sense to me

I guess at the end of it, if I want mapped and unmapped reads in the primary alignment I would run something like this?

#mapped
samtools view -b -F 4 -F 256 bamfile > mapped
# OR
samtools view -b -F 260  bamfile > mapped
# then sort them
samtools sort -n mapped > mapped_sorted

#unmapped
samtools view -b -f 4 -F 256 bamfile > unmapped1
samtools view -b -f 8 -F 256 bamfile > unmapped2
samtools view -b -f 12 -F 256 bamfile > unmapped3
samtools merge -b unmapped[123] | samtools sort -n - unmapped_sorted

But here I would be keeping the -F 256 flag each time for this to make sense right?

I just want to know what is best practise and if the flags I've chosen make sense because I haven't seen this code in other tutorials but their flags confuse me.

samtools • 1.2k views
ADD COMMENT
0
Entering edit mode
17 months ago

The -F 264 is confusing to understand. Isn't that just the combination of flags 4 and 256?

Wouldn't it be 256 + 8?

Once again, according to samtools flags on the Picard website, 260 translates to mate unmapped (0x8) and not primary alignment (0x100)

260 = 256 + 4.

ADD COMMENT
0
Entering edit mode

Okay but if I already have the flag -f 4 and then I want to call NOT primary alignment, it would be -F 256. So in a sense -f 4 and -F 256, I don't understand how in the same line of code people write -f 4 -F 260. It seems like the -F 260 is including the -f 4 flag but instead of a -f it is -F which to me is the opposite of what I want in that case?

And yes I typo'd I meant 4+256 = 260.

ADD REPLY
0
Entering edit mode

So in Novocraft and other examples they write:

samtools view -u  -f 4 -F 264 alignments.bam  > tmps1.bam
samtools view -u -f 8 -F 260 alignments.bam  > tmps2.bam
samtools view -u -f 12 -F 256 alignments.bam > tmps3.bam

But why isn't it

samtools view -u  -f 4 -F 256 alignments.bam  > tmps1.bam
samtools view -u -f 8 -F 256 alignments.bam  > tmps2.bam
samtools view -u -f 12 -F 256 alignments.bam > tmps3.bam

The -F 264 means NOT mate unmapped and not primary alignment? But that means I am also extracting mapped mates and primary alignment, along with the -f 4 which means unmapped reads.

ADD REPLY

Login before adding your answer.

Traffic: 2677 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6