I have a very ignorant question. Let's say the SNP X has an allele A with a frequency of 0.52 and 0.002 in populations 1 and 2, respectively. In some papers I have read that people remove SNPs with MAF<5% in either of the populations when calculating Fst. These values suggest that A is very differentiated between pop1 and pop2. Indeed, I calculated Fst for SNP X and it has a value of ~0.9. But if I use the MAF>5% criterion, I would remove this strong signal of population differentiation. This does not make much sense for me. I would very much appreciate to have some feedback. Thanks !
These guys, published in Genome Research, have addressed just this issue of allele frequency when calculating Fst: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3759727/
Their results show just as you have implied, i.e., that the Fst is dependent on the allele frequency, but in addition they imply that the sample size is important. On that note, rare variants, being rare, will naturally be encountered less in populations and it is possible only now (recent years) that we have accumulated sequencing data on 1000s of individuals such that we can actually begin to analyse rare variants in various metrics, including Fst.