I am estimating nucleotide diversity and other population genetic parameters from samples of flies where there appears to be extensive identity by descent among some pairs of individuals (indicated by much lower pairwise genetic distances). In order to eliminate sample bias due to effective redundancy, I would like the mask the identical by descent (IBD) regions of genome in those pairs of flies.
If I were dealing with haploid genomes (e.g. X chromosomes from male flies only), this would be straightforward: I would take large chromosome windows and identify those that had average pairwise differences per site < alpha, where alpha is some threshhold << average pairwise difference among unrelated individuals. As a point of reference, the average pairwise difference among the related pairs is about 1/2 that of other pairwise differences.
In contrast, I compare autosomal chromosomes (or female X chromosomes), I need to consider diploid-diploid comparisons, or, in the case of male X vs female X, haploid-diploid comparisons. What is the most reasonable way to adjust my alpha for diploid genome IBD estimation, to take into account the fact that I have some combination of heterozygous and homozygous genotypes? If all genotypes were homozygous, obviously I would still use alpha, but as the frequency of heterozygotes increases, it would seem that using alpha/2 becomes more reasonable.
Is there some standard way to do this? If there are tools out there specifically for this that someone can direct me to as well, all the better.
