I am wondering if anyone can help me figure out how to build a neighbor joining tree in R, where the pairwise distances (from sequence alignments) are estimated under the GTR model. I have tried the dist.dna() method from R's phanghorn package, but it does not have support for the GTR model. I am new to phylogenetics, and any help would be great.
While you can estimate distances under a handful of models in R, to my knowledge you are only able to estimate trees under the GTR model when using likelihood-based approaches (as, for example, in
optim.pml() and others). Unfortunately, I think you'll have to do this outside of R. Most sophisticated phylogenetic estimation isn't done in R, anyway.
I don't have time to skim through the PAUP manual and provide detailed instructionsl, but you can calculate distances under the GTR model and estimate a tree generally following this tutorial.