There are at least 2 things that I would try:
1) Another method (DESeq2, limma-voom), etc. There are also multiple ways to calculate a p-value within edgeR (such as with dispersions estimated for "edgeR-robust")
You may not be able to lock-down the exact right one to use for a project ahead of time. Sometimes the results are similar, sometimes they are very different.
For example, it is a work in progress, but (when I have some free time) I have been trying to show this with public data, and I have accordingly added a "Update" to my GitHub acknowledgement. For example, for 2 out of the 3 E-MTAB-2682 comparisons, I could identify the gene being altered with DESeq2 or limma-voom but not edgeR (but the point is that you will find a project where the method doesn't work if you test enough projects, not that edgeR is worse than DESeq2 or limma-voom). You can see this in the Target_Recovery_Status.xlsx file on the SourceForge page (which should continually change over time, but probably slowly).
2) You can try increasing the FDR to 0.25 (or I have even seen the FDR increased to 0.50 to try and decrease false negatives). However, if you use a method for RNA-Seq analysis, I think it is rare to find no differences with FDR < 0.50 with any method. So, it is probably good to think of those results like a hypothesis.
Incomplete information. Can you provide the complete script? How do the samples look on PCA plot or distance matrix?
The number and nature of the samples would be more important I guess. It probably comes down to an underpowered experiment.
In any of these three glmGLFTest I didn't get FDR <0.05, while 100s of genes with Pvalue <0.05 (even 60+ genes P< 0.01)
Hi. As everyone suggested. Check the PCA on raw counts and with normalized counts and see clustering pattern. You might have some unwanted variations present that is affecting the FDR. How many replicates you have? You may try different normalisation methods like upper quartile or tmm and see how PCA changes. Hope this help.
In addition to the PCA plot of your sample could you post the p-value (not FDR) histogram? Will help us till if the model you build is okay.