How To Shuffle For Simulation-Based Fdr

Question

[brentp]

After finding differentially expressed genes (or methylation sites) that pass some Benjamini-Hochberg FDR cutoff, I often shuffle the data, re-run the pipeline and see how many (spurious) differently expressed genes are found that pass that cutoff.

Say we have a model like:

expression ~ disease + age + gender + race

Generally, I shuffle the entire clinical data set, so that each individual is associated a random expression vector. Is there and advantage to instead, shuffling just the single covariate of interest. So, in the case above, I'd shuffle only the disease covariate instead of shuffling all the individuals.

Any literature on this?

Answer 1

[Istvan Albert]

Well this are just musings rather than a answer based on careful analysis.

I would think that it all comes down to wether the covariates do actually affect the expression levels or not. Randomizing a meaningful covariate will lead to a different result than that of a non-meaningful one. If the predictive power of some attributes is really low randomizing that may still lead to equally good predictions.

Answer 2

[Arun]

Just a side note: In case you're looking at quite a few thousand genes, then I think you wouldn't see much of a difference between BH and permutation/bootstrap techniques.

To answer your question, I think it depends on how essential each predictor turned out in your initial p-value. If there is no significant effect of "race" towards your analysis, then there wouldn't be much contribution to detecting events due to chance by permuting it, as it already doesn't seem to. Sorry if that wasn't too helpful. But you should post this question on stats.stackexchange.com. Its more appropriate forum. I'd love for you to post this question there to see their replies.

[brentp]

thanks for the ideas. I search on stats.stackexchange and found some things to look at, e.g.:http://en.wikipedia.org/wiki/Resampling_(statistics) . (I do see a difference between BH and permutation-based correction of pvalues)