Measure distance between genesets

Question

[Folder40g]

I have several genesets from differential expression analyses (30 to 400 genes at each geneset). These genesets come from performing expression analysis comparing different cell types, I would like to see whether those genesets from same cell type are more similar than those from other cell types.

I've tried to use jaccard index and hamming distance, the first doesn't seem to show anything, hamming distance seems to reveal the underlaying pattern a little bit better but still not clear.

Is there any methodology, package or best distance metric to measure distances between genesets? I've also tried hypergeometric tests. Thanks

[Joe]

Jaccard woul have been my suggestion, but you could try the cosine distance perhaps? (Treat all the gene names as a word vector)