Quantiles Normalization On Massive Dataset
2
2
Entering edit mode
10.9 years ago
Paul ▴ 760

Hi,

I've got a massive dataset of 160 Affy Exon arrays that I need to quantiles normalize at probe level (the reasons for which are complicated!), which means ~6 million probes per sample.

Trying to load the into memory maxes out 32gigs of memory on one of our cluster nodes.

I clearly need to find a way to do this without loading it into memory, would anyone perhaps have a pointer/suggestion?

Thanks!

normalization • 2.2k views
ADD COMMENT
6
Entering edit mode
10.9 years ago
Laurent ★ 1.7k

A few suggestions:

  • Aroma affy or xps for on-disk access instead of loading everything into memory,
  • fRMA (frozen RMA) to pre-process single or batches of arrays against precomputed reference probe effects.

Hope this helps.

ADD COMMENT
0
Entering edit mode
10.8 years ago

I also recommend Aroma affy, but you might also consider RMAExpress

ADD COMMENT

Login before adding your answer.

Traffic: 1328 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6