Question: Is there a way to use multiple processors (parallelize) to create a heatmap for a large dataset?
gravatar for Pratik Mehta
7 weeks ago by
Pratik Mehta80
University of Connecticut
Pratik Mehta80 wrote:

I have a gene-gene correlation matrix of about 20,000 genes by 20,000 genes. I am trying to generate a heatmap similar to how JMP can create plots like this:

Image from Sunil Archak (

Image from Sunil Archak (

I want to see how the resulting data will look. Unfortunately, as I was typing this R Studio Aborted the session as it was trying to generate the heatmap. Too much to handle?

Any ideas?

Alternatively, I was advised to plot genes by principle components. I may be leaning towards this approach now... unless someone knows a solution to this. I do have a decent amount of cores (16) and and decent amount of RAM 128GB.

Any help would be appreciated.

Very Respectfully, Pratik

ADD COMMENTlink modified 6 weeks ago • written 7 weeks ago by Pratik Mehta80

If you have access to a cluster, it might help to use that and generate a file output instead of interactive graphic output.

ADD REPLYlink written 7 weeks ago by _r_am32k

Thank you for respond sir. Let me try this...

ADD REPLYlink written 7 weeks ago by Pratik Mehta80

Sir, do you know if there is a way to parallelize the process though, because it is not an issue with computing power? The issue here is R and the R packages I am using are not using the full computing power. Nearly the whole server I am using, is idle... If there is some way to use all of the processors, this would speed up the process substantially for me.

Very Respectfully, Pratik

ADD REPLYlink written 7 weeks ago by Pratik Mehta80

Unfortunately, no. If the package is not designed to use multiple cores, there's not much you can do about it.

ADD REPLYlink written 7 weeks ago by _r_am32k

Maybe try in the standard R (command-line)? R Studio uses unnecessary extra resources.

pvclust can do clustering in a parallelised fashion, but it does not generate heatmaps.

If you want, start a new session on command-line and show the output of sessionInfo()?

ADD REPLYlink modified 6 weeks ago • written 6 weeks ago by Kevin Blighe69k

Thank you for your response sir. I understand now why @_r_am was suggesting to access a cluster for this. This is a huge job even for my personal server, I think? I'm asking the machine to plot a 20,000 genes x 20,000 genes matrix with colors associated with different levels of correlation. The matrix is 2.7GB alone.

Do you think hierarchical clustering using pvclust and then using the generated clusters to plot a heatmap/dendrogram will speed up the computation, or will hierachical clustering be yet another layer of data ontop of the gene-gene correlations?

EDIT: I'm pretty sure this is what I wanted to do from the beginning just not sure how to... This was also the suggestion I got from a mentor. Cluster first and then plot in dendrogram-heatmap. But I'm wondering about how computationally tractable it will be to do this?

I guess I could set it up either way, maybe? genes by clusters if I want to? or genes by genes with cluster labeling? The latter would probably be more computationally heavy? (I think that these types of dendrogram-heatmaps only work on a square matrix so the latter may be required for this?

The next challenge, I think, would be figuring out how to take the cluster assignments from pvclust, group one of the 20,000 genes by 20,000 genes axes into the cluster assignments, and then use a package like Heatmap2.0 to create a beautiful plot like this: Image obtained from:

Please correct me if I'm wrong on this! Or if you have any pointers on this process, please.

Here is my sessionInfo():

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/

 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.0.2

I know it's not a purely command-line server (using Ubuntu desktop), but it does have some decent computing power.

Thank you again @_r_am and @Kevin Blighe.

Very Respectfully, Pratik

ADD REPLYlink modified 6 weeks ago • written 6 weeks ago by Pratik Mehta80

See if you can do that with ComplexHeatmap. It's quite extensible and might allow for the clustering and dendrograms to be precomputed - or you should atleast be able to pre-compute row and column orders and separate the clustering compute consumption from the graphical rendering time consumption.

ADD REPLYlink written 6 weeks ago by _r_am32k

Thank you very much sir! I will try this!

Very Respectfully, Pratik

ADD REPLYlink modified 6 weeks ago • written 6 weeks ago by Pratik Mehta80
gravatar for Pratik Mehta
6 weeks ago by
Pratik Mehta80
University of Connecticut
Pratik Mehta80 wrote:

Just thought I'd post this as my solution, in case anyone is stuck in a similar situation.

How do I subset a Seurat object using variable features?

I know it's common practice in analyzing data (ie. to take a sample of the larger dataset to get a gist of the data). Some sort of heatmap-dendrogram is better than no heatmap-denodrgram!!

So rather than doing all 20,000 by 20,000 genes I used Seurat to take the top 1000 variable features and then made the corresponding gene-gene correlation matrix after transposing the table using cor() and then plotted a "rough-draft" using the base-R heatmap() function:

Heatmap of top 1000 variable features by top 1000 variable features


I do plan to use ComplexHeatmap as suggested by @_r_am to generate a prettier dendrogram-heatmap.

Hope someone finds this useful!

Very Respectfully, Pratik

ADD COMMENTlink written 6 weeks ago by Pratik Mehta80
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1104 users visited in the last hour