Question: How to let WGCNA use multiple cores
0
gravatar for syrttgump
23 months ago by
syrttgump30
USA/Newark/New Jersey Institute of Technology
syrttgump30 wrote:

Hi guys, I am using WGCNA to deal with some large datasets (~20000 genes on ~1000 samples). I read in the help documents that WGCNA support multi-thread to speed up calculation. But when I run functions like blockwiseModules, TOMsimilarityFromExpr, cor etc, I can set parameter nThreads, it works. But when I check CPU usages by "top", it seems all multiple threads are running on one CPU core, so I didn't gain any advantage by multi-thread (I am using a linux cluster with 24 CPU cores, and set enableWGCNAThreads() already). How can I set WGCNA to use multiple cores?

multe-thread parallel wgcna • 1.4k views
ADD COMMENTlink modified 7 months ago by yplee6140 • written 23 months ago by syrttgump30

have you fix it problem?

ADD REPLYlink written 7 months ago by yplee6140

Can you try running these lines at the top of your script(s):

require(parallel)
require(doParallel)

cpucores <- makeCluster(detectCores(), type='PSOCK') ; 

registerDoParallel(cpucores) ; 

Sys.setenv("MC_CORES"=cpucores) ;

The last line (setting MC_CORES variable) is only for Windows OS.

Also take a look at these other WGCNA functions: https://rdrr.io/cran/WGCNA/man/allowWGCNAThreads.html

ADD REPLYlink modified 7 months ago • written 7 months ago by Kevin Blighe39k

I met this problem too, and using the above code could not solve the problem.

Aifu

ADD REPLYlink written 6 months ago by afli170

Then please show what is the error message that you are receiving. In certain situations, you will, simply, not have enough power on your computer to do the analysis that you want. You should consider reducing the size of your input data [to WGCNA]

ADD REPLYlink written 6 months ago by Kevin Blighe39k

I haven't met any error, the WGCNA works well, but it only use one CPU, I've tried using enableWGCNAThreads() or allowWGCNAThreads(nThreads = 22) and so on, but have no effect. I think maybe this package just use one CPU. And I use small dataset or large dataset to WGCNA, also no change.

ADD REPLYlink written 6 months ago by afli170

Which operating system are you using? The parallel functions in R differ based on the 'system architecture' that you use.

ADD REPLYlink modified 6 months ago • written 6 months ago by Kevin Blighe39k

Maybe, I use Ubuntu 16.04 LTS, and the R package is 3.5.0 version. Can the WGCNA(r package) works with multiple threads on your system?

ADD REPLYlink written 6 months ago by afli170

I also use Ubuntu 16.04.

Firstly, try the enableWGCNAThreads() function, and do not use allowWGCNAThreads()

If that does not work, then first run these lines:

#Set CPU cores for parallel-related functions
  cpucores <- 16
  require(parallel)
  options("mc.cores"=cpucores)

  #Set CPU cores for doParallel-related functions
  require(doParallel)
  cores <- makeCluster(detectCores(), type='PSOCK')
  registerDoParallel(cores)

...then run enableWGCNAThreads() again.

Check if multiple threads are running via the top command in BASH. You will see the R command being forked

ADD REPLYlink modified 6 months ago • written 6 months ago by Kevin Blighe39k

Thanks Kevin, I try the codes above and add enableWGCNAThreads(), but this could not increase the threads number. The program has cost more than 10 hs, and is still running for just one thread. One of my friends told me that WGCNA just uses one core. Maybe there're other methods to increase the thread number. I upload the data and paste my codes as follows:

library(WGCNA)
options(stringsAsFactors = FALSE)
enableWGCNAThreads(10)
lnames=load(file="filtered_data_ready_for_WGCNA_0.5tpm.RData")
datExpr <- datExpr0
rm(datExpr0)
setwd("./maxP0.05")
power <- 7
bwnet = blockwiseModules(datExpr, maxBlockSize = 46000, power = power, networkType = "signed hybrid", TOMType = "unsigned", minModuleSize = 30, corType="bicor", corOptions = "use = 'p', maxPOutliers = 0.05", reassignThreshold = 0, mergeCutHeight = 0.15, numericLabels = TRUE, saveTOMs = TRUE, saveTOMFileBase = "Signed_hybrid_TOM-blockwise_maxP0.05", verbose = 3)

You can try this, if you could run with multiple threads please share the good news with me!

data link: https://de.cyverse.org/dl/d/1D0B60A0-D1EF-4A4F-A13E-A21901301900/filtered_data_ready_for_WGCNA_0.5tpm.RData

ADD REPLYlink modified 6 months ago • written 6 months ago by afli170

Okay, let me try

ADD REPLYlink written 6 months ago by Kevin Blighe39k
1

I tried, and, indeed, that particular function does not have parallel-processing enabled - I looked through the code. I then found this, where one of the WGCNA developers states exactly this:

I notice a lot of for loops in the blockwiseModules code, which could be replaced with foreach %dopar%. I initially tried but ran into errors because the function is >3000 lines of code.

I previously 'parallelised' another much smaller functiun that also performs k-means (or other) clustering: https://github.com/kevinblighe/clusGapKB

ADD REPLYlink modified 6 months ago • written 6 months ago by Kevin Blighe39k

Thank you very much Kevin for your detailed reply! I accept the reality that I should use just one thread to run WGCNA at the moment.

ADD REPLYlink written 6 months ago by afli170

im not sure about it. this is what my initial code for wgcna

library(WGCNA)
enableWGCNAThreads()

it works ...

ADD REPLYlink written 6 months ago by krushnach80470

Do you mean that many threads work simultaneously? Could you please show your system information, and WGCNA version?

I tried again and failed again...

ADD REPLYlink modified 6 months ago • written 6 months ago by afli170
1

krushnach80, good to see you again. Can you confirm that it is the blockwiseModules() function that you are using? We believe, based on This, that it is just the blockwiseModules() function that is not yet multi-thread enabled.

I imagine that you are using a different WGCNA function.

ADD REPLYlink modified 6 months ago • written 6 months ago by Kevin Blighe39k
1

@Kevin sorry for the late response yes im not using "blockwiseModules()" .

ADD REPLYlink written 6 months ago by krushnach80470
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1233 users visited in the last hour