Question: For loop parallelisation
0
gravatar for melania 2282
4 months ago by
melania 228270
melania 228270 wrote:

Hello, please I need your help to make this loop works, its a permutation+metaanlysis on two sets of data, but i have an error message showing I need foreach object, The idea is to get the results of N permutations (n.iter in my code) of the meta-anlysis of two sets of data, the association model is perforemd seperately on 1000 SNPs so for every merta analysis I need the results of 1000 SNPs in the two studies, that's why I am using foreach twice. Thanks for your help :

foreach(n.iter) %dopar%   
foreach(i=22:7269) %dopar% {    

Y_1<-as.vector(sample(V1,100,replace = F))  
fit_1<-glm(Y_1~data[,i],data=data,family=binomial())    

Y_2<-as.vector(sample(V2,100,replace = F))  
fit_2<-glm(Y_2~data2[,i],data=data2,family=binomial())      

sfit_y1<-summary(fit_y1)  
sfit_y2<-summary(fit_y2)  

p_yt<-sfit_y1$coefficients[3,4]  
p_ct<-sfit_y2$coefficients[3,4]  

p<-as.data.frame(p1=p_yt,n1=100,p2=p_ct,n2=100)  
meta2<-metap(p,2,verbose="N")  

meta<-(-log10(meta2[,2]))  
return(meta)   
}

and this is the original for loop I am trying to rewrite for parallelized computing as it takes too long time :

for (s in 1:n.iter) {
  for(i in 22:nvar)
  {p<-data.frame(p1=numeric(n.var),n1=rep(100,n.var),p2=numeric(n.var),n2=rep(100,n.var))
  Y_y1<-as.vector(sample(data$PHENOTYPE,100,replace = F))
  fit_y1<-glm(Y_y1~data[,i],data=data,family=binomial())  
  Y_y2<-as.vector(sample(data2$PHENOTYPE,100,replace = F))
  fit_y2<-glm(Y_y2~data2[,i],data=data2,family=binomial()) 
  j<-i-21
  sfit_y1<-summary(fit_y1)
  sfit_y2<-summary(fit_y2)
  p$p1[3]<-sfit_y1$coefficients[3,4]
  p$p2[3]<-sfit_y2$coefficients[3,4]
  x<-metap(p[j,],2,verbose="N")
  }
  meta[s,j]<-x[,2]
}
write.csv2(meta,"meta.txt",row.names = F)

Thank you for your help !

R • 205 views
ADD COMMENTlink modified 4 months ago • written 4 months ago by melania 228270
1

Why are you using foreach twice ?

ADD REPLYlink written 4 months ago by Kevin Blighe56k

Thank you Kevin, I edited my post to explain better the objective of my code

ADD REPLYlink written 4 months ago by melania 228270
4
gravatar for Kevin Blighe
4 months ago by
Kevin Blighe56k
Kevin Blighe56k wrote:

foreach returns the output of each loop as a list object, which can then be 'bound' together like this:

res <- foreach(...) %dopar% {
.. }

do.call(rbind, res)

Depending on what you're doing inside the foreach loop, one can also use:

data.table::rbindlist(res)

You can also control how the output is bound together within the foreach() function call itself:

foreach(l = seq_len(blocks),
    .combine = rbind,
    .multicombine = TRUE,
    .inorder = FALSE,
    .packages = c('data.table', 'doParallel')) %dopar% {...}

It can be tricky to get familiarised with how it works, but very powerful once done.

I have implemented this 'nested' parallel processing (mclapply / parLapply inside foreach loops) in the Bioconductor package RegParallel, which has not proved hugely popular but which serves a very specific function: to rapidly run 1000s or even millions of independent regression models in chunks / batches.

Here are code chunks from it that may help you:

ADD COMMENTlink modified 4 months ago • written 4 months ago by Kevin Blighe56k
1

Many thanks Kevin it's very usefull for me !

ADD REPLYlink written 4 months ago by melania 228270
3
gravatar for zx8754
4 months ago by
zx87549.1k
London
zx87549.1k wrote:

I think the first forloop should be a normal forloop:

for(iter in seq(n.iter)){ 
   foreach(i=22:7269) %dopar% {
   ...
   }
  }
ADD COMMENTlink written 4 months ago by zx87549.1k

Thank you I will try this

ADD REPLYlink written 4 months ago by melania 228270
1

Indeed, this would be easier. You can also actually do mclapply() (or parLapply() on Windows) within a foreach statement. If you then choose, for example, 3 cores, this will result in 9 concurrent processes.

ADD REPLYlink written 4 months ago by Kevin Blighe56k

Thank you Kevin, so I can have the output as data.frame or list ?

ADD REPLYlink written 4 months ago by melania 228270
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 2139 users visited in the last hour