Question: mclapply on AWS
gravatar for joe
13 months ago by
joe140 wrote:

I'm calling functions with Rscript from an AWS terminal, and I encountered an error when a function uses mclapply(). The function runs fine in RStudio (AWS AMI) when using the same environment, its just from the terminal with Rscript myFunction.R --json=config.json that an error occurs.

mclapply(1:dim(this.split)[1], SomeFun, mc.preschedule=TRUE)

mclapply quits with the error

all scheduled cores encountered errors in user code

Anyone have an idea if a there is some required parameter when calling from the terminal?

Here is my R environment (printed from inside the function)

R version 3.5.1 (2018-07-02)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: Amazon Linux 2

Matrix products: default
BLAS/LAPACK: /anaconda3/envs/r35p27/lib/R/lib/

 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C

attached base packages:
[1] stats4    parallel  stats     graphics  grDevices utils     datasets
[8] methods   base

other attached packages:
 [1] rlist_0.4.6.1               stringi_1.4.3
 [3] stringr_1.3.1               seqinr_3.4-5
 [5] rBLAST_0.99.2               ShortRead_1.40.0
 [7] GenomicAlignments_1.18.1    SummarizedExperiment_1.12.0
 [9] DelayedArray_0.8.0          matrixStats_0.54.0
[11] Biobase_2.42.0              Rsamtools_1.34.0
[13] GenomicRanges_1.34.0        GenomeInfoDb_1.18.1
[15] Biostrings_2.50.2           XVector_0.22.0
[17] IRanges_2.16.0              S4Vectors_0.20.1
[19] BiocParallel_1.16.6         BiocGenerics_0.28.0
[21] aws.s3_0.3.12               jsonlite_1.5
[23] configr_0.3.3               optparse_1.6.2

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.1             RColorBrewer_1.1-2     compiler_3.5.1
 [4] base64enc_0.1-3        bitops_1.0-6           tools_3.5.1
 [7] zlibbioc_1.28.0        digest_0.6.20          lattice_0.20-35
[10] Matrix_1.2-14          yaml_2.2.0             GenomeInfoDbData_1.2.1
[13] hwriter_1.3.2          httr_1.3.1             xml2_1.2.0
[16] ade4_1.7-13            grid_3.5.1             getopt_1.20.2
[19] data.table_1.12.2      glue_1.3.1             R6_2.2.2
[22] latticeExtra_0.6-28    magrittr_1.5           MASS_7.3-51.4
[25] aws.signature_0.5.0    ini_0.3.1              RCurl_1.95-4.12
[28] RcppTOML_0.1.6
aws mclapply cran parallel R • 621 views
ADD COMMENTlink modified 13 months ago • written 13 months ago by joe140

Check that it is not trying to use more cores than available and try setting the option mc.cores explicitly to the number of cores you want to use or that are actually available.

ADD REPLYlink written 13 months ago by Jean-Karim Heriche23k

Yes, ensure that you have registered the cores properly. I go over this, here: Parallel processing in R

ADD REPLYlink written 13 months ago by Kevin Blighe65k
gravatar for joe
13 months ago by
joe140 wrote:

Thanks all. I figured out the issue - there were some variables required for the function called by mclapply that I had assigned locally with <- instead of globally with <<- which I guess RStudio could handle and Rscript could not. It might be that the variables were automatically (inadvertently) globally assigned while testing, allowing it to work in RStudio.

ADD COMMENTlink written 13 months ago by joe140
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1168 users visited in the last hour