2

我想在 R 中运行 1000 次模拟迭代。每次迭代需要 20 秒，这需要大约 6 个小时的串行时间。而且我有几十个模拟要运行，每个都有 1000 次迭代。所以我想用并行计算在更短的时间内完成这项工作。

我对此很陌生。在阅读了网络上的资料后，我现在使用 Amazon 的 EC2 服务，使用 Louis Aslett 提供的 RStudio Server 和 OpenBLAS 运行 AMI。加载到 AMI 的软件的详细信息在这里：http ://www.louisaslett.com/RStudio_AMI 。

我尝试了几个 EC2 实例，目前正在使用具有 36 个内核的 c4.8xlarge ubuntu 实例。或虚拟内核，我不完全确定区别。

我的问题是我似乎无法在可用的 36 个内核附近使用任何地方。我已经成功使用了多达 10 个内核，但是当我尝试使用 >10 个内核时出现错误。

这是重现错误的最少代码（使用汽车数据集）：

library(parallel)
detectCores() #36
ptm<-proc.time()
cl <- makeCluster(getOption("cl.cores", 20)) #specify number of cores
clusterSetRNGStream(cl, 123) 
sims <- clusterEvalQ(cl, { 
  cars[sample(1:nrow(cars), 10, replace=FALSE),]
}) 
stopCluster(cl)

当我在 makeCluster() 语句中指定 10 个内核时，该代码通常可以正常工作。它生成一个包含 10 个元素的列表，每个元素由 10 个核心中的一个生成。但是指定 >10 个核心（即如上所述）通常会导致错误，例如我在下面粘贴的内容。

正如我上面提到的，我是新手，我的代码基于我在网上找到的一个示例，该示例旨在在单台多核计算机（而不是 EC2 机器）上运行。所以我意识到我可能为了在 EC2 上运行而错误地指定了 makeCluster() 语句。

我将不胜感激任何帮助或建议以使其正常工作！

非常感谢，

马克 B。

这是我尝试使用 >10 个内核时遇到的错误：

OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
/usr/lib/R/bin/R: fork: retry: Resource temporarily unavailable
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
/usr/lib/R/bin/R: fork: retry: No child processes
/usr/lib/R/bin/R: fork: retry: Resource temporarily unavailable
/usr/lib/R/bin/R: fork: retry: No child processes
/usr/lib/R/bin/R: fork: retry: No child processes
/usr/lib/R/bin/R: fork: retry: Resource temporarily unavailable
/usr/lib/R/bin/R: fork: retry: No child processes
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 241473 current, 241473 max
/usr/lib/R/bin/R: fork: retry: No child processes
/usr/lib/R/bin/R: fork: retry: No child processes
/usr/lib/R/bin/R: fork: retry: No child processes
/usr/lib/R/bin/R: fork: retry: No child processes
/usr/lib/R/bin/R: fork: retry: No child processes
/usr/lib/R/bin/R: fork: retry: No child processes
/usr/lib/R/bin/R: fork: Resource temporarily unavailable
ERROR: option '-e' requires a non-empty argument
4

1 回答 1

1

本周我们也遇到了这个问题。我所做的是从 EC2 中删除 openBLAS 并退出 R，然后在没有 openBLAS 的情况下重新安装 R。然后它起作用了。但是如果你真的需要 openBLAS 来加速你的算法，你可能需要使用 openBLAS 从源代码构建 R（我们还没有尝试过这种方法）。

于 2016-07-22T03:52:20.970 回答