2

I am trying to calculate the cosine similarity between columns in a matrix. I am able to get it to work using standard for loops, but when I try to make it run in parallel to make the code run faster it does not give me the same answer. The problem is that I am unable to get the same answer using the foreach loop approach. I suspect that I am not using the correct syntax, because I have had single foreach loops work. I have tried to make the second loop a regular for loop and I used the %:% parameter with the foreach loop, but then the function doesn't even run.

Please see my attached code below. Thanks in advance for any help.

## Function that calculates cosine similarity using paralel functions.

#for calculating parallel processing
library(doParallel)

## Set up cluster on 8 cores

cl = makeCluster(8)

registerDoParallel(cl)

#create an example data
x=array(data=sample(1000*100), dim=c(1000, 100))

## Cosine similarity function using sequential for loops

cosine_seq =function (x) {

  co = array(0, c(ncol(x), ncol(x)))

  for (i in 2:ncol(x)) {
    for (j in 1:(i - 1)) {

      co[i, j] = crossprod(x[, i], x[, j])/sqrt(crossprod(x[, i]) * crossprod(x[, j]))
    }
  }

  co = co + t(co)

  diag(co) = 1

  return(as.matrix(co))

}

## Cosine similarity function using parallel for loops

cosine_par =function (x) {

  co = array(0, c(ncol(x), ncol(x)))

  foreach (i=2:ncol(x)) %dopar% {

    for (j in 1:(i - 1)) {

      co[i, j] = crossprod(x[, i], x[, j])/sqrt(crossprod(x[, i]) * crossprod(x[, j]))
    }
  }

  co = co + t(co)

  diag(co) = 1

  return(as.matrix(co))

}

## Calculate cosine similarity

tm_seq=system.time(
{

  x_cosine_seq=cosine_seq(x)

})

tm_par=system.time(
{

  x_cosine_par=cosine_par(x)

})

## Test equality of cosine similarity functions

all.equal(x_cosine_seq, x_cosine_par)

#stop cluster
stopCluster(cl)
4

2 回答 2

3

嵌套循环使用的正确并行化%:%（在此处阅读）。

library(foreach)
library(doParallel)
registerDoParallel(detectCores())    
cosine_par1 <- function (x) {  
  co <- foreach(i=1:ncol(x)) %:%
    foreach (j=1:ncol(x)) %dopar% {    
      co = crossprod(x[, i], x[, j])/sqrt(crossprod(x[, i]) * crossprod(x[, j]))
    }
  matrix(unlist(co), ncol=ncol(x))
}

我建议您在 Rcpp 中编写它，而不是并行运行它，因为foreach(i=2:n, .combine=cbind)不会总是以正确的顺序绑定列。此外，在上面的代码中，我只删除了下三角条件，但运行时间比非并行代码时间慢得多。

set.seed(186)
x=array(data=sample(1000*100), dim=c(1000, 100))
cseq <- cosine_seq(x)
cpar <- cosine_par1(x)
 all.equal(cpar, cseq)
#[1] TRUE
head(cpar[,1])
#[1] 1.0000000 0.7537411 0.7420011 0.7496145 0.7551984 0.7602620
head(cseq[,1])
#[1] 1.0000000 0.7537411 0.7420011 0.7496145 0.7551984 0.7602620

附录：对于这个特定的问题，（半）矢量化cosine_seq是可能的；cosine_vec比 .快大约 40-50 倍cosine_seq。

cosine_vec <- function(x){
  crossprod(x) / sqrt(tcrossprod(apply(x, 2, crossprod)))
}
all.equal(cosine_vec(x), cosine_seq(x))
#[1] TRUE
library(microbenchmark)
microbenchmark(cosine_vec(x), cosine_seq(x), times=20L, unit="relative")
#Unit: relative
#          expr      min       lq     mean   median       uq      max neval
# cosine_vec(x)  1.00000  1.00000  1.00000  1.00000  1.00000  1.00000    20
# cosine_seq(x) 55.81694 52.80404 50.36549 52.17623 49.56412 42.94437    20
于 2015-01-19T18:48:23.637 回答
0

要做嵌套循环foreach并使用并行实现，有两种方法。

  1. %:%+%dopar%
  2. %dopar%+%do%

请注意，对于 (1)，它实际上创建了一个 foreach 对象，您不能在其间添加任何内容。否则，您将收到一条错误消息："%:%" was passed an illegal right operand。

对于 (2)，您可以在两者之间插入任何您想要的内容。但是请记住在外部循环中添加foreach参数.package，因为内部 foreach 使用foreach包。

以下是解决余弦矩阵问题的一种巧妙方法。请注意，为了说明 (2)，我额外添加了一行，请记住将其删除以进行余弦矩阵计算。

testfunc <- function (x) {
  cl<-makeCluster(4)
  registerDoParallel(cl)
  co <- foreach(i=1:ncol(x), .combine = 'rbind', .packages = c('foreach', 'stats')) %dopar% {
    k <- rnorm(3)
    foreach (j=1:ncol(x), .combine = 'c') %do% {
      crossprod(x[, i], x[, j])/sqrt(crossprod(x[, i]) * crossprod(x[, j])) + k - k
    }
  }
  stopCluster(cl)
  co
}
x <- array(data=sample(20*10), dim=c(20, 10))
testfunc(x)
于 2020-05-01T19:47:13.593 回答