0

我的理解是 /: 与 foldLeft 相同，如果使用“par”将列表转换为并行集合，聚合是 foldLeft 的更快版本。如果我是正确的，为什么以下代码显示 :/ 和 foldLeft 比列表中与 'par' 一起使用的聚合更快。

我正在计算一个大列表的元素总和和元素数量，并将结果存储在一个元组 [Double,Double] 中。

//initial tuple2 (sum,count)

val tsc:Tuple2[Double,Double] = Tuple2(0.0,0.0) 

//create a large list
val largeList = List.tabulate(500000)(n=>n*n)

//note time
val time1 = System.currentTimeMillis

//using aggregate without par
largeList.aggregate(tsc) ((tsc,elem) => (tsc._1+elem, tsc._2+1), (tsc1, tsc2)=>((tsc1._1+tsc2._1), (tsc1._2+tsc2._2)))

//note time
val time2 = System.currentTimeMillis

//use aggregate with par

largeList.par.aggregate(tsc) ((tsc,elem) => (tsc._1+elem, tsc._2+1), (tsc1, tsc2)=>((tsc1._1+tsc2._1), (tsc1._2+tsc2._2)))

//note time
val time3 = System.currentTimeMillis

//use /:
(tsc /: largeList)((tsc,elem)=>(tsc._1+elem, tsc._2+1))

//note time
val time4 = System.currentTimeMillis

//use foldLeft
largeList.foldLeft(tsc)((tsc,elem)=>(tsc._1+elem, tsc._2+1))

//note time
val time5 = System.currentTimeMillis

//calcualte time difference
println ("Time without par (millisecond)"+(time2-time1))

println ("Time with par (millisecond)"+(time3-time2))

println ("Time with /: (millisecond)"+(time4-time3))

println ("Time with FoldLeft (millisecond)"+(time5-time4)

我得到以下结果

第一次运行的结果

Time without par (millisecond)1198
Time with par (millisecond)1479
Time with /: (millisecond)626
Time with FoldLeft (millisecond)661

第二次运行结果

Time without par (millisecond)703
Time with par (millisecond)581
Time with /: (millisecond)435
Time with FoldLeft (millisecond)423

I am running this in windows 10 using cmd. /: and FoldLeft seem similar in performance and are considerably better than aggregate. Aggregate with par was actually more time consuming in 1st run. Could it be an issue due to 'cmd' (console) in window not being able to leverage multi-threading (just guessing here)

4

1 回答 1

1

Several problems. Your data set is really quite small (so the overhead of parallelizing and thread management is significant). Using List means the merge step of your aggregate is O(N) - this seems to be the most significant factor once you increase the dataset size.

Switching to Vector and using a Vector 10 times as large, I get:

Time without par (millisecond)271
Time with par (millisecond)297
Time with /: (millisecond)216
Time with FoldLeft (millisecond)274

which is less dramatic. I was just doing one run, in a Scala worksheet, so seeing a lot of variability in these figures

(I only have a two-core system so the overhead of parallelization is significant compared to the gains, it seems)

于 2016-11-15T08:33:13.750 回答