0

我目前正在尝试在 SnappyData 上运行 TPC-H 查询。起初，查询给了我一个错误说

ERROR 38000: (SQLState=38000 Severity=-1) (Server=localhost[1528],Thread[DRDAConnThread_29,5,gemfirexd.daemons]) 异常'此连接的双方都超出了广播阈值，计算它可能会令人望而却步昂贵的。要显式启用它，请设置 spark.sql.crossJoin.enabled = true;' 在评估表达式时抛出。

启用 spark 的 sql crossjoins 并重新运行查询后，错误弹出说：

java.lang.RuntimeException: Can't acquire 1049600 bytes memory to build hash relation, got 74332 bytes
    at org.apache.spark.sql.execution.joins.HashedRelationCache$.get(LocalJoin.scala:621)
    at org.apache.spark.sql.execution.joins.HashedRelationCache.get(LocalJoin.scala)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.init(Unknown Source)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8.apply(WholeStageCodegenExec.scala:367)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8.apply(WholeStageCodegenExec.scala:364)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$25.apply(RDD.scala:820)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$25.apply(RDD.scala:820)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
    at org.apache.spark.scheduler.Task.run(Task.scala:86)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Can't acquire 1049600 bytes memory to build hash relation, got 74332 bytes
    at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.ensureAcquireMemory(HashedRelation.scala:414)
    at org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.init(HashedRelation.scala:424)
    at org.apache.spark.sql.ex

请让我知道如何增加用于构建哈希关系的内存量。

以防万一，查询在下面，我试图在 1GB 数据集上运行它（我确实在空数据集上尝试了查询，它确实有效）。TPC-H 查询 16：

SELECT i_name,
       substr(i_data, 1, 3) AS brand,
       i_price,
       count(DISTINCT (pmod((s_w_id * s_i_id),10000))) AS supplier_cnt
FROM stock,
     item
WHERE i_id = s_i_id
  AND i_data NOT LIKE 'zz%'
  AND (pmod((s_w_id * s_i_id),10000) NOT IN
    (SELECT su_suppkey
     FROM supplier
     WHERE su_comment LIKE '%bad%'))
GROUP BY i_name,
         substr(i_data, 1, 3),
         i_price
ORDER BY supplier_cnt DESC;
4

1 回答 1

2

在服务器 (conf/servers) 的配置文件中，将 jvm 内存设置为 -J-Xms5g 所以 conf/server 看起来像 localhost -locators=localhost:10334 -J-Xms5g

于 2016-10-17T04:56:44.270 回答