1

我正在使用pysparkDatabriks，我有一个如下所示的数据点表

pingsGeo.show(5)
+--------------------+--------------------+----------+--------------------+
|                  ID|               point|      date|            distance|
+--------------------+--------------------+----------+--------------------+
|00007436cf7f96cb1...|POINT (-82.640937...|2020-03-19|0.022844737780675896|
|00007436cf7f96cb1...|POINT (-82.641281...|2020-03-19|3.946137920280456...|
|00007436cf7f96cb1...|POINT (-82.650238...|2020-03-19| 0.00951798692682881|
|00007436cf7f96cb1...|POINT (-82.650947...|2020-03-19|7.503617154519347E-4|
|00007436cf7f96cb1...|POINT (-82.655853...|2020-03-19|0.007148426134394903|
+--------------------+--------------------+----------+--------------------+

root
 |-- ID: string (nullable = true)
 |-- point: geometry (nullable = false)
 |-- date: date (nullable = true)
 |-- distance: double (nullable = false)

还有另一个多边形表（来自 shapefile）

zoneShapes.show(5)
+--------+--------------------+
|COUNTYNS|            geometry|
+--------+--------------------+
|01026336|POLYGON ((-78.901...|
|01025844|POLYGON ((-80.497...|
|01074088|POLYGON ((-81.686...|
|01213687|POLYGON ((-76.813...|
|01384015|POLYGON ((-95.152...|

我想为每个点分配一个COUNTYNS

我正在用geospark函数来做。我正在执行以下操作：

queryOverlap = """
        SELECT p.ID, z.COUNTYNS as zone,  p.date, p.point, p.distance
        FROM pingsGeo as p, zoneShapes as z
        WHERE ST_Intersects(p.point, z.geometry))
    """

spark.sql(queryOverlap).show(5)

此查询适用于小数据集，但对于较大的数据集则失败。

org.apache.spark.SparkException: Job aborted due to stage failure: Task 117 in stage 51.0 failed 4 times, most recent failure: Lost task 117.3 in stage 51.0 (TID 4879, 10.17.21.12, executor 13): org.apache.spark.memory.SparkOutOfMemoryError: Unable to acquire 16384 bytes of memory, got 0

我想知道是否有办法优化这个过程。

4

1 回答 1

0

你的问题有点含糊，但这是我首先要看的内容。

有几件事情需要考虑： 1. Spark 集群可用的物理资源 2. 表分区 - 如果分区不正确，您可能正在执行大于默认大小的数据混洗

另外，考虑在最大的表上使用索引。

于 2020-05-08T09:00:23.533 回答