我有大量的 json 文件,Spark 可以在 36 秒内读取,但 Spark 3.0 需要将近 33 分钟才能读取相同的文件。仔细分析后,看起来 Spark 3.0 选择的 DAG 与 Spark 2.0 不同。有谁知道发生了什么?Spark 3.0 是否有任何配置问题。
火花 2.4
scala> spark.time(spark.read.json("/data/20200528"))
Time taken: 19691 ms
res61: org.apache.spark.sql.DataFrame = [created: bigint, id: string ... 5 more fields]
scala> spark.time(res61.count())
Time taken: 7113 ms
res64: Long = 2605349
火花3.0
scala> spark.time(spark.read.json("/data/20200528"))
20/06/29 08:06:53 WARN package: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.
Time taken: 849652 ms
res0: org.apache.spark.sql.DataFrame = [created: bigint, id: string ... 5 more fields]
scala> spark.time(res0.count())
Time taken: 8201 ms
res2: Long = 2605349
以下是详细信息: