0

我们在 Databricks 环境中使用结构化流，每次我们运行这个程序时 - kAFKA - 结构化流（DBR6.6，Spark 2.4.5） - 写入 CosmosDB，在我们执行最终操作之前，我们都会遇到与下面相同的异常连接以将数据保存到 Cosmos DB。我们没有修改任何 spark 特定设置并利用默认 spark /DBR 配置。

Caused by: org.apache.spark.SparkException:
           Job aborted due to stage failure:
           Task 174 in stage 9353.0 failed 4 times, most recent failure:
           Lost task 174.3 in stage 9353.0 (TID 60863, 10.139.64.9, executor 1): 
           java.lang.IllegalStateException:
           Error reading delta file dbfs:/raw_zone/uffRetail_jointbl_dev_cp1/state/8/174/left-keyToNumValues/1.delta of HDFSStateStoreProvider[id = (op=8,part=174),dir = dbfs:/raw_zone/uffRetail_jointbl_dev_cp1/state/8/174/left-keyToNumValues]: 
           dbfs:/raw_zone/uffRetail_jointbl_dev_cp1/state/8/174/left-keyToNumValues/1.delta does not exist
Caused by: java.io.FileNotFoundException:
           /6455647419774311/raw_zone/uffRetail_jointbl_dev_cp1/state/8/174/left-keyToNumValues/1.delta
4

0 回答 0