1

由于数据引擎中的迭代而面临错误，因此stackoverflow异常如下：

ERROR org.apache.spark.executor.Executor [Executor task launch worker-0] - Exception in task 0.0 in stage 30.0 (TID 76)
java.lang.StackOverflowError
    at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2774)
    at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1450)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1512)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
4

1 回答 1

3

得到了错误的解决方案：

1.只需在预测引擎的engine.json文件中减少算法的numIterations参数。

或者如果这不起作用，请使用下面的另一个解决方案。

2.添加检查点，防止代码库使用的递归造成溢出。首先，创建一个新目录来存储检查点。然后，让您的 SparkContext 使用该目录进行检查点。这是 Python 中的示例：

sc.setCheckpointDir('checkpoint/') 您可能还需要向 ALS 添加检查点，但我无法确定这是否会产生影响。要在此处添加检查点（可能没有必要），只需执行以下操作：

ALS.checkpointInterval = 2

于 2015-12-08T05:32:54.670 回答