下面的代码在本地运行,但不在集群上。它挂在 GroupReduceFunction 上,即使在几小时后也不会终止(大数据需要 9 分钟才能在本地计算)。日志中的最后一条消息:
GroupReduce (GroupReduce at main(MyClass.java:80)) (1/1) (...) switched from DEPLOYING to RUNNING.
代码片段:
DataSet<MyData1> myData1 = env.createInput(new UserDefinedFunctions.MyData1Set());
DataSet<MyData2> myData2 = DataSetUtils.sampleWithSize(myData1, false, 8, Long.MAX_VALUE)
.reduceGroup(new GroupReduceFunction<MyData1, MyData2>() {
@Override
public void reduce(Iterable<MyData1> itrbl, Collector<MyData2> clctr) throws Exception {
int id = 0;
for (MyData1 myData1 : itrbl) {
clctr.collect(new MyData2(id++, myData1));
}
}
});
有什么想法可以并行运行这个部分吗?提前致谢!