hadoop - Hadoop MapReduce 输入键错误

Question

我想传递一个浮点数组作为映射器中的键，

Such as [0.6, 0, 10]

我使用 StringUtils.join 将其转换为字符串，然后编写

context.write(new Text(str), value);

但是在减速器类中，我发现我得到的是：

"0 0 1"

如何解决？我正在使用 hadoop 1.0.4

Answer 1

您可以使用 . 发出浮点数组ArrayWritable。我不确定将数组作为键发出，下面是您如何发出与值相同的值。

ArrayWritable does not implement WritableComparable, so it can't currently
be used as a mapper output key - those keys have to be sorted during the
shuffle, and thus the type must be WritableComparable.

通常我们使用自定义的 Writable Comparable Class 来发射键。

为此，您应该在驱动程序类 public class Driver {

public static class DoubleArrayWritable extends ArrayWritable {
    public DoubleArrayWritable() {
        super(DoubleWritable.class);
    }
}

public static void main(String[] args) {

在映射器中

DoubleArrayWritable pointArray = new DoubleArrayWritable();
DoubleWritable[] data = new DoubleWritable[numAttributes];
for (int k = 0; k < numAttributes; k++) {
    data[k] = new DoubleWritable(point[k]);
}
pointArray.set(data);
context.write(new Text("Array"), pointArray);

希望这在某些方面有所帮助。