1

我有一个 mapreduce 程序，只有映射器，没有减速器集。我想测试一下。我有以下测试代码

@Test
    public void testMapper() throws IOException {

      mapDriver.withInput(new LongWritable(0l), new Text(
              "af00bac654249b9d27982f19064338f4,54.0258822077885,-1.56832133466378,20121022,105507,026542913532,2093,87"));
      mapDriver.withOutput(null, [some value]);
      mapDriver.runTest();
    }

调用 mapDriver.withOutput(null, [some value]); 这条线抛出异常

java.lang.NullPointerException at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:58) at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:91 ) 在 org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:104)

Note: Mapper Generic Signature Mapper<LongWritable, Text, Void, GenericRecord>

有人可以让我知道如何为发出 null 的映射器编写测试场景吗？

如果我执行 Nullwritable.get，那么我在 org.apache.hadoop.mrunit 的 org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:73) 处得到如下 java.lang.NullPointerException 异常。 internal.io.Serialization.copy(Serialization.java:91) at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:104) at org.apache.hadoop.mrunit.TestDriver.copy( TestDriver.java:608) 在 org.apache.hadoop.mrunit.TestDriver.copyPair(TestDriver.java:612) 在 org.apache.hadoop.mrunit.TestDriver.addOutput(TestDriver.java:118) 在 org.apache.hadoop .mrunit.TestDriver.withOutput(TestDriver.java:138) 在 com.gfk.gxl.etl.common.ExtractCSVTest.testMapper(ExtractCSVTest.java:73) 在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

看起来更像是序列化中带有 Avro NullPointerException 的 MRUnit 但是答案并没有解决我的问题

 with few more research i have below update
    class org.apache.avro.generic.GenericData$Record is not able to get serializer and deserializer
    in org.apache.hadoop.mrunit.internal.io.Serialization and both are coming as null which is causing the null pointer exception



 From API code snippet  for org.apache.hadoop.mrunit.internal.io.Serialization starting at line      no 61 to 70

  try {
      serializer = (Serializer<Object>) serializationFactory
          .getSerializer(clazz);
      deserializer = (Deserializer<Object>) serializationFactory
          .getDeserializer(clazz);
    } catch (NullPointerException e) {
      throw new IllegalStateException(
          "No applicable class implementing Serialization in conf at io.serializations for "
              + orig.getClass(), e);
    }
above method serializer \ deserializer  are coming null . do we have some way to avoid it
4

2 回答 2

2

使用 NullWritable.get() 方法 insted。希望这会有所帮助。

于 2014-11-07T10:11:21.750 回答
0

不幸的是，虽然 Hadoop 可以接受空键，但您现在不能在 MRUnit 中使用空键，MRUnit 团队计划将来支持空键，请参阅此处允许空键和值作为输出，预期输出

于 2015-02-05T00:15:30.347 回答